MATH301 — Linear Algebra and Its Applications
Core Concepts
Personal study notes for Skipper. Primary sources: Lay, Lay & McDonald, Linear Algebra and Its Applications, 5th ed. (Pearson, 2016) and Strang, Introduction to Linear Algebra, 5th ed. (Wellesley-Cambridge, 2016).
0. What Is This Subject Really About?
Most first courses present linear algebra as "how to solve systems of equations with matrices." That is where the subject starts, but it is not what the subject is. Linear algebra studies linear structure: spaces where you can add things and scale them, and maps between those spaces that respect adding and scaling. Once you look for this structure, it shows up across the sciences. Examples include atomic coordinates of a peptide, absorbance spectra, quantum wavefunctions, stoichiometric coefficients, and gene expression profiles.
Calculus (MATH101/102) already contains the key idea. A derivative is the best linear approximation to a function near a point. Calculus works by approximating curved things locally with linear things. Linear algebra is the full theory of those linear things. Whenever you linearize a chemical system, you hand the problem to linear algebra. This includes expanding a potential energy surface to second order around a minimum, linearizing rate equations near a steady state, and treating small perturbations in spectroscopy.
"Linear algebra is a fantastic subject. ... The central problem is to solve linear equations, and the central objects are vectors and matrices. But the deeper goal is to understand the spaces they create."
— paraphrasing the framing of [Strang, 2016, Preface and Ch. 1]
There are three ways to look at a matrix, and fluency means moving between them easily:
- A table of numbers. This is the computational view: row operations and arithmetic.
- A collection of column vectors. This is the geometric view.
Axis a combination of the columns of A, with weights given by x. - A function (linear transformation). This is the dynamic view. A matrix does something: it rotates, stretches, projects, or collapses space.
Strang's main teaching move is to replace "row times column" with "combination of columns" as the default picture of Ax [Strang, 2016, §1.3, §2.1]. Lay et al. make the same move by defining the matrix-vector product as a linear combination of columns from the beginning [Lay, Lay & McDonald, 2016, §1.4]. Adopt this view now. Much of what follows becomes obvious once you do.
1. Linear Systems, Row Reduction, and Echelon Form
A system of m linear equations in n unknowns can be written compactly as
A x = b, A ∈ ℝ^(m×n), x ∈ ℝ^n, b ∈ ℝ^m
The row picture reads each equation as a hyperplane, and the solution is where all the hyperplanes intersect. The column picture asks: which combination of the columns of A produces b?
x₁·(col 1) + x₂·(col 2) + ... + xₙ·(col n) = b
The column picture is the one that generalizes. The question "does Ax = b have a solution?" becomes "is b in the span of the columns of A?" That is a geometric question about subspaces, and the rest of the course builds on it.
Row reduction (Gaussian elimination) uses three elementary row operations: swapping rows, scaling a row by a nonzero constant, and adding a multiple of one row to another. None of these changes the solution set. Reduction ends in echelon form, or, if pushed all the way, in the unique reduced row echelon form (RREF). In RREF, each pivot is 1, each pivot is the only nonzero entry in its column, and the pivots move strictly right as you go down [Lay et al., 2016, §1.2].
Reading the RREF gives you the whole story:
- Pivot columns correspond to basic variables. Non-pivot columns correspond to free variables.
- A row of the form
[0 0 ... 0 | c]with c ≠ 0 means the system is inconsistent (no solution). - If the system is consistent and has at least one free variable, there are infinitely many solutions.
- If it is consistent with no free variables, the solution is unique.
The general solution of a consistent system has a structure that is worth understanding properly:
x = x_p + x_h
(particular solution) + (any solution of A x = 0)
You have seen this exact structure before. In MATH102, the general solution of a linear ODE was a particular solution plus the general homogeneous solution. This is not a coincidence. A linear differential operator is a linear transformation, and the theorem is the same theorem [Lay et al., 2016, §1.5].
Chemistry flashback: balancing a chemical equation is a homogeneous linear system. For CH₄ + O₂ → CO₂ + H₂O, write one equation per element (C, H, O) in the unknown coefficients (a, b, c, d):
C: a - c = 0 H: 4a - 2d = 0 O: 2b - 2c - d = 0 Null space spanned by (a, b, c, d) = (1, 2, 1, 2)
The balanced equation is the null space of the element-composition matrix. If that null space were 2-dimensional, the "reaction" would really be two independent reactions superimposed. This is exactly how stoichiometric network analysis in metabolism works.
2. Matrix Multiplication Is Composition of Functions
The matrix product rule, (AB)ij = Σk aikbkj, looks arbitrary until you see why it is defined that way. If B represents the transformation x ↦ Bx and A represents y ↦ Ay, then AB is the single matrix that represents "first do B, then do A":
(AB) x = A (B x)
The rule exists to make composition work [Lay et al., 2016, §2.1]. Several consequences follow immediately:
- Non-commutativity is natural. Rotating a molecule about x and then about z is generally not the same as doing it in the reverse order. Pick up a book and try it. Matrices inherit this non-commutativity because they represent actions.
- Associativity is free, because composition of functions is always associative.
- Shapes must match: (m×n)(n×p) = (m×p). The inner dimensions agree because B's output must be A's input.
Strang emphasizes four equivalent ways to compute AB [Strang, 2016, §2.4]:
- Entry by entry: row i of A dotted with column j of B.
- Column by column: A times each column of B. Each column of AB is a combination of the columns of A.
- Row by row: each row of AB is a combination of the rows of B.
- Sum of outer products: AB = Σk (column k of A)(row k of B).
The fourth view will seem like a curiosity at first. It turns out to be the most important for applications. It is exactly how the SVD expresses a matrix as a sum of rank-one pieces (§12).
3. The Inverse and the Invertible Matrix Theorem
A square matrix A is invertible if there exists A⁻¹ with AA⁻¹ = A⁻¹A = I. Geometrically, A is invertible when its transformation can be undone: no information is lost and no two inputs land on the same output.
One of the most satisfying results in the course is the Invertible Matrix Theorem [Lay et al., 2016, §2.3, extended in §§2.9, 3.2, 5.2, 7.4]. It states that for an n×n matrix A, a long list of apparently different conditions are all equivalent:
- A is invertible.
- A is row equivalent to In (n pivots).
- Ax = 0 has only the trivial solution.
- The columns of A are linearly independent.
- The columns of A span ℝⁿ.
- The map x ↦ Ax is one-to-one (equivalently, onto).
- Ax = b has a unique solution for every b.
- Aᵀ is invertible.
- det A ≠ 0.
- 0 is not an eigenvalue of A.
- rank A = n, and Nul A = {0}.
- A has n nonzero singular values.
Treat this theorem as a hub. Each new topic in the course adds another equivalent condition. When you face a question about one condition, you can check whichever condition is easiest. One caveat: the theorem applies only to square matrices. For a non-square matrix, "one-to-one" and "onto" come apart.
The useful algebraic rules are:
(AB)⁻¹ = B⁻¹ A⁻¹ (reverse order: socks then shoes, undo shoes then socks) (Aᵀ)⁻¹ = (A⁻¹)ᵀ (AB)ᵀ = Bᵀ Aᵀ 2×2 case: [a b; c d]⁻¹ = (1/(ad − bc)) · [d −b; −c a]
A practical warning: numerical scientists almost never compute A⁻¹ explicitly to solve Ax = b. They factor A (for example, A = LU) and back-substitute. This is faster and more stable. The inverse is mainly a conceptual tool [Strang, 2016, §2.6].
4. Determinants: The Volume-Scaling Factor
The cofactor expansion formula for determinants is usually taught first. It is the least illuminating way to understand them.
det A = Σⱼ (−1)^(i+j) a_ij · det(A_ij) (expansion along row i)
The geometric meaning is this: |det A| is the factor by which the transformation A scales n-dimensional volume. The sign of det A tells you whether orientation is preserved (+) or reversed (−). A unit square mapped by a 2×2 matrix becomes a parallelogram with area |det A| [Lay et al., 2016, §3.3].
With that picture, the main properties become intuitive:
- det A = 0 ⇔ A is singular. The transformation squashes space into a lower dimension, giving zero volume, so information is lost and cannot be recovered.
- det(AB) = det A · det B. Volume scalings compose by multiplication.
- det(A⁻¹) = 1 / det A.
- det(Aᵀ) = det A.
- For a triangular matrix, det A is the product of the diagonal entries.
- Swapping two rows flips the sign. Scaling a row by k scales det by k. Adding a multiple of one row to another leaves det unchanged.
The last property gives the efficient way to compute determinants: row reduce to triangular form while tracking swaps and scalings. Cofactor expansion takes on the order of n! operations. Elimination takes about n³/3.
Cramer's rule gives an explicit formula for each component of the solution of an invertible system:
xᵢ = det(Aᵢ(b)) / det(A) where Aᵢ(b) is A with column i replaced by b.
Cramer's rule is beautiful and theoretically useful. For example, it shows that the solution depends continuously and rationally on the entries of A. Computationally, however, it is hopeless for large n [Lay et al., 2016, §3.3].
Connection to calculus: the Jacobian determinant in a change of variables (Cartesian to spherical, ∫∫∫ r² sin θ dr dθ dφ) is exactly this volume-scaling factor, applied to the local linearization of a nonlinear map. You used this in PHYS202 and in hydrogen-atom orbital normalization in CHEM101 without knowing it came from linear algebra.
5. Vector Spaces, Subspaces, Span, Independence, Basis, Dimension
Here the subject becomes abstract. The abstraction pays off because one theory then covers many different settings.
A vector space is any set V with addition and scalar multiplication satisfying ten axioms: closure, associativity, commutativity, zero vector, additive inverses, distributive laws, and so on [Lay et al., 2016, §4.1]. Examples:
- ℝⁿ, the familiar one.
- The polynomials of degree ≤ n.
- The continuous functions on [a, b].
- The solutions of a linear homogeneous ODE.
- The set of m×n matrices.
- Quantum wavefunctions. The superposition principle is exactly the statement that they form a vector space.
A subspace H ⊆ V is a subset that is itself a vector space. To check, verify three things: 0 ∈ H, H is closed under addition, and H is closed under scalar multiplication. Geometrically, subspaces of ℝ³ are {0}, lines through the origin, planes through the origin, and ℝ³ itself. A line that misses the origin is not a subspace.
Four definitions form the core vocabulary:
Span{v₁,...,vₚ} = { c₁v₁ + ... + cₚvₚ : cᵢ ∈ ℝ }
Linearly independent ⇔ c₁v₁ + ... + cₚvₚ = 0 forces all cᵢ = 0
Basis of V = linearly independent set that spans V
dim V = number of vectors in any basis of V
A useful way to think about these: a spanning set has enough vectors to reach everything, and an independent set has no redundant vectors. A basis does both, so it is exactly the right size. The non-obvious theorem is that every basis of a given space has the same number of elements [Lay et al., 2016, §4.5]. That is why "dimension" is well defined. It is a property of the space, not of the coordinates you choose.
Once you fix a basis ℬ = {b₁, ..., bₙ}, every vector has unique coordinates [x]ℬ, and V behaves exactly like ℝⁿ (V is isomorphic to ℝⁿ). This is why a 4-dimensional polynomial space can be handled entirely with 4-component column vectors.
Analogy: a basis is like a set of reference reagents. Every mixture in the space can be made from them in exactly one way, none can be made from the others, and the number of reference reagents you need is fixed by the system, not by which ones you happen to choose.
6. The Four Fundamental Subspaces and the Rank–Nullity Theorem
Strang calls this "the Fundamental Theorem of Linear Algebra" and organizes much of his book around it [Strang, 2016, §§3.5, 4.1]. Every m×n matrix A of rank r has four associated subspaces:
Subspace Lives in Dimension Definition ---------------- ---------- ----------- ------------------------------ Column space C(A) ℝ^m r all A x Null space N(A) ℝ^n n − r all x with A x = 0 Row space C(Aᵀ) ℝ^n r all Aᵀ y Left null N(Aᵀ) ℝ^m m − r all y with Aᵀ y = 0
Two striking facts:
- Row rank = column rank. The number of independent rows always equals the number of independent columns, even for a 3×1000 matrix. This is far from obvious.
- Orthogonal complements. The row space and null space are orthogonal complements in ℝⁿ. The column space and left null space are orthogonal complements in ℝᵐ. The reason is direct: Ax = 0 means x is perpendicular to every row of A.
The Rank–Nullity Theorem is the dimensional accounting for this picture:
rank(A) + dim N(A) = n (number of columns)
Interpretation: the n input dimensions split into those that survive the transformation (rank) and those that are annihilated (nullity). Nothing is created or destroyed. It works like a conservation law for dimensions.
How to find bases in practice [Lay et al., 2016, §§4.2–4.6]:
- Column space: take the pivot columns of the original A, not of the RREF.
- Row space: take the nonzero rows of the RREF. Row operations preserve the row space.
- Null space: solve Ax = 0 from the RREF. Each free variable contributes one basis vector.
7. Linear Transformations, Kernel, Image, and Change of Basis
A map T: V → W is linear if
T(u + v) = T(u) + T(v) and T(c u) = c T(u)
Every linear map ℝⁿ → ℝᵐ is multiplication by a unique matrix, the standard matrix:
A = [ T(e₁) T(e₂) ... T(eₙ) ]
This is a remarkable compression. To know T everywhere, you only need to know what it does to the n basis vectors [Lay et al., 2016, §1.9]. Linearity fills in the rest.
The kernel (ker T = {x: T(x) = 0}) is the null space. The image (range) is the column space. T is one-to-one ⇔ ker T = {0}. This test is very useful: to check whether distinct inputs give distinct outputs, you only need to check what maps to zero.
Change of basis. The same linear transformation looks different in different coordinate systems. If P has the new basis vectors as its columns, the matrix of T in the new basis is
A' = P⁻¹ A P (A and A' are "similar" matrices)
Similar matrices share everything intrinsic to the transformation: determinant, trace, rank, eigenvalues, and characteristic polynomial. Much of advanced linear algebra is the search for the best basis, the one in which a transformation looks simplest. Diagonalization, the spectral theorem, the SVD, and the Fourier transform are all answers to that question, each for a different kind of problem.
The theme to carry through the rest of the course: choose coordinates adapted to the problem, and hard problems become easy.
This mirrors a habit from physical chemistry. You choose center-of-mass and internal coordinates for a diatomic, or spherical coordinates for the hydrogen atom, because the problem separates in those coordinates.
8. Eigenvalues, Eigenvectors, and Diagonalization
An eigenvector of A is a nonzero vector whose direction is unchanged by A. It is only stretched, by a factor λ called the eigenvalue:
A v = λ v, v ≠ 0
To find eigenvalues, note that (A − λI)v = 0 must have a nontrivial solution, so A − λI must be singular:
det(A − λI) = 0 ← the characteristic polynomial (degree n)
For each eigenvalue λ, the eigenvectors (together with 0) form the eigenspace N(A − λI).
Two useful checks, valid for any square matrix:
trace(A) = Σ λᵢ det(A) = Π λᵢ (counted with multiplicity, including complex λ)
Diagonalization. If A has n linearly independent eigenvectors, collect them as the columns of P, and put the eigenvalues on the diagonal of D. Then
A = P D P⁻¹ ⇒ Aᵏ = P Dᵏ P⁻¹
In the eigenbasis, A acts by scaling each coordinate independently. Coupled problems become uncoupled ones. Matrix powers, matrix exponentials, and linear ODE systems all become easy:
dx/dt = A x ⇒ x(t) = Σ cᵢ e^(λᵢ t) vᵢ
Direct connection to CHEM102 kinetics: a network of first-order reactions (A ⇌ B ⇌ C) is exactly dc/dt = Kc, with K the rate-constant matrix. The eigenvalues of K are the observable relaxation rates, and the eigenvectors are the "normal modes" of concentration change. One eigenvalue is zero, and its eigenvector is the equilibrium state. This is the reason relaxation kinetics in temperature-jump experiments shows a sum of exponentials.
When does diagonalization fail? When an eigenvalue's geometric multiplicity (dimension of its eigenspace) is less than its algebraic multiplicity (multiplicity as a root). The standard example is the shear [1 1; 0 1]. It has λ = 1 twice but only one independent eigenvector [Lay et al., 2016, §5.3]. Distinct eigenvalues always give independent eigenvectors, so a matrix with n distinct eigenvalues is always diagonalizable.
Complex eigenvalues are not a breakdown. They signal rotation. A real 2×2 matrix with eigenvalues a ± bi acts as a rotation combined with scaling by √(a² + b²) [Lay et al., 2016, §5.5]. In a dynamical system, complex eigenvalues produce oscillations.
9. Symmetric Matrices and the Spectral Theorem
When A = Aᵀ, the theory becomes much cleaner. The Spectral Theorem [Lay et al., 2016, §7.1; Strang, 2016, §6.4] states:
- All eigenvalues of a real symmetric matrix are real.
- Eigenvectors for distinct eigenvalues are orthogonal.
- A is always diagonalizable, and it is orthogonally diagonalizable:
A = Q Λ Qᵀ, QᵀQ = I (so Q⁻¹ = Qᵀ) Spectral decomposition: A = λ₁ q₁q₁ᵀ + λ₂ q₂q₂ᵀ + ... + λₙ qₙqₙᵀ
Every symmetric matrix is a weighted sum of projections onto perpendicular directions. Geometrically, a symmetric matrix stretches space along n mutually perpendicular axes, with no shear and no rotation.
This is why the theorem matters so much in physics and chemistry. Observables in quantum mechanics are Hermitian operators, the complex analogue of symmetric matrices. That they have real eigenvalues is why measurements give real numbers. That their eigenvectors are orthogonal is why distinct energy levels have orthogonal wavefunctions, a fact you used in PHYS202 and in CHEM101 orbital theory. The spectral theorem is the mathematical guarantee behind these facts.
Quadratic forms and definiteness. A symmetric A defines a quadratic form Q(x) = xᵀAx. Changing to the eigenbasis removes the cross terms:
xᵀ A x = λ₁ y₁² + λ₂ y₂² + ... + λₙ yₙ² (y = Qᵀx)
A is positive definite if all λ > 0. The quadratic form is then a bowl, and a critical point with a positive definite Hessian is a local minimum. This is the second-derivative test from calculus generalized to n dimensions, and it is how computational chemists recognize minima (all positive Hessian eigenvalues) and transition states (exactly one negative eigenvalue) on a potential energy surface.
10. Orthogonality, Projections, and Gram–Schmidt
The dot product (more generally, an inner product) brings geometry into linear algebra: lengths, angles, and perpendicularity.
u · v = uᵀv = Σ uᵢvᵢ ‖u‖ = √(u · u) cos θ = (u · v) / (‖u‖ ‖v‖)
Orthogonal bases make calculation easy. If {u₁, ..., uₚ} is an orthogonal basis, the coordinates of any y come directly from dot products, with no system of equations to solve:
y = Σ ( (y · uᵢ) / (uᵢ · uᵢ) ) uᵢ
Orthogonal projection onto a subspace W gives the point of W closest to y (the Best Approximation Theorem) [Lay et al., 2016, §6.3]. If the columns of A form a basis for W:
proj_W(y) = A (AᵀA)⁻¹ Aᵀ y P = A(AᵀA)⁻¹Aᵀ, P² = P, Pᵀ = P If W has an orthonormal basis (columns of Q): P = Q Qᵀ
The error y − projW(y) is perpendicular to W. Remember this picture, because least squares is built on it: the shortest path from a point to a plane is the perpendicular.
Gram–Schmidt converts any basis {x₁, ..., xₚ} into an orthogonal one. Each new vector is made orthogonal to the previous ones by subtracting its projections onto them:
v₁ = x₁ v₂ = x₂ − (x₂·v₁ / v₁·v₁) v₁ v₃ = x₃ − (x₃·v₁ / v₁·v₁) v₁ − (x₃·v₂ / v₂·v₂) v₂ ⋮ Normalize: qᵢ = vᵢ / ‖vᵢ‖
Chemists have already met this procedure: the construction of orthogonal hybrid orbitals and of orthogonalized basis sets in quantum chemistry (Löwdin orthogonalization is a symmetric variant) rest on the same idea. In function spaces, applying Gram–Schmidt to 1, x, x², ... with the inner product ∫₋₁¹ f g dx produces the Legendre polynomials, the same polynomials that appear in the angular part of hydrogen-atom orbitals.
11. Least Squares, Normal Equations, and QR Factorization
Real data is noisy, so Ax = b usually has no solution when there are more equations than unknowns (an overdetermined system). Fitting a line to 20 calibration points is a typical case. The least-squares answer is to find x̂ minimizing ‖b − Ax‖². Geometrically, Ax̂ is the projection of b onto the column space C(A). The residual must be perpendicular to C(A), so Aᵀ(b − Ax̂) = 0. This gives the normal equations:
AᵀA x̂ = Aᵀ b ⇒ x̂ = (AᵀA)⁻¹ Aᵀ b (if columns of A independent)
The name comes from the fact that the residual is normal (perpendicular) to the column space [Lay et al., 2016, §6.5; Strang, 2016, §4.3].
Connection to MATH201: linear regression is least squares. For the model y = β₀ + β₁x, the design matrix has a column of ones and a column of x-values, and the normal equations reproduce the familiar slope and intercept formulas. Under Gaussian noise, the least-squares estimate is also the maximum-likelihood estimate. So the geometry (orthogonal projection) and the statistics (MLE) lead to the same estimator, as you saw in MATH201.
QR factorization. Applying Gram–Schmidt to the columns of A gives A = QR, with Q having orthonormal columns and R upper triangular and invertible. Then:
A = QR ⇒ R x̂ = Qᵀ b (solve by back-substitution)
Why bother, when the normal equations already work? Because forming AᵀA squares the condition number: κ(AᵀA) = κ(A)² in the 2-norm. If A is even moderately ill-conditioned, for example a polynomial fit with nearly collinear columns, solving the normal equations can lose many digits of precision. QR avoids forming AᵀA and is the standard approach in numerical practice [Strang, 2016, §4.4]. (Production codes use Householder reflections or modified Gram–Schmidt instead of classical Gram–Schmidt, for further stability.)
12. The Singular Value Decomposition (SVD), Compression, and PCA
Strang has called the SVD the climax of linear algebra, and it deserves the description. Diagonalization needs a square matrix and may fail. The spectral theorem needs symmetry. The SVD works for every matrix, of any shape and any rank:
A = U Σ Vᵀ A: m×n U: m×m orthogonal Σ: m×n diagonal (σ₁ ≥ σ₂ ≥ ... ≥ σᵣ > 0) V: n×n orthogonal A = σ₁ u₁v₁ᵀ + σ₂ u₂v₂ᵀ + ... + σᵣ uᵣvᵣᵀ
Geometric meaning: every linear transformation, however complicated, does three things in sequence. It rotates (and possibly reflects) by Vᵀ, stretches along perpendicular axes by Σ, then rotates (and possibly reflects) by U. The unit sphere is always mapped to an ellipsoid, with semi-axes σᵢuᵢ [Lay et al., 2016, §7.4].
How it connects to the spectral theorem: AᵀA is symmetric and positive semidefinite, so it can be orthogonally diagonalized. Its eigenvectors are the vᵢ, and its eigenvalues are σᵢ². Similarly, AAᵀ has eigenvectors uᵢ. The SVD also provides orthonormal bases for all four fundamental subspaces at once:
- v₁…vᵣ span the row space, and vᵣ₊₁…vₙ span the null space.
- u₁…uᵣ span the column space, and uᵣ₊₁…uₘ span the left null space.
So the SVD is a computational form of the Fundamental Theorem (§6).
Low-rank approximation (Eckart–Young theorem). Truncating the sum after k terms gives the best rank-k approximation to A, in both the 2-norm and the Frobenius norm:
Aₖ = σ₁u₁v₁ᵀ + ... + σₖuₖvₖᵀ ‖A − Aₖ‖₂ = σₖ₊₁
An image stored as a 1000×1000 matrix needs 10⁶ numbers. If the singular values decay quickly, keeping k = 50 terms needs only 50 × (1000 + 1000 + 1) ≈ 10⁵ numbers, with little visible loss. This is data compression.
Principal Component Analysis (PCA). Arrange data as an n×p matrix X, with rows as observations and columns as variables, and center each column by subtracting its mean. The sample covariance matrix is S = XᵀX/(n − 1). Its eigenvectors, which are the right singular vectors of X, are the principal components: the orthogonal directions of maximum variance. The variance along the i-th component is σᵢ²/(n − 1) [Lay et al., 2016, §7.5].
PCA asks: in which few directions does my high-dimensional data actually vary? When a dataset with hundreds of variables lies mostly in a 2- or 3-dimensional subspace, PCA finds that subspace.
13. Fourier Series and the DFT as a Change of Basis
This topic ties the course together. The Fourier transform is not a separate subject. It is a change to an orthogonal basis of complex exponentials.
On functions over [−π, π], with inner product ⟨f, g⟩ = ∫ f(x) g(x) dx, the functions {1, cos x, sin x, cos 2x, sin 2x, ...} are mutually orthogonal. A Fourier series is just the "coordinates from dot products" formula of §10, applied in an infinite-dimensional space:
f(x) = a₀/2 + Σ [ aₖ cos(kx) + bₖ sin(kx) ] aₖ = (1/π) ∫ f(x) cos(kx) dx bₖ = (1/π) ∫ f(x) sin(kx) dx
Each coefficient is a projection, with the normalization coming from ⟨cos kx, cos kx⟩ = π.
For a discretely sampled signal of length n, the Discrete Fourier Transform is multiplication by the Fourier matrix:
(F)ⱼₖ = ωʲᵏ, ω = e^(−2πi/n), j, k = 0, 1, ..., n−1 F* F = n I (columns are orthogonal; F/√n is unitary) DFT: ŷ = F y Inverse: y = (1/n) F* ŷ
The columns of F are sampled sinusoids at n different frequencies. Taking the DFT means rewriting your signal in the frequency basis instead of the time basis [Strang, 2016, §9.3]. The Fast Fourier Transform (FFT) is a factorization of F into sparse matrices that reduces the cost from n² to roughly n log₂ n. It is one of the most consequential algorithms in science.
Why the Fourier basis works so well: complex exponentials are the eigenvectors of every shift-invariant (circulant) linear operator, including convolution and the derivative operator d/dx (since d/dx e^(ikx) = ik e^(ikx)). The Fourier transform diagonalizes convolution in the same sense that P diagonalizes A in §8. This is why convolution in time becomes multiplication in frequency.
Connections to Peptide Chemistry
This course is a mathematics course, but much of modern peptide and protein science depends on it. Here is where these ideas will reappear.
(a) Structural superposition and RMSD: the SVD at work
To compare two conformations of a peptide (a crystal structure and an MD snapshot, or two NMR models), you must find the rotation R that best superimposes them before computing the root-mean-square deviation. Translate both coordinate sets (N×3 matrices P, Q) so their centroids sit at the origin. Then form H = PᵀQ and take its SVD. The Kabsch algorithm [Kabsch, 1976, Acta Cryst. A32:922] gives the optimal rotation:
H = PᵀQ = U Σ Vᵀ d = sign( det(V Uᵀ) ) (guards against an improper rotation, i.e. a reflection) R = V · diag(1, 1, d) · Uᵀ
The determinant check matters chemically. A reflection would turn an L-amino acid into a D-amino acid, so it is not an allowed "superposition." This is a concrete case where det = −1 (orientation reversal, §4) has physical meaning.
(b) Normal-mode analysis: eigenvalues of the Hessian
Expanding a peptide's potential energy to second order around a minimum gives a quadratic form governed by the (mass-weighted) Hessian, a real symmetric 3N×3N matrix. By the spectral theorem, its eigenvectors are orthogonal normal modes, and its eigenvalues are ω² (squared vibrational frequencies). For a nonlinear molecule, six eigenvalues are zero: three translations and three rotations. These form the null space of the Hessian, which has dimension 6, so the number of genuine vibrations is 3N − 6 by rank–nullity. This is the origin of the 3N − 6 rule from CHEM101 and spectroscopy. The amide I band in peptide IR spectra (~1650 cm⁻¹, mainly C=O stretch) is one family of such modes, and its exact frequency reports on secondary structure.
(c) PCA of molecular dynamics: "essential dynamics"
An MD simulation of a 20-residue peptide produces thousands of frames in a space of hundreds of coordinates. After Kabsch-superposing all frames, PCA of the coordinate covariance matrix typically shows that a handful of principal components capture most of the large-scale motion, such as hinge bending or helix-coil transitions. This approach, known as essential dynamics [Amadei, Linssen & Berendsen, 1993, Proteins 17:412], is a standard tool for mapping conformational free-energy landscapes.
(d) Least-squares deconvolution of spectra
A peptide's far-UV circular dichroism spectrum is often modeled, to a first approximation, as a linear combination of reference spectra for α-helix, β-sheet, turn, and random coil. Estimating the secondary-structure fractions is then an overdetermined least-squares problem (§11): the measured spectrum at many wavelengths is b, and the reference spectra are the columns of A. The method is an approximation, because reference spectra depend on context, and constrained or regularized variants are common. The same idea underlies Beer–Lambert analysis of multicomponent mixtures from CHEM101L. Absorbances at multiple wavelengths give a linear system for the concentrations.
(e) Fourier transforms in NMR, IR, and crystallography
- FT-NMR records a time-domain free-induction decay and converts it to a frequency spectrum by FFT (§13). Multidimensional peptide NMR (COSY, TOCSY, NOESY), used for sequential assignment and distance restraints, applies FFTs along each time dimension.
- FTIR spectrometers record an interferogram and Fourier-transform it.
- X-ray crystallography measures the magnitudes of the Fourier coefficients of the electron density. The famous "phase problem" is the loss of their complex phases.
(f) Rotation matrices and peptide geometry
Building a peptide backbone from its (φ, ψ, ω) dihedral angles, the coordinates of a Ramachandran plot, is done by composing rotation matrices along the chain. Each rotation is an orthogonal matrix with det = +1, and their product (§2) places each successive atom. Non-commutativity is the reason the order of operations in such code matters.
(g) Hückel theory and the peptide bond
From CHEM202 you know that the peptide bond's partial double-bond character comes from delocalization over O=C–N. Simple Hückel theory models π systems through the eigenvalues of a symmetric matrix built from the connectivity: α on the diagonal and β for bonded neighbors. For benzene, the eigenvalues are α + 2β, α + β (twice), α − β (twice), and α − 2β, which explains its aromatic stability. The spectral theorem guarantees real orbital energies and orthogonal molecular orbitals.
Common Misconceptions to Watch Out For
- "AB = BA." Almost never true. Even when both products are defined, they usually differ, and AB can be zero when neither A nor B is zero.
- "det(A + B) = det A + det B." False. The determinant is multiplicative (det(AB) = det A det B), not additive. Also, det(cA) = cⁿ det A, not c·det A.
- "Row operations preserve the column space." False. They preserve the row space and the null space, but they change the column space. That is why column-space bases must use the pivot columns of the original matrix.
- "Rank is the number of nonzero rows of A." Only after row reduction. For A itself, count pivots in its echelon form.
- "Every matrix is diagonalizable." No. Shear matrices are the standard counterexample. Symmetric matrices are always diagonalizable, and orthogonally so.
- "Invertible ⇔ diagonalizable." These are independent properties. [1 1; 0 1] is invertible but not diagonalizable. [1 0; 0 0] is diagonalizable but not invertible. Invertibility concerns whether 0 is an eigenvalue. Diagonalizability concerns whether there are enough eigenvectors.
- "f(x) = mx + b is a linear transformation." Not in linear algebra's sense when b ≠ 0, because it does not send 0 to 0. It is affine. Translations are not linear, which is why superposition (§Peptide a) centers coordinates first.
- "A set containing a line or plane is a subspace." Only if it passes through the origin. Always check that 0 ∈ H first.
- "Singular values are just the absolute values of eigenvalues." True only for normal matrices, which include symmetric ones. In general, singular values are the square roots of the eigenvalues of AᵀA, and they can differ greatly from |λ|. Non-square matrices have singular values but no eigenvalues.
- "Eigenvectors are unique." Any nonzero scalar multiple of an eigenvector is also one, and repeated eigenvalues can have whole planes of eigenvectors. Software may return different signs or scalings. A principal component pointing in the "opposite" direction is the same component.
- "Solving the normal equations is always fine." Mathematically yes, numerically not always. Forming AᵀA squares the condition number. Use QR or the SVD for ill-conditioned problems.
- "PCA works on raw data." PCA requires centering. Without it, the first component mostly points toward the mean. When variables have different units, also consider standardizing, using the correlation matrix instead of the covariance matrix.
- "Orthogonal matrix" means orthogonal columns. The term requires orthonormal columns (QᵀQ = I). A matrix with merely orthogonal, unnormalized columns is not called orthogonal. The terminology is historically unfortunate.
- "Computing A⁻¹ is the way to solve Ax = b." Conceptually yes. In computation, factor A (LU, QR, or Cholesky) and solve instead.
A Closing Thought
If you remember one idea from this course, make it this: a linear transformation is the same object regardless of the coordinates you use to describe it, and choosing the right coordinates is most of the work. Eigenvectors are the right coordinates for repeated application of a matrix. Orthonormal bases are the right coordinates for geometry. Singular vectors are the right coordinates for data. Fourier modes are the right coordinates for anything shift-invariant. When a peptide simulation, spectrum, or kinetic scheme looks hopelessly coupled, ask which basis would decouple it.
References: Lay, D. C., Lay, S. R., & McDonald, J. J. (2016). Linear Algebra and Its Applications, 5th ed. Pearson. Strang, G. (2016). Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press. Supplementary (not assigned): Kabsch, W. (1976). Acta Crystallographica A32, 922–923; Amadei, A., Linssen, A. B. M., & Berendsen, H. J. C. (1993). Proteins 17, 412–425. Section numbers refer to the editions listed. Verify against your own copy, as numbering can shift slightly between printings.