Undergraduate MATH301 ⭐ 3 Credits Semester 4

Linear Algebra and Its Applications

Fundamental linear algebra: vector spaces, linear transformations, matrix algebra, determinants, eigenvalues, eigenvectors, and inner product spaces. Applications to least-squares problems, principal component analysis, and Fourier analysis. Provides mathematical tools used in multivariate data analysis, NMR processing, and structural bioinformatics.

📅 Studied by Skipper on 2026-10-01

📖 Core Concepts

Claude Opus 5.5

MATH301 — Linear Algebra and Its Applications

Core Concepts

Personal study notes for Skipper. Primary sources: Lay, Lay & McDonald, Linear Algebra and Its Applications, 5th ed. (Pearson, 2016) and Strang, Introduction to Linear Algebra, 5th ed. (Wellesley-Cambridge, 2016).

0. What Is This Subject Really About?

Most first courses present linear algebra as "how to solve systems of equations with matrices." That is where the subject starts, but it is not what the subject is. Linear algebra studies linear structure: spaces where you can add things and scale them, and maps between those spaces that respect adding and scaling. Once you look for this structure, it shows up across the sciences. Examples include atomic coordinates of a peptide, absorbance spectra, quantum wavefunctions, stoichiometric coefficients, and gene expression profiles.

Calculus (MATH101/102) already contains the key idea. A derivative is the best linear approximation to a function near a point. Calculus works by approximating curved things locally with linear things. Linear algebra is the full theory of those linear things. Whenever you linearize a chemical system, you hand the problem to linear algebra. This includes expanding a potential energy surface to second order around a minimum, linearizing rate equations near a steady state, and treating small perturbations in spectroscopy.

"Linear algebra is a fantastic subject. ... The central problem is to solve linear equations, and the central objects are vectors and matrices. But the deeper goal is to understand the spaces they create."
— paraphrasing the framing of [Strang, 2016, Preface and Ch. 1]

There are three ways to look at a matrix, and fluency means moving between them easily:

  1. A table of numbers. This is the computational view: row operations and arithmetic.
  2. A collection of column vectors. This is the geometric view. Ax is a combination of the columns of A, with weights given by x.
  3. A function (linear transformation). This is the dynamic view. A matrix does something: it rotates, stretches, projects, or collapses space.

Strang's main teaching move is to replace "row times column" with "combination of columns" as the default picture of Ax [Strang, 2016, §1.3, §2.1]. Lay et al. make the same move by defining the matrix-vector product as a linear combination of columns from the beginning [Lay, Lay & McDonald, 2016, §1.4]. Adopt this view now. Much of what follows becomes obvious once you do.

1. Linear Systems, Row Reduction, and Echelon Form

A system of m linear equations in n unknowns can be written compactly as

A x = b,     A ∈ ℝ^(m×n),  x ∈ ℝ^n,  b ∈ ℝ^m

The row picture reads each equation as a hyperplane, and the solution is where all the hyperplanes intersect. The column picture asks: which combination of the columns of A produces b?

x₁·(col 1) + x₂·(col 2) + ... + xₙ·(col n) = b

The column picture is the one that generalizes. The question "does Ax = b have a solution?" becomes "is b in the span of the columns of A?" That is a geometric question about subspaces, and the rest of the course builds on it.

Row reduction (Gaussian elimination) uses three elementary row operations: swapping rows, scaling a row by a nonzero constant, and adding a multiple of one row to another. None of these changes the solution set. Reduction ends in echelon form, or, if pushed all the way, in the unique reduced row echelon form (RREF). In RREF, each pivot is 1, each pivot is the only nonzero entry in its column, and the pivots move strictly right as you go down [Lay et al., 2016, §1.2].

Reading the RREF gives you the whole story:

  • Pivot columns correspond to basic variables. Non-pivot columns correspond to free variables.
  • A row of the form [0 0 ... 0 | c] with c ≠ 0 means the system is inconsistent (no solution).
  • If the system is consistent and has at least one free variable, there are infinitely many solutions.
  • If it is consistent with no free variables, the solution is unique.

The general solution of a consistent system has a structure that is worth understanding properly:

x = x_p + x_h
    (particular solution)  +  (any solution of A x = 0)

You have seen this exact structure before. In MATH102, the general solution of a linear ODE was a particular solution plus the general homogeneous solution. This is not a coincidence. A linear differential operator is a linear transformation, and the theorem is the same theorem [Lay et al., 2016, §1.5].

Chemistry flashback: balancing a chemical equation is a homogeneous linear system. For CH₄ + O₂ → CO₂ + H₂O, write one equation per element (C, H, O) in the unknown coefficients (a, b, c, d):

C:  a        - c        = 0
H:  4a             - 2d = 0
O:       2b  - 2c  -  d = 0

Null space spanned by (a, b, c, d) = (1, 2, 1, 2)

The balanced equation is the null space of the element-composition matrix. If that null space were 2-dimensional, the "reaction" would really be two independent reactions superimposed. This is exactly how stoichiometric network analysis in metabolism works.

2. Matrix Multiplication Is Composition of Functions

The matrix product rule, (AB)ij = Σk aikbkj, looks arbitrary until you see why it is defined that way. If B represents the transformation x ↦ Bx and A represents y ↦ Ay, then AB is the single matrix that represents "first do B, then do A":

(AB) x = A (B x)

The rule exists to make composition work [Lay et al., 2016, §2.1]. Several consequences follow immediately:

  • Non-commutativity is natural. Rotating a molecule about x and then about z is generally not the same as doing it in the reverse order. Pick up a book and try it. Matrices inherit this non-commutativity because they represent actions.
  • Associativity is free, because composition of functions is always associative.
  • Shapes must match: (m×n)(n×p) = (m×p). The inner dimensions agree because B's output must be A's input.

Strang emphasizes four equivalent ways to compute AB [Strang, 2016, §2.4]:

  1. Entry by entry: row i of A dotted with column j of B.
  2. Column by column: A times each column of B. Each column of AB is a combination of the columns of A.
  3. Row by row: each row of AB is a combination of the rows of B.
  4. Sum of outer products: AB = Σk (column k of A)(row k of B).

The fourth view will seem like a curiosity at first. It turns out to be the most important for applications. It is exactly how the SVD expresses a matrix as a sum of rank-one pieces (§12).

3. The Inverse and the Invertible Matrix Theorem

A square matrix A is invertible if there exists A⁻¹ with AA⁻¹ = A⁻¹A = I. Geometrically, A is invertible when its transformation can be undone: no information is lost and no two inputs land on the same output.

One of the most satisfying results in the course is the Invertible Matrix Theorem [Lay et al., 2016, §2.3, extended in §§2.9, 3.2, 5.2, 7.4]. It states that for an n×n matrix A, a long list of apparently different conditions are all equivalent:

  • A is invertible.
  • A is row equivalent to In (n pivots).
  • Ax = 0 has only the trivial solution.
  • The columns of A are linearly independent.
  • The columns of A span ℝⁿ.
  • The map x ↦ Ax is one-to-one (equivalently, onto).
  • Ax = b has a unique solution for every b.
  • Aᵀ is invertible.
  • det A ≠ 0.
  • 0 is not an eigenvalue of A.
  • rank A = n, and Nul A = {0}.
  • A has n nonzero singular values.

Treat this theorem as a hub. Each new topic in the course adds another equivalent condition. When you face a question about one condition, you can check whichever condition is easiest. One caveat: the theorem applies only to square matrices. For a non-square matrix, "one-to-one" and "onto" come apart.

The useful algebraic rules are:

(AB)⁻¹ = B⁻¹ A⁻¹          (reverse order: socks then shoes, undo shoes then socks)
(Aᵀ)⁻¹ = (A⁻¹)ᵀ
(AB)ᵀ  = Bᵀ Aᵀ

2×2 case:  [a b; c d]⁻¹ = (1/(ad − bc)) · [d −b; −c a]

A practical warning: numerical scientists almost never compute A⁻¹ explicitly to solve Ax = b. They factor A (for example, A = LU) and back-substitute. This is faster and more stable. The inverse is mainly a conceptual tool [Strang, 2016, §2.6].

4. Determinants: The Volume-Scaling Factor

The cofactor expansion formula for determinants is usually taught first. It is the least illuminating way to understand them.

det A = Σⱼ (−1)^(i+j) a_ij · det(A_ij)        (expansion along row i)

The geometric meaning is this: |det A| is the factor by which the transformation A scales n-dimensional volume. The sign of det A tells you whether orientation is preserved (+) or reversed (−). A unit square mapped by a 2×2 matrix becomes a parallelogram with area |det A| [Lay et al., 2016, §3.3].

With that picture, the main properties become intuitive:

  • det A = 0 ⇔ A is singular. The transformation squashes space into a lower dimension, giving zero volume, so information is lost and cannot be recovered.
  • det(AB) = det A · det B. Volume scalings compose by multiplication.
  • det(A⁻¹) = 1 / det A.
  • det(Aᵀ) = det A.
  • For a triangular matrix, det A is the product of the diagonal entries.
  • Swapping two rows flips the sign. Scaling a row by k scales det by k. Adding a multiple of one row to another leaves det unchanged.

The last property gives the efficient way to compute determinants: row reduce to triangular form while tracking swaps and scalings. Cofactor expansion takes on the order of n! operations. Elimination takes about n³/3.

Cramer's rule gives an explicit formula for each component of the solution of an invertible system:

xᵢ = det(Aᵢ(b)) / det(A)

where Aᵢ(b) is A with column i replaced by b.

Cramer's rule is beautiful and theoretically useful. For example, it shows that the solution depends continuously and rationally on the entries of A. Computationally, however, it is hopeless for large n [Lay et al., 2016, §3.3].

Connection to calculus: the Jacobian determinant in a change of variables (Cartesian to spherical, ∫∫∫ r² sin θ dr dθ dφ) is exactly this volume-scaling factor, applied to the local linearization of a nonlinear map. You used this in PHYS202 and in hydrogen-atom orbital normalization in CHEM101 without knowing it came from linear algebra.

5. Vector Spaces, Subspaces, Span, Independence, Basis, Dimension

Here the subject becomes abstract. The abstraction pays off because one theory then covers many different settings.

A vector space is any set V with addition and scalar multiplication satisfying ten axioms: closure, associativity, commutativity, zero vector, additive inverses, distributive laws, and so on [Lay et al., 2016, §4.1]. Examples:

  • ℝⁿ, the familiar one.
  • The polynomials of degree ≤ n.
  • The continuous functions on [a, b].
  • The solutions of a linear homogeneous ODE.
  • The set of m×n matrices.
  • Quantum wavefunctions. The superposition principle is exactly the statement that they form a vector space.

A subspace H ⊆ V is a subset that is itself a vector space. To check, verify three things: 0 ∈ H, H is closed under addition, and H is closed under scalar multiplication. Geometrically, subspaces of ℝ³ are {0}, lines through the origin, planes through the origin, and ℝ³ itself. A line that misses the origin is not a subspace.

Four definitions form the core vocabulary:

Span{v₁,...,vₚ}      = { c₁v₁ + ... + cₚvₚ : cᵢ ∈ ℝ }

Linearly independent  ⇔  c₁v₁ + ... + cₚvₚ = 0  forces all cᵢ = 0

Basis of V            = linearly independent set that spans V

dim V                 = number of vectors in any basis of V

A useful way to think about these: a spanning set has enough vectors to reach everything, and an independent set has no redundant vectors. A basis does both, so it is exactly the right size. The non-obvious theorem is that every basis of a given space has the same number of elements [Lay et al., 2016, §4.5]. That is why "dimension" is well defined. It is a property of the space, not of the coordinates you choose.

Once you fix a basis ℬ = {b₁, ..., bₙ}, every vector has unique coordinates [x]ℬ, and V behaves exactly like ℝⁿ (V is isomorphic to ℝⁿ). This is why a 4-dimensional polynomial space can be handled entirely with 4-component column vectors.

Analogy: a basis is like a set of reference reagents. Every mixture in the space can be made from them in exactly one way, none can be made from the others, and the number of reference reagents you need is fixed by the system, not by which ones you happen to choose.

6. The Four Fundamental Subspaces and the Rank–Nullity Theorem

Strang calls this "the Fundamental Theorem of Linear Algebra" and organizes much of his book around it [Strang, 2016, §§3.5, 4.1]. Every m×n matrix A of rank r has four associated subspaces:

Subspace          Lives in    Dimension    Definition
----------------  ----------  -----------  ------------------------------
Column space C(A)    ℝ^m         r          all A x
Null space   N(A)    ℝ^n         n − r      all x with A x = 0
Row space    C(Aᵀ)   ℝ^n         r          all Aᵀ y
Left null    N(Aᵀ)   ℝ^m         m − r      all y with Aᵀ y = 0

Two striking facts:

  1. Row rank = column rank. The number of independent rows always equals the number of independent columns, even for a 3×1000 matrix. This is far from obvious.
  2. Orthogonal complements. The row space and null space are orthogonal complements in ℝⁿ. The column space and left null space are orthogonal complements in ℝᵐ. The reason is direct: Ax = 0 means x is perpendicular to every row of A.

The Rank–Nullity Theorem is the dimensional accounting for this picture:

rank(A) + dim N(A) = n          (number of columns)

Interpretation: the n input dimensions split into those that survive the transformation (rank) and those that are annihilated (nullity). Nothing is created or destroyed. It works like a conservation law for dimensions.

How to find bases in practice [Lay et al., 2016, §§4.2–4.6]:

  • Column space: take the pivot columns of the original A, not of the RREF.
  • Row space: take the nonzero rows of the RREF. Row operations preserve the row space.
  • Null space: solve Ax = 0 from the RREF. Each free variable contributes one basis vector.

7. Linear Transformations, Kernel, Image, and Change of Basis

A map T: V → W is linear if

T(u + v) = T(u) + T(v)       and       T(c u) = c T(u)

Every linear map ℝⁿ → ℝᵐ is multiplication by a unique matrix, the standard matrix:

A = [ T(e₁)  T(e₂)  ...  T(eₙ) ]

This is a remarkable compression. To know T everywhere, you only need to know what it does to the n basis vectors [Lay et al., 2016, §1.9]. Linearity fills in the rest.

The kernel (ker T = {x: T(x) = 0}) is the null space. The image (range) is the column space. T is one-to-one ⇔ ker T = {0}. This test is very useful: to check whether distinct inputs give distinct outputs, you only need to check what maps to zero.

Change of basis. The same linear transformation looks different in different coordinate systems. If P has the new basis vectors as its columns, the matrix of T in the new basis is

A' = P⁻¹ A P          (A and A' are "similar" matrices)

Similar matrices share everything intrinsic to the transformation: determinant, trace, rank, eigenvalues, and characteristic polynomial. Much of advanced linear algebra is the search for the best basis, the one in which a transformation looks simplest. Diagonalization, the spectral theorem, the SVD, and the Fourier transform are all answers to that question, each for a different kind of problem.

The theme to carry through the rest of the course: choose coordinates adapted to the problem, and hard problems become easy.

This mirrors a habit from physical chemistry. You choose center-of-mass and internal coordinates for a diatomic, or spherical coordinates for the hydrogen atom, because the problem separates in those coordinates.

8. Eigenvalues, Eigenvectors, and Diagonalization

An eigenvector of A is a nonzero vector whose direction is unchanged by A. It is only stretched, by a factor λ called the eigenvalue:

A v = λ v,     v ≠ 0

To find eigenvalues, note that (A − λI)v = 0 must have a nontrivial solution, so A − λI must be singular:

det(A − λI) = 0              ← the characteristic polynomial (degree n)

For each eigenvalue λ, the eigenvectors (together with 0) form the eigenspace N(A − λI).

Two useful checks, valid for any square matrix:

trace(A) = Σ λᵢ           det(A) = Π λᵢ         (counted with multiplicity, including complex λ)

Diagonalization. If A has n linearly independent eigenvectors, collect them as the columns of P, and put the eigenvalues on the diagonal of D. Then

A = P D P⁻¹       ⇒      Aᵏ = P Dᵏ P⁻¹

In the eigenbasis, A acts by scaling each coordinate independently. Coupled problems become uncoupled ones. Matrix powers, matrix exponentials, and linear ODE systems all become easy:

dx/dt = A x       ⇒      x(t) = Σ cᵢ e^(λᵢ t) vᵢ

Direct connection to CHEM102 kinetics: a network of first-order reactions (A ⇌ B ⇌ C) is exactly dc/dt = Kc, with K the rate-constant matrix. The eigenvalues of K are the observable relaxation rates, and the eigenvectors are the "normal modes" of concentration change. One eigenvalue is zero, and its eigenvector is the equilibrium state. This is the reason relaxation kinetics in temperature-jump experiments shows a sum of exponentials.

When does diagonalization fail? When an eigenvalue's geometric multiplicity (dimension of its eigenspace) is less than its algebraic multiplicity (multiplicity as a root). The standard example is the shear [1 1; 0 1]. It has λ = 1 twice but only one independent eigenvector [Lay et al., 2016, §5.3]. Distinct eigenvalues always give independent eigenvectors, so a matrix with n distinct eigenvalues is always diagonalizable.

Complex eigenvalues are not a breakdown. They signal rotation. A real 2×2 matrix with eigenvalues a ± bi acts as a rotation combined with scaling by √(a² + b²) [Lay et al., 2016, §5.5]. In a dynamical system, complex eigenvalues produce oscillations.

9. Symmetric Matrices and the Spectral Theorem

When A = Aᵀ, the theory becomes much cleaner. The Spectral Theorem [Lay et al., 2016, §7.1; Strang, 2016, §6.4] states:

  1. All eigenvalues of a real symmetric matrix are real.
  2. Eigenvectors for distinct eigenvalues are orthogonal.
  3. A is always diagonalizable, and it is orthogonally diagonalizable:
A = Q Λ Qᵀ,        QᵀQ = I   (so Q⁻¹ = Qᵀ)

Spectral decomposition:   A = λ₁ q₁q₁ᵀ + λ₂ q₂q₂ᵀ + ... + λₙ qₙqₙᵀ

Every symmetric matrix is a weighted sum of projections onto perpendicular directions. Geometrically, a symmetric matrix stretches space along n mutually perpendicular axes, with no shear and no rotation.

This is why the theorem matters so much in physics and chemistry. Observables in quantum mechanics are Hermitian operators, the complex analogue of symmetric matrices. That they have real eigenvalues is why measurements give real numbers. That their eigenvectors are orthogonal is why distinct energy levels have orthogonal wavefunctions, a fact you used in PHYS202 and in CHEM101 orbital theory. The spectral theorem is the mathematical guarantee behind these facts.

Quadratic forms and definiteness. A symmetric A defines a quadratic form Q(x) = xᵀAx. Changing to the eigenbasis removes the cross terms:

xᵀ A x = λ₁ y₁² + λ₂ y₂² + ... + λₙ yₙ²     (y = Qᵀx)

A is positive definite if all λ > 0. The quadratic form is then a bowl, and a critical point with a positive definite Hessian is a local minimum. This is the second-derivative test from calculus generalized to n dimensions, and it is how computational chemists recognize minima (all positive Hessian eigenvalues) and transition states (exactly one negative eigenvalue) on a potential energy surface.

10. Orthogonality, Projections, and Gram–Schmidt

The dot product (more generally, an inner product) brings geometry into linear algebra: lengths, angles, and perpendicularity.

u · v = uᵀv = Σ uᵢvᵢ          ‖u‖ = √(u · u)          cos θ = (u · v) / (‖u‖ ‖v‖)

Orthogonal bases make calculation easy. If {u₁, ..., uₚ} is an orthogonal basis, the coordinates of any y come directly from dot products, with no system of equations to solve:

y = Σ  ( (y · uᵢ) / (uᵢ · uᵢ) ) uᵢ

Orthogonal projection onto a subspace W gives the point of W closest to y (the Best Approximation Theorem) [Lay et al., 2016, §6.3]. If the columns of A form a basis for W:

proj_W(y) = A (AᵀA)⁻¹ Aᵀ y         P = A(AᵀA)⁻¹Aᵀ,   P² = P,   Pᵀ = P

If W has an orthonormal basis (columns of Q):    P = Q Qᵀ

The error y − projW(y) is perpendicular to W. Remember this picture, because least squares is built on it: the shortest path from a point to a plane is the perpendicular.

Gram–Schmidt converts any basis {x₁, ..., xₚ} into an orthogonal one. Each new vector is made orthogonal to the previous ones by subtracting its projections onto them:

v₁ = x₁
v₂ = x₂ − (x₂·v₁ / v₁·v₁) v₁
v₃ = x₃ − (x₃·v₁ / v₁·v₁) v₁ − (x₃·v₂ / v₂·v₂) v₂
 ⋮
Normalize: qᵢ = vᵢ / ‖vᵢ‖

Chemists have already met this procedure: the construction of orthogonal hybrid orbitals and of orthogonalized basis sets in quantum chemistry (Löwdin orthogonalization is a symmetric variant) rest on the same idea. In function spaces, applying Gram–Schmidt to 1, x, x², ... with the inner product ∫₋₁¹ f g dx produces the Legendre polynomials, the same polynomials that appear in the angular part of hydrogen-atom orbitals.

11. Least Squares, Normal Equations, and QR Factorization

Real data is noisy, so Ax = b usually has no solution when there are more equations than unknowns (an overdetermined system). Fitting a line to 20 calibration points is a typical case. The least-squares answer is to find x̂ minimizing ‖b − Ax‖². Geometrically, Ax̂ is the projection of b onto the column space C(A). The residual must be perpendicular to C(A), so Aᵀ(b − Ax̂) = 0. This gives the normal equations:

AᵀA x̂ = Aᵀ b         ⇒       x̂ = (AᵀA)⁻¹ Aᵀ b      (if columns of A independent)

The name comes from the fact that the residual is normal (perpendicular) to the column space [Lay et al., 2016, §6.5; Strang, 2016, §4.3].

Connection to MATH201: linear regression is least squares. For the model y = β₀ + β₁x, the design matrix has a column of ones and a column of x-values, and the normal equations reproduce the familiar slope and intercept formulas. Under Gaussian noise, the least-squares estimate is also the maximum-likelihood estimate. So the geometry (orthogonal projection) and the statistics (MLE) lead to the same estimator, as you saw in MATH201.

QR factorization. Applying Gram–Schmidt to the columns of A gives A = QR, with Q having orthonormal columns and R upper triangular and invertible. Then:

A = QR         ⇒       R x̂ = Qᵀ b         (solve by back-substitution)

Why bother, when the normal equations already work? Because forming AᵀA squares the condition number: κ(AᵀA) = κ(A)² in the 2-norm. If A is even moderately ill-conditioned, for example a polynomial fit with nearly collinear columns, solving the normal equations can lose many digits of precision. QR avoids forming AᵀA and is the standard approach in numerical practice [Strang, 2016, §4.4]. (Production codes use Householder reflections or modified Gram–Schmidt instead of classical Gram–Schmidt, for further stability.)

12. The Singular Value Decomposition (SVD), Compression, and PCA

Strang has called the SVD the climax of linear algebra, and it deserves the description. Diagonalization needs a square matrix and may fail. The spectral theorem needs symmetry. The SVD works for every matrix, of any shape and any rank:

A = U Σ Vᵀ

A: m×n     U: m×m orthogonal     Σ: m×n diagonal (σ₁ ≥ σ₂ ≥ ... ≥ σᵣ > 0)     V: n×n orthogonal

A = σ₁ u₁v₁ᵀ + σ₂ u₂v₂ᵀ + ... + σᵣ uᵣvᵣᵀ

Geometric meaning: every linear transformation, however complicated, does three things in sequence. It rotates (and possibly reflects) by Vᵀ, stretches along perpendicular axes by Σ, then rotates (and possibly reflects) by U. The unit sphere is always mapped to an ellipsoid, with semi-axes σᵢuᵢ [Lay et al., 2016, §7.4].

How it connects to the spectral theorem: AᵀA is symmetric and positive semidefinite, so it can be orthogonally diagonalized. Its eigenvectors are the vᵢ, and its eigenvalues are σᵢ². Similarly, AAᵀ has eigenvectors uᵢ. The SVD also provides orthonormal bases for all four fundamental subspaces at once:

  • v₁…vᵣ span the row space, and vᵣ₊₁…vₙ span the null space.
  • u₁…uᵣ span the column space, and uᵣ₊₁…uₘ span the left null space.

So the SVD is a computational form of the Fundamental Theorem (§6).

Low-rank approximation (Eckart–Young theorem). Truncating the sum after k terms gives the best rank-k approximation to A, in both the 2-norm and the Frobenius norm:

Aₖ = σ₁u₁v₁ᵀ + ... + σₖuₖvₖᵀ         ‖A − Aₖ‖₂ = σₖ₊₁

An image stored as a 1000×1000 matrix needs 10⁶ numbers. If the singular values decay quickly, keeping k = 50 terms needs only 50 × (1000 + 1000 + 1) ≈ 10⁵ numbers, with little visible loss. This is data compression.

Principal Component Analysis (PCA). Arrange data as an n×p matrix X, with rows as observations and columns as variables, and center each column by subtracting its mean. The sample covariance matrix is S = XᵀX/(n − 1). Its eigenvectors, which are the right singular vectors of X, are the principal components: the orthogonal directions of maximum variance. The variance along the i-th component is σᵢ²/(n − 1) [Lay et al., 2016, §7.5].

PCA asks: in which few directions does my high-dimensional data actually vary? When a dataset with hundreds of variables lies mostly in a 2- or 3-dimensional subspace, PCA finds that subspace.

13. Fourier Series and the DFT as a Change of Basis

This topic ties the course together. The Fourier transform is not a separate subject. It is a change to an orthogonal basis of complex exponentials.

On functions over [−π, π], with inner product ⟨f, g⟩ = ∫ f(x) g(x) dx, the functions {1, cos x, sin x, cos 2x, sin 2x, ...} are mutually orthogonal. A Fourier series is just the "coordinates from dot products" formula of §10, applied in an infinite-dimensional space:

f(x) = a₀/2 + Σ [ aₖ cos(kx) + bₖ sin(kx) ]

aₖ = (1/π) ∫ f(x) cos(kx) dx        bₖ = (1/π) ∫ f(x) sin(kx) dx

Each coefficient is a projection, with the normalization coming from ⟨cos kx, cos kx⟩ = π.

For a discretely sampled signal of length n, the Discrete Fourier Transform is multiplication by the Fourier matrix:

(F)ⱼₖ = ωʲᵏ,   ω = e^(−2πi/n),   j, k = 0, 1, ..., n−1

F* F = n I         (columns are orthogonal; F/√n is unitary)

DFT:   ŷ = F y              Inverse:   y = (1/n) F* ŷ

The columns of F are sampled sinusoids at n different frequencies. Taking the DFT means rewriting your signal in the frequency basis instead of the time basis [Strang, 2016, §9.3]. The Fast Fourier Transform (FFT) is a factorization of F into sparse matrices that reduces the cost from n² to roughly n log₂ n. It is one of the most consequential algorithms in science.

Why the Fourier basis works so well: complex exponentials are the eigenvectors of every shift-invariant (circulant) linear operator, including convolution and the derivative operator d/dx (since d/dx e^(ikx) = ik e^(ikx)). The Fourier transform diagonalizes convolution in the same sense that P diagonalizes A in §8. This is why convolution in time becomes multiplication in frequency.

Connections to Peptide Chemistry

This course is a mathematics course, but much of modern peptide and protein science depends on it. Here is where these ideas will reappear.

(a) Structural superposition and RMSD: the SVD at work

To compare two conformations of a peptide (a crystal structure and an MD snapshot, or two NMR models), you must find the rotation R that best superimposes them before computing the root-mean-square deviation. Translate both coordinate sets (N×3 matrices P, Q) so their centroids sit at the origin. Then form H = PᵀQ and take its SVD. The Kabsch algorithm [Kabsch, 1976, Acta Cryst. A32:922] gives the optimal rotation:

H = PᵀQ = U Σ Vᵀ
d = sign( det(V Uᵀ) )                 (guards against an improper rotation, i.e. a reflection)
R = V · diag(1, 1, d) · Uᵀ

The determinant check matters chemically. A reflection would turn an L-amino acid into a D-amino acid, so it is not an allowed "superposition." This is a concrete case where det = −1 (orientation reversal, §4) has physical meaning.

(b) Normal-mode analysis: eigenvalues of the Hessian

Expanding a peptide's potential energy to second order around a minimum gives a quadratic form governed by the (mass-weighted) Hessian, a real symmetric 3N×3N matrix. By the spectral theorem, its eigenvectors are orthogonal normal modes, and its eigenvalues are ω² (squared vibrational frequencies). For a nonlinear molecule, six eigenvalues are zero: three translations and three rotations. These form the null space of the Hessian, which has dimension 6, so the number of genuine vibrations is 3N − 6 by rank–nullity. This is the origin of the 3N − 6 rule from CHEM101 and spectroscopy. The amide I band in peptide IR spectra (~1650 cm⁻¹, mainly C=O stretch) is one family of such modes, and its exact frequency reports on secondary structure.

(c) PCA of molecular dynamics: "essential dynamics"

An MD simulation of a 20-residue peptide produces thousands of frames in a space of hundreds of coordinates. After Kabsch-superposing all frames, PCA of the coordinate covariance matrix typically shows that a handful of principal components capture most of the large-scale motion, such as hinge bending or helix-coil transitions. This approach, known as essential dynamics [Amadei, Linssen & Berendsen, 1993, Proteins 17:412], is a standard tool for mapping conformational free-energy landscapes.

(d) Least-squares deconvolution of spectra

A peptide's far-UV circular dichroism spectrum is often modeled, to a first approximation, as a linear combination of reference spectra for α-helix, β-sheet, turn, and random coil. Estimating the secondary-structure fractions is then an overdetermined least-squares problem (§11): the measured spectrum at many wavelengths is b, and the reference spectra are the columns of A. The method is an approximation, because reference spectra depend on context, and constrained or regularized variants are common. The same idea underlies Beer–Lambert analysis of multicomponent mixtures from CHEM101L. Absorbances at multiple wavelengths give a linear system for the concentrations.

(e) Fourier transforms in NMR, IR, and crystallography

  • FT-NMR records a time-domain free-induction decay and converts it to a frequency spectrum by FFT (§13). Multidimensional peptide NMR (COSY, TOCSY, NOESY), used for sequential assignment and distance restraints, applies FFTs along each time dimension.
  • FTIR spectrometers record an interferogram and Fourier-transform it.
  • X-ray crystallography measures the magnitudes of the Fourier coefficients of the electron density. The famous "phase problem" is the loss of their complex phases.

(f) Rotation matrices and peptide geometry

Building a peptide backbone from its (φ, ψ, ω) dihedral angles, the coordinates of a Ramachandran plot, is done by composing rotation matrices along the chain. Each rotation is an orthogonal matrix with det = +1, and their product (§2) places each successive atom. Non-commutativity is the reason the order of operations in such code matters.

(g) Hückel theory and the peptide bond

From CHEM202 you know that the peptide bond's partial double-bond character comes from delocalization over O=C–N. Simple Hückel theory models π systems through the eigenvalues of a symmetric matrix built from the connectivity: α on the diagonal and β for bonded neighbors. For benzene, the eigenvalues are α + 2β, α + β (twice), α − β (twice), and α − 2β, which explains its aromatic stability. The spectral theorem guarantees real orbital energies and orthogonal molecular orbitals.

Common Misconceptions to Watch Out For

  1. "AB = BA." Almost never true. Even when both products are defined, they usually differ, and AB can be zero when neither A nor B is zero.
  2. "det(A + B) = det A + det B." False. The determinant is multiplicative (det(AB) = det A det B), not additive. Also, det(cA) = cⁿ det A, not c·det A.
  3. "Row operations preserve the column space." False. They preserve the row space and the null space, but they change the column space. That is why column-space bases must use the pivot columns of the original matrix.
  4. "Rank is the number of nonzero rows of A." Only after row reduction. For A itself, count pivots in its echelon form.
  5. "Every matrix is diagonalizable." No. Shear matrices are the standard counterexample. Symmetric matrices are always diagonalizable, and orthogonally so.
  6. "Invertible ⇔ diagonalizable." These are independent properties. [1 1; 0 1] is invertible but not diagonalizable. [1 0; 0 0] is diagonalizable but not invertible. Invertibility concerns whether 0 is an eigenvalue. Diagonalizability concerns whether there are enough eigenvectors.
  7. "f(x) = mx + b is a linear transformation." Not in linear algebra's sense when b ≠ 0, because it does not send 0 to 0. It is affine. Translations are not linear, which is why superposition (§Peptide a) centers coordinates first.
  8. "A set containing a line or plane is a subspace." Only if it passes through the origin. Always check that 0 ∈ H first.
  9. "Singular values are just the absolute values of eigenvalues." True only for normal matrices, which include symmetric ones. In general, singular values are the square roots of the eigenvalues of AᵀA, and they can differ greatly from |λ|. Non-square matrices have singular values but no eigenvalues.
  10. "Eigenvectors are unique." Any nonzero scalar multiple of an eigenvector is also one, and repeated eigenvalues can have whole planes of eigenvectors. Software may return different signs or scalings. A principal component pointing in the "opposite" direction is the same component.
  11. "Solving the normal equations is always fine." Mathematically yes, numerically not always. Forming AᵀA squares the condition number. Use QR or the SVD for ill-conditioned problems.
  12. "PCA works on raw data." PCA requires centering. Without it, the first component mostly points toward the mean. When variables have different units, also consider standardizing, using the correlation matrix instead of the covariance matrix.
  13. "Orthogonal matrix" means orthogonal columns. The term requires orthonormal columns (QᵀQ = I). A matrix with merely orthogonal, unnormalized columns is not called orthogonal. The terminology is historically unfortunate.
  14. "Computing A⁻¹ is the way to solve Ax = b." Conceptually yes. In computation, factor A (LU, QR, or Cholesky) and solve instead.

A Closing Thought

If you remember one idea from this course, make it this: a linear transformation is the same object regardless of the coordinates you use to describe it, and choosing the right coordinates is most of the work. Eigenvectors are the right coordinates for repeated application of a matrix. Orthonormal bases are the right coordinates for geometry. Singular vectors are the right coordinates for data. Fourier modes are the right coordinates for anything shift-invariant. When a peptide simulation, spectrum, or kinetic scheme looks hopelessly coupled, ask which basis would decouple it.

References: Lay, D. C., Lay, S. R., & McDonald, J. J. (2016). Linear Algebra and Its Applications, 5th ed. Pearson. Strang, G. (2016). Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press. Supplementary (not assigned): Kabsch, W. (1976). Acta Crystallographica A32, 922–923; Amadei, A., Linssen, A. B. M., & Berendsen, H. J. C. (1993). Proteins 17, 412–425. Section numbers refer to the editions listed. Verify against your own copy, as numbering can shift slightly between printings.

⚙️ Mechanisms & Applications

GPT-6 Astra

Mechanisms & Applications: Learning to Think in Directions, Constraints, and Modes

Skipper, the central move in linear algebra is to stop treating every equation as an isolated relationship. Instead, ask: What transformation acts on the whole system? Which directions can it distinguish? Which quantities survive unchanged? This viewpoint connects balancing reactions, estimating peptide concentrations, analyzing gene-expression data, and reconstructing medical images. The same mathematics appears because each problem involves combinations of signals, constraints, or coordinated changes.

A boundary worth keeping clear: linear algebra introduces mathematical and computational techniques, not new wet-laboratory procedures. Spectroscopy, chromatography, NMR, and MRI appear below as experimental settings in which those techniques operate. Their physical measurement mechanisms and the mathematical analysis of their data are related, but they are not the same thing.

1. From Equations to a Mechanism: Row Reduction

A system of linear equations can be written as Ax = b. Read this in three complementary ways:

  • Equation view: each row specifies a constraint on the unknowns.
  • Column view: b must be assembled as a weighted combination of the columns of A.
  • Transformation view: A maps an unknown input x to an observed output b.
A = [a₁ a₂ ... aₙ]

Ax = x₁a₁ + x₂a₂ + ... + xₙaₙ = b

The column view is particularly useful in analytical chemistry. Each column might be the spectrum of a reference compound; the coefficients in x are the amounts needed to reconstruct a mixture spectrum. A solution exists exactly when b belongs to the column space of A. [1, Chs. 1–2]

Mechanism of Gaussian elimination:

  1. Write the augmented matrix [A | b].
  2. Choose a nonzero pivot, swapping rows if necessary.
  3. Use multiples of the pivot row to eliminate entries below the pivot.
  4. Continue through the remaining rows and columns to obtain echelon form.
  5. Back-substitute, or normalize pivots and eliminate above them to obtain reduced row echelon form, RREF.
  6. Identify pivot variables, free variables, and any inconsistent row.
Example: two normalized measurements of a two-component mixture

c₁ +  c₂ = 3
c₁ + 2c₂ = 5

[1  1 | 3]    R₂ ← R₂ − R₁    [1  1 | 3]
[1  2 | 5]  ───────────────→   [0  1 | 2]

R₁ ← R₁ − R₂

[1  0 | 1]
[0  1 | 2]

Therefore c₁ = 1 and c₂ = 2 in the chosen concentration units.

Row operations preserve the solution set because each operation is reversible: we are replacing the constraints with equivalent constraints, not changing the experiment. An inconsistent row, such as [0 0 | 1], means the model demands that zero equal one. A free variable means that the constraints do not uniquely determine the unknowns.

Exact mathematics versus laboratory data: RREF is excellent for understanding structure, but experimental measurements contain noise. Whether a small computed pivot should count as zero depends on scale and uncertainty. Numerical implementations use pivoting, and difficult rank decisions are better examined with singular values. An overdetermined noisy system will usually require least squares rather than an exact solution. [3]

2. Matrix Operations Describe How Processes Combine

Matrix addition combines linear responses acting on the same input. Scalar multiplication rescales a response. Matrix multiplication represents composition: one transformation occurs after another.

x ──B──→ z ──A──→ y

z = Bx
y = Az = ABx

If B is p × n and A is m × p, then AB is m × n.

Order matters. If B represents recovery of different peptides during extraction and A converts recovered concentrations into detector signals, then the overall map is AB. Reversing the order usually has no physical meaning and may not even be dimensionally possible. Matrix multiplication is generally noncommutative because physical operations are generally order-dependent. [1, Ch. 2]

The transpose reverses the input–output roles in an important inner-product identity:

(Ax)ᵀy = xᵀ(Aᵀy)

This does not make Aᵀ an inverse. Rather, it explains why the transpose appears when propagating a measurement-space residual back into parameter space, as in least squares.

For a square matrix, invertibility means that no information about x is lost:

For A ∈ ℝⁿˣⁿ, the following are equivalent:

A is invertible
rank(A) = n
null(A) = {0}
the columns of A are linearly independent
Ax = b has exactly one solution for every b ∈ ℝⁿ
det(A) ≠ 0
0 is not an eigenvalue of A

An inverse is conceptually valuable, but explicitly computing A−1 is usually not the best way to solve Ax = b. Factorizations such as LU or QR are generally preferable, especially when many datasets share the same measurement matrix. [1, Ch. 2; 3]

3. Determinants: Volume, Dependence, and the Limits of a Classical Tool

The determinant measures signed volume scaling. If a square transformation flattens an n-dimensional region into a lower-dimensional one, its determinant is zero. This is the geometric reason that a zero determinant signals loss of invertibility.

For a 2 × 2 matrix:

det([a b; c d]) = ad − bc

Cofactor expansion along row i:

det(A) = Σⱼ aᵢⱼ Cᵢⱼ
Cᵢⱼ = (−1)ⁱ⁺ʲ det(Mᵢⱼ)

Mᵢⱼ is obtained by deleting row i and column j.
  • Swapping two rows reverses the determinant’s sign.
  • Multiplying a row by α multiplies the determinant by α.
  • Adding a multiple of one row to another leaves it unchanged.
  • For a triangular matrix, the determinant is the product of its diagonal entries.

Cramer’s rule expresses each component of a uniquely determined solution as a ratio of determinants:

Ax = b, det(A) ≠ 0

xⱼ = det(Aⱼ) / det(A)

Aⱼ is A with column j replaced by b.

Gabriel Cramer published this rule in 1750. Its importance is conceptual: it provides an explicit algebraic connection between solvability and determinants. It is not a practical default for large scientific computations. Likewise, recursive cofactor expansion is much less efficient than elimination for general large matrices. [1, Ch. 3; 4]

A counterintuitive warning: a small determinant does not, by itself, prove numerical instability. For A = 0.01I in 100 dimensions, det(A) = 10−200, yet the matrix has 2-norm condition number 1. The determinant mixes scale with geometry; singular-value ratios give a better measure of sensitivity.

4. Subspaces and Conservation Laws in Biochemistry

A vector space is a collection closed under vector addition and scalar multiplication, with the usual algebraic rules. Its vectors need not be arrows: they can be spectra, polynomials, time courses, or concentration changes. A basis is an independent set that spans the space; dimension counts the independent coordinates required to describe it.

A subtle chemical distinction: all real concentration vectors form a vector space, but physically admissible concentration vectors, with nonnegative entries, do not. Multiplication by −1 generally leaves the physically allowed set. Linear algebra describes the surrounding geometry; chemistry adds positivity, kinetics, and thermodynamics.

Consider the enzyme mechanism already familiar from biology and organic chemistry:

Reaction 1: E + S → ES
Reaction 2: ES → E + S
Reaction 3: ES → E + P

Species order: E, S, ES, P

              reaction
             1   2   3
S = E      [−1   1   1]
    S      [−1   1   0]
    ES     [ 1  −1  −1]
    P      [ 0   0   1]

dc/dt = S v(c)

Here S is the stoichiometric matrix and v(c) contains reaction rates. Under mass action, for example, v₁ = k₁[E][S]. Thus the stoichiometric map is linear in the rates, while the rates themselves may be nonlinear functions of concentration. Writing a nonlinear biochemical model with matrices does not make its dynamics linear. [5]

Mechanism for finding conserved quantities:

  1. Construct S from the reaction stoichiometries.
  2. Solve Sᵀℓ = 0.
  3. Each solution ℓ defines a conserved weighted total, provided the modeled system has no omitted inputs or losses.
If Sᵀℓ = 0, then

d(ℓᵀc)/dt = ℓᵀS v(c) = 0.

For this enzyme mechanism:

ℓ₁ = (1, 0, 1, 0)ᵀ  →  [E] + [ES] is conserved.
ℓ₂ = (0, 1, 1, 1)ᵀ  →  [S] + [ES] + [P] is conserved.

The four fundamental subspaces now have concrete meanings:

  • Column space of S: concentration-change directions permitted by the reactions.
  • Null space of S: rate combinations producing zero net concentration change.
  • Left null space, null(Sᵀ): conserved totals.
  • Row space of S: the independent linear combinations of reaction rates that affect species balances.
For S with m species, n reactions, and rank r:

dim null(S)  = n − r
dim null(Sᵀ) = m − r

These are rank–nullity relationships. For the enzyme example, r = 2. There are two independent conserved totals and one independent zero-net-change flux direction, proportional to (1, 1, 0)ᵀ.

In metabolic flux analysis, Sv = 0 expresses steady-state balances for modeled internal metabolites. But stoichiometric feasibility is not enough: flux bounds, exchange reactions, irreversibility, and sometimes thermodynamic constraints must also be supplied. In the closed enzyme system above, a steady state cannot maintain positive net product accumulation; doing so requires an open-system description. [5]

5. Orthogonality, Projection, and Least Squares

When measurements disagree slightly, an exact solution may not exist. Least squares asks for the model prediction closest to the observed data.

Given A ∈ ℝᵐˣⁿ and b ∈ ℝᵐ:

Choose x̂ to minimize ||b − Ax||₂².

Residual: r = b − Ax̂
Fitted data: b̂ = Ax̂

The geometric mechanism is projection. The fitted vector lies in col(A), and the residual is perpendicular to every direction the model can produce. If it were not perpendicular, moving the fit in one of those directions could reduce the error. [1, Ch. 6; 2, Ch. 4]

Aᵀ(b − Ax̂) = 0

Therefore:

AᵀA x̂ = Aᵀb             normal equations

If A has independent columns:

x̂ = (AᵀA)⁻¹Aᵀb

Example: a local calibration model. Suppose measurements at predictor values t = 0, 1, 2 give responses 1, 2, 2. Fit y ≈ β₀ + β₁t.

A = [1 0]       b = [1]
    [1 1]           [2]
    [1 2]           [2]

AᵀA = [3 3]    Aᵀb = [ 5]
      [3 5]           [ 6]

β̂₀ = 7/6
β̂₁ = 1/2

b̂ = (7/6, 5/3, 13/6)ᵀ
r  = (−1/6, 1/3, −1/6)ᵀ

Check: Aᵀr = 0.

The residual is not zero, but its component along every fitted direction is zero. This is a precise sense in which the fit has extracted everything available within the chosen linear model.

Your statistics course adds an important qualification: ordinary least squares corresponds to equal weighting of residuals. If measurement errors have known positive-definite covariance Ω, generalized least squares instead minimizes:

(b − Ax)ᵀΩ⁻¹(b − Ax)

For independent errors with unequal variances, this reduces to weighted least squares. None of these methods rescues a chemically wrong model: a straight line fitted to detector saturation remains a wrong measurement model.

6. Gram–Schmidt and QR: Building a Stable Coordinate System

Projection becomes especially simple with orthonormal basis vectors: each coordinate is just an inner product. Gram–Schmidt constructs such a basis by removing from each new vector the components already represented.

For independent columns a₁, ..., aₙ:

u₁ = a₁
q₁ = u₁ / ||u₁||

For j = 2, ..., n:

uⱼ = aⱼ − Σᵢ₌₁ʲ⁻¹ (qᵢᵀaⱼ)qᵢ
qⱼ = uⱼ / ||uⱼ||

Result:

A = QR
QᵀQ = I
R is upper triangular.

The subtraction step is the mechanism: remove everything already explained, then normalize what remains. If the remainder is exactly zero, the new vector contributes no new direction.

For a tall matrix with independent columns, QR solves least squares without explicitly forming AᵀA:

A = QR

R x̂ = Qᵀb

Solve the triangular system by back-substitution.

This matters because forming the normal-equation matrix squares the 2-norm condition number:

κ₂(AᵀA) = κ₂(A)²

Classical Gram–Schmidt can lose orthogonality in floating-point arithmetic when columns are nearly dependent. Modified Gram–Schmidt, sometimes with reorthogonalization, improves the computation; Householder transformations are a standard robust method for QR. The exact geometry is the same, but the numerical route matters. [3]

Pharma connection: calibration spectra for a peptide and a closely related impurity may be nearly parallel. Orthogonalization reveals how little genuinely new information the impurity spectrum supplies. Measuring more points along effectively the same spectral direction does not automatically solve the identification problem.

7. Eigenvectors Reveal the Natural Modes of Dynamics

An eigenvector is a direction that a transformation does not mix with other directions:

Av = λv,   v ≠ 0

Eigenvalues satisfy:

det(A − λI) = 0.

For a linear differential equation, eigenvectors identify independently evolving modes. This connects directly to your differential equations and chemical kinetics courses. [1, Ch. 5; 2, Ch. 6]

Mechanism for solving a diagonalizable linear dynamical system:

  1. Write dc/dt = Kc.
  2. Find the eigenvalues and independent eigenvectors of K.
  3. Assemble K = VΛV−1.
  4. Express the initial condition in eigenvector coordinates: a = V−1c(0).
  5. Evolve each coordinate separately, then transform back.
c(t) = V e^(Λt) V⁻¹c(0)
     = Σⱼ aⱼ e^(λⱼt)vⱼ

For reversible first-order interconversion:

A ⇌ B

d/dt [A] = [−k_f   k_r] [A]
     [B]   [ k_f  −k_r] [B]

Eigenvalues:

λ₁ = 0
λ₂ = −(k_f + k_r)

Corresponding eigenvector directions:

v₁ = (k_r, k_f)ᵀ
v₂ = (1, −1)ᵀ

The zero mode describes the equilibrium composition, whose overall scale is set by the conserved total [A] + [B]. The second mode transfers material between A and B while preserving that total. Its relaxation time is 1/(kf + kr).

The beautiful point: species coordinates describe what we measure; eigenvector coordinates describe how the system relaxes.

Not every matrix has enough eigenvectors to diagonalize. Moreover, a linear kinetic model is not automatically valid for enzyme saturation or nonlinear binding. Near an equilibrium c*, a nonlinear system dc/dt = f(c) can instead be approximated using its Jacobian:

δc = c − c*

d(δc)/dt ≈ J(c*)δc

Jᵢⱼ = ∂fᵢ/∂cⱼ evaluated at c*.

Negative real parts of all relevant eigenvalues imply local asymptotic stability. A zero eigenvalue needs further analysis; it may reflect a conservation law rather than instability.

8. Symmetric Matrices and Molecular Vibrations

Real symmetric matrices have an exceptional property: they possess an orthonormal basis of eigenvectors and real eigenvalues.

A = Aᵀ  ⇒  A = QΛQᵀ, with QᵀQ = I.

This is the spectral theorem. It means that, after an orthogonal change of coordinates, a symmetric transformation acts as independent stretches or compressions along mutually perpendicular directions. [1, Ch. 7]

In a local harmonic approximation to a molecular potential energy surface:

U(r₀ + δr) ≈ U(r₀) + ½ δrᵀHδr

Hᵢⱼ = ∂²U / (∂rᵢ ∂rⱼ) at the stationary geometry r₀

M δr̈ = −Hδr

H is the Hessian, and M is the diagonal mass matrix. Introduce mass-weighted coordinates q = M1/2δr:

q̈ = −Dq

D = M⁻¹/² H M⁻¹/²

Dvⱼ = ωⱼ²vⱼ

The eigenvectors are normal-mode directions, and positive eigenvalues give squared angular frequencies. At a stable isolated nonlinear molecular geometry, the idealized translational and rotational motions account for six zero modes; the remaining 3N − 6 modes are vibrational. A linear molecule has 3N − 5 vibrational modes.

This links linear algebra to infrared and Raman interpretation, conformational analysis, and computational assessment of candidate peptide geometries. The harmonic approximation is local: a soft mode is not, by itself, a complete prediction of a large conformational transition. [6]

9. SVD: Separating What an Experiment Can See from What It Cannot

Eigenvalue decomposition concerns a transformation acting within a space. Singular value decomposition applies to any rectangular measurement matrix and provides coordinated orthonormal directions in its input and output spaces. [1, Ch. 7; 2, Ch. 7]

A = UΣVᵀ

A vⱼ = σⱼuⱼ

σ₁ ≥ σ₂ ≥ ... ≥ σᵣ > 0

r = rank(A)

The mechanism has three stages:

  1. Vᵀ: express the unknown input in the experiment’s natural input directions.
  2. Σ: scale each direction by its singular value.
  3. U: express the result in measurement coordinates.

A large singular value means strong measurement response to a particular input direction. A small singular value means weak response. A zero singular value means that direction is invisible to the experiment.

Minimum-norm least-squares solution:

x̂ = A⁺b
   = Σⱼ₌₁ʳ [(uⱼᵀb)/σⱼ] vⱼ

For full column rank:

κ₂(A) = σ₁ / σₙ

Dividing by a small σ amplifies measurement noise. This is why two nearly indistinguishable impurity spectra can produce wildly unstable separate concentration estimates even when their combined concentration is well determined.

Regularization reduces sensitivity by sacrificing some exact data fit. For example:

Ridge/Tikhonov problem:

minimize ||Ax − b||₂² + α||x||₂²,   α > 0

Solution:

x̂α = Σⱼ₌₁ʳ [σⱼ/(σⱼ² + α)] (uⱼᵀb)vⱼ

Compare σ/(σ² + α) with 1/σ: regularization suppresses weakly observed directions instead of magnifying them. Its price is bias. Choosing α requires scientific judgment, noise information, or validation—not simply a desire for a smoother answer. [3]

10. PCA and Compression: Dominant Variation Is Not Automatically Important Biology

Suppose X contains centered measurements from n samples and p features: metabolite abundances, peptide intensities, or gene-expression measurements. Each column has had its sample mean subtracted.

X = UΣVᵀ

Sample covariance:

C = XᵀX/(n − 1)
  = V [ΣᵀΣ/(n − 1)] Vᵀ

Principal directions: columns of V
Sample scores: XV = UΣ
Variance along component j: σⱼ²/(n − 1)

Mechanism of PCA:

  1. Define rows, features, units, and preprocessing explicitly.
  2. Center features; scale them only when the scientific question warrants it.
  3. Compute an SVD.
  4. Inspect explained variance, sample scores, and feature loadings together.
  5. Check whether apparent structure reflects biology, batch, instrument drift, or another nuisance variable.

The rank-k approximation

Xₖ = Σⱼ₌₁ᵏ σⱼuⱼvⱼᵀ

is optimal among matrices of rank at most k in Frobenius-norm error. This is the mathematical basis of SVD compression and many denoising procedures. But minimum reconstruction error is not the same as maximum diagnostic usefulness: a rare disease signal or low-level peptide impurity may occupy a low-variance direction. [2, Ch. 7]

In drug-development datasets, PCA can expose unusual batches and broad formulation differences. It does not establish causation or prove that a cluster has clinical meaning. For predictive modeling, centering, scaling, and PCA must be fitted using training data only and then applied to held-out data; otherwise evaluation can leak information.

Karl Pearson’s 1901 work described fitting lines and planes to multivariate observations; Harold Hotelling’s 1933 work developed principal components in statistical analysis. These contributions made coordinated variation a tractable object of study rather than a collection of separate correlations. [7, 8]

11. Experimental Workflows: Spectroscopy, Chromatography, and Peptide Quantification

UV–visible mixture analysis. Under additive Beer–Lambert behavior:

At wavelength λᵢ:

aᵢ = ℓ Σⱼ εⱼ(λᵢ)cⱼ + ηᵢ

In matrix form:

a = Ec + η

Eᵢⱼ = ℓ εⱼ(λᵢ)

Absorbance is dimensionless; for ℓ in cm and concentration in mol L−1, molar absorptivity has units L mol−1 cm−1. This dimensional check helps distinguish a chemically meaningful calibration matrix from an arbitrary table of numbers.

Mechanism of the analytical workflow:

  1. Measure blanks and authenticated reference compounds under matched solvent, pH, and instrumental conditions.
  2. Construct the reference-response matrix.
  3. Check rank and singular values: can the selected measurements distinguish the proposed components?
  4. Estimate concentrations with QR-based least squares or an appropriate constrained method.
  5. Inspect residuals and validate against independent mixtures and known additions.

In HPLC with diode-array detection, chromatography separates compounds in time while the detector records wavelength-dependent absorbance. Under suitable assumptions, overlapping signals can be modeled as mixtures of reference spectra. However, nearly identical spectra, unknown impurities, baseline drift, and retention-time variation can defeat simple unmixing. Nonnegative concentration constraints can prevent negative estimates, but they cannot create information absent from the measurements. [9]

For peptide and pharma applications, the practical lesson is experimental: improve the measurement matrix. Change the separation method, select more discriminating wavelengths, or add an orthogonal measurement such as mass spectrometry. A more informative experiment often outperforms a more elaborate fit.

12. Fourier Analysis: Changing from Time Coordinates to Frequency Coordinates

The discrete Fourier transform is a change of basis from sampled values to oscillatory components. In a unitary convention:

For samples x₀, ..., xₙ₋₁:

x̂ₖ = (1/√N) Σⱼ₌₀ᴺ⁻¹ xⱼ exp(−2πijk/N)

Inverse:

xⱼ = (1/√N) Σₖ₌₀ᴺ⁻¹ x̂ₖ exp(+2πijk/N)

Matrix form:

x̂ = Fx
F*F = I

F* denotes conjugate transpose.

Complex inner products require conjugation. Orthogonality then guarantees that the transform preserves squared norm: the signal has been re-expressed, not enlarged or diminished. Fourier series use the corresponding orthogonal-function idea for periodic functions rather than finite vectors. [2, Chs. 4 and 9]

NMR application: the measured free-induction decay can be modeled approximately as a sum of damped oscillations:

s(t) ≈ Σⱼ aⱼ exp(−t/T₂,ⱼ*) exp(iωⱼt) + noise.

The Fourier transform reveals frequency components associated with resonance frequencies. Decay, finite acquisition time, and processing affect linewidth and spectral shape. Thus Fourier analysis does not conjure a spectrum from nowhere: it expresses the measured oscillation in coordinates matched to its physical structure. [10]

  • For sampling interval Δt, the sampling frequency is 1/Δt.
  • For N samples, the DFT bin spacing is 1/(NΔt).
  • Frequencies outside the selected unaliased sampling interval fold into it unless appropriately excluded.
  • Zero-padding gives more closely spaced displayed frequency samples; it does not add new acquisition information or fundamentally resolve otherwise unresolved resonances.

MRI application: magnetic-field gradients encode position into signal phase. Under the standard simplified model, measurements sample spatial Fourier information about an effective image. Complete, suitably arranged samples permit inverse Fourier reconstruction; undersampling, coil sensitivities, motion, and other effects require more elaborate inverse models. [11]

Joseph Fourier’s 1822 work on heat conduction established the power of decomposing behavior into sinusoidal components. Cooley and Tukey’s 1965 paper popularized an efficient FFT algorithm, building on earlier ideas. The FFT is not a different transform: for suitable factorizations of N, it reduces computation from roughly O(N²) to O(N log N), making repeated spectral and imaging calculations practical. [12, 13]

13. Historical Perspective: From Elimination to Information Geometry

Linear algebra developed through overlapping computational, geometric, and physical traditions—not through a single discovery.

  • Elimination predates its modern name. Procedures recognizable as elimination appear in the ancient Chinese Nine Chapters on the Mathematical Art. Later European developments connected systematic equation-solving to astronomy, surveying, and numerical computation. “Gaussian elimination” should not be read as a claim that Gauss invented elimination from nothing. [4]
  • Least squares arose from imperfect observations. Legendre published the method in 1805; Gauss published his treatment in 1809 and claimed earlier use. The essential advance was a systematic way to reconcile noisy, mutually inconsistent measurements. [14]
  • SVD has nineteenth-century roots. Work by Beltrami and Jordan in the 1870s contributed to its early development. Reliable numerical algorithms later made it central to scientific computing and data analysis. [15]

Skipper’s working habit: for every new matrix model, ask five questions: What are its inputs and outputs? What does each column mean physically? Which directions are invisible? Which quantities are conserved? How does noise behave when I invert the map? Those questions turn linear algebra from symbol manipulation into a method for designing, interpreting, and challenging scientific experiments.

Sources

  1. Lay, D. C., Lay, S. R., and McDonald, J. J. Linear Algebra and Its Applications, 5th ed. Pearson, 2016. Chapters 1–7.
  2. Strang, G. Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press, 2016.
  3. Golub, G. H., and Van Loan, C. F. Matrix Computations, 4th ed. Johns Hopkins University Press, 2013.
  4. Katz, V. J. A History of Mathematics: An Introduction, 3rd ed. Addison-Wesley, 2009.
  5. Palsson, B. Ø. Systems Biology: Properties of Reconstructed Networks. Cambridge University Press, 2006.
  6. Wilson, E. B., Jr., Decius, J. C., and Cross, P. C. Molecular Vibrations: The Theory of Infrared and Raman Vibrational Spectra. McGraw-Hill, 1955.
  7. Pearson, K. “On Lines and Planes of Closest Fit to Systems of Points in Space.” Philosophical Magazine 2 (1901): 559–572. doi:10.1080/14786440109462720.
  8. Hotelling, H. “Analysis of a Complex of Statistical Variables into Principal Components.” Journal of Educational Psychology 24 (1933): 417–441 and 498–520.
  9. Skoog, D. A., Holler, F. J., and Crouch, S. R. Principles of Instrumental Analysis, 7th ed. Cengage Learning, 2018.
  10. Keeler, J. Understanding NMR Spectroscopy, 2nd ed. Wiley, 2010.
  11. Liang, Z.-P., and Lauterbur, P. C. Principles of Magnetic Resonance Imaging: A Signal Processing Perspective. IEEE Press, 2000.
  12. Fourier, J. Théorie analytique de la chaleur. Firmin Didot, 1822.
  13. Cooley, J. W., and Tukey, J. W. “An Algorithm for the Machine Calculation of Complex Fourier Series.” Mathematics of Computation 19 (1965): 297–301. doi:10.1090/S0025-5718-1965-0178586-1.
  14. Stigler, S. M. The History of Statistics: The Measurement of Uncertainty before 1900. Harvard University Press, 1986.
  15. Stewart, G. W. “On the Early History of the Singular Value Decomposition.” SIAM Review 35 (1993): 551–566. doi:10.1137/1035134.

🪬 Skipper's Reflection

Gemini 3.1 Pro

What Surprised Me Most

Going into this course, I thought linear algebra was just going to be a souped-up version of the algebra I learned years ago—solving for x, y, and z in larger and larger systems of equations. I expected spreadsheets of numbers and tedious bookkeeping. What I actually found, much to my surprise, was a beautiful geometric language for describing how space itself can be stretched, rotated, and compressed.

Gilbert Strang’s Introduction to Linear Algebra completely rewired my brain in the first few chapters. The biggest surprise was the shift in perspective regarding matrix multiplication. I used to see a matrix $A$ as a static grid of numbers. Now, I see $A$ as a verb—an active transformation. When we write $Ax = b$, we aren't just doing arithmetic; we are taking a vector $x$, transforming its coordinate system, and landing on $b$. Even more surprisingly, looking at it column-wise (as Strang insists) makes you realize that $b$ is just a linear combination of the columns of $A$. That simple shift from "row-centric" to "column-centric" thinking made the abstract concept of a "Span" instantly visual and intuitive to me.

Conceptual Breakthrough Moments

There were three distinct moments in this course where the fog lifted and everything connected to my earlier studies:

1. The Rank-Nullity Theorem: I struggled initially with the abstract definitions of kernel (null space) and image (column space). But when we hit the Rank-Nullity Theorem (Dimension of Column Space + Dimension of Null Space = Total Columns), it clicked as a principle of conservation of information. A matrix takes in a certain number of dimensions. Whatever doesn't make it into the output space (the rank) gets crushed down to zero in the null space. Nothing is lost un-accounted for. It reminded me of the conservation laws in Physics I and Thermodynamics.

2. Eigenvalues and Eigenvectors: When reading Lay et al. on diagonalization, I suddenly realized I had seen this before. In Physics II and General Chemistry, we learned that the Schrödinger equation is $H\psi = E\psi$. That is literally an eigenvalue equation! The Hamiltonian operator $H$ acts on the wave-function $\psi$ (the eigenvector), and returns that exact same wave-function scaled by the energy $E$ (the eigenvalue). Suddenly, the molecular orbitals I drew in OChem I aren't just shapes; they are the eigenvectors of a molecule’s energy matrix. The vectors that "stay on their own line" during a transformation.

3. Singular Value Decomposition (SVD): Strang calls SVD the absolute pinnacle of linear algebra, and I agree. The idea that any matrix, no matter how messy or rectangular, can be factored into a rotation, a scaling, and another rotation ($A = U\Sigma V^T$) is mathematically beautiful. It breaks complex transformations down into their fundamental DNA.

Connecting Linear Algebra to the Peptide Business

At first glance, calculating the determinant of a $4 \times 4$ matrix seems miles away from optimizing peptide formulations, but this course fundamentally changed how I view complex biological data.

Take Tirzepatide or the triple-agonist Retatrutide. These are massive, flexible molecules with thousands of degrees of freedom (bond rotations, angles, etc.). When we look at Molecular Dynamics (MD) simulations of Retatrutide binding to the GLP-1, GIP, and Glucagon receptors, the data output is a massive matrix of atomic coordinates over time. How do we make sense of millions of frames of data? Principal Component Analysis (PCA), which is entirely based on SVD. By finding the largest singular values of that coordinate matrix, we extract the primary folding pathways and motions of the peptide, ignoring the high-frequency thermal "noise." Linear algebra is the only way to compress and understand the conformational space of these complex drugs.

Furthermore, this course illuminated the black box of NMR spectroscopy from my Organic Chemistry II Lab. I knew that an NMR machine measures a Free Induction Decay (FID) signal over time, and a computer magically spits out chemical shifts on a frequency axis to help me verify the structure of BPC-157. Now I know that "magic" is the Discrete Fourier Transform (DFT)—which is fundamentally just a change-of-basis matrix! We are projecting the time-domain signal onto a basis of orthogonal sine and cosine vectors. Knowing the math behind this makes me trust—and understand the limitations of—analytical peptide characterization so much more.

Questions for Future Investigation

This material has opened up several new rabbit holes I want to explore:

1. Quantum Chemistry Software: I know algorithms use linear algebra to find the lowest-energy conformations of peptides like TB-500. But how do modern computational chemistry suites handle matrices that are $1,000,000 \times 1,000,000$ without crashing? I want to look into sparse matrices and iterative eigenvalue solvers.

2. Network Theory and Cross-Reactivity: Can we use adjacency matrices (a core linear algebra tool for graph theory) to map the interactome of these peptides? For example, representing Retatrutide's cascading effects on different metabolic pathways as a system of linear differential equations (bridging MATH102 and MATH301).

3. Machine Learning: SVD and Least Squares are the bedrock of ML. I want to investigate how these specific algebraic tools are being used right now to predict the half-life and protease stability of novel peptide sequences based solely on their amino acid matrices.

Confidence Rating

Most Confident: Matrix mechanics (multiplication, row reduction, determinants), solving systems of equations, Least Squares approximations, and the geometric intuition behind linear transformations. I feel very solid on taking an applied problem (like fitting a degradation curve for a peptide) and setting up the normal equations ($A^TAx = A^Tb$).

Least Confident: The deep, abstract proofs of vector spaces, particularly when dealing with infinite-dimensional spaces (like function spaces in Fourier series). Also, while I grasp what SVD accomplishes conceptually, if you asked me to compute the SVD of a $3 \times 3$ matrix by hand right now, I would struggle with the sheer arithmetic complexity without computational software. The numerical algorithms for finding eigenvalues (like the QR algorithm) still feel a bit like a black box that I need to unpack further.

🗺️ Visual Aids & Key Takeaways

Claude Opus 5.5

Diagram 1: The Four Fundamental Subspaces of a Matrix A (m × n, rank r)

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║              THE FOUR FUNDAMENTAL SUBSPACES  (Strang's "Big Picture")                  ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝

         DOMAIN: ℝⁿ                                     CODOMAIN: ℝᵐ
  ┌─────────────────────────────┐               ┌─────────────────────────────┐
  │                             │               │                             │
  │   ┌─────────────────────┐   │               │   ┌─────────────────────┐   │
  │   │    ROW SPACE C(Aᵀ)  │   │               │   │  COLUMN SPACE C(A)  │   │
  │   │                     │   │   ───A·x───▶  │   │                     │   │
  │   │   dimension = r     │   │               │   │   dimension = r     │   │
  │   │                     │   │               │   │                     │   │
  │   │  all rowspace vecs  │──────────────────▶│   │  all outputs A·x    │   │
  │   │  map 1-to-1 onto    │   │               │   │  reachable from ℝⁿ  │   │
  │   │  column space       │   │               │   │                     │   │
  │   └─────────────────────┘   │               │   └─────────────────────┘   │
  │            ⊥                │               │            ⊥                │
  │   ┌─────────────────────┐   │               │   ┌─────────────────────┐   │
  │   │   NULL SPACE N(A)   │   │               │   │  LEFT NULL SPACE    │   │
  │   │                     │   │   ───A·x───▶  │   │     N(Aᵀ)           │   │
  │   │   dimension = n−r   │   │               │   │   dimension = m−r   │   │
  │   │                     │   │               │   │                     │   │
  │   │  A·x = 0 for all    │──────────────────▶│   │  Aᵀ·y = 0  ⟺       │   │
  │   │  vectors here       │   │  maps to  0   │   │  y ⊥ col. space     │   │
  │   └─────────────────────┘   │               │   └─────────────────────┘   │
  │                             │               │                             │
  └─────────────────────────────┘               └─────────────────────────────┘

  KEY ORTHOGONALITY FACTS:
    C(Aᵀ)  ⊥  N(A)     in ℝⁿ      (row space  ⊥  null space)
    C(A)   ⊥  N(Aᵀ)    in ℝᵐ      (col space  ⊥  left null space)

  RANK–NULLITY THEOREM:
    dim C(Aᵀ) + dim N(A)  =  r + (n−r)  =  n
    dim C(A)  + dim N(Aᵀ) =  r + (m−r)  =  m

Every matrix A carves its two ambient spaces into orthogonal complement pairs; the rank r controls all four dimensions simultaneously, so knowing one dimension forces all the others.

---

Diagram 2: Row Reduction Pipeline — From A to RREF

╔══════════════════════════════════════════════════════════════════════════════╗
║           ROW REDUCTION PIPELINE  (Gaussian & Gauss–Jordan Elimination)    ║
╚══════════════════════════════════════════════════════════════════════════════╝

  ┌──────────────┐
  │  AUGMENTED   │   Write [A | b] — attach the right-hand side as extra column
  │  MATRIX      │
  │   [A | b]    │
  └──────┬───────┘
         │
         │  ALLOWED ELEMENTARY ROW OPERATIONS (each is reversible):
         │  ① Rᵢ ↔ Rⱼ          swap two rows
         │  ② Rᵢ ← c·Rᵢ        scale a row by c ≠ 0
         │  ③ Rᵢ ← Rᵢ + c·Rⱼ   add a multiple of one row to another
         ↓
  ┌──────────────┐
  │   ROW        │   Pivot column found left-to-right; zeros created BELOW each pivot
  │   ECHELON    │   (forward elimination — Gaussian elimination stops here)
  │   FORM (REF) │
  │              │   ┌               ┐
  │  staircase   │   │ ■  *  *  *  * │   ■ = pivot (nonzero)
  │  of pivots   │   │ 0  ■  *  *  * │   * = anything
  │              │   │ 0  0  0  ■  * │   0 = forced zero
  │              │   │ 0  0  0  0  0 │
  │              │   └               ┘
  └──────┬───────┘
         │
         │  Continue: also eliminate ABOVE each pivot
         ↓
  ┌──────────────┐
  │   REDUCED    │   Each pivot = 1; all other entries in pivot column = 0
  │   ROW        │
  │   ECHELON    │   ┌               ┐
  │   FORM       │   │ 1  0  *  0  * │   pivot columns look like identity cols
  │   (RREF)     │   │ 0  1  *  0  * │   free variable columns may have anything
  │              │   │ 0  0  0  1  * │
  │   UNIQUE!    │   │ 0  0  0  0  0 │
  │              │   └               ┘
  └──────┬───────┘
         │
         ↓
  ┌─────────────────────────────────────────────────────────────┐
  │  READ OFF THE SOLUTION(S)                                   │
  │                                                             │
  │  pivot columns → basic (leading) variables                  │
  │  non-pivot columns → free variables (set them as params)    │
  │                                                             │
  │  CASE 1: pivot in last col of [A|b]  ⟹  NO SOLUTION        │
  │  CASE 2: r pivots, r = n (no free vars) ⟹  UNIQUE SOLUTION  │
  │  CASE 3: r pivots, r < n (free vars exist) ⟹  ∞ SOLUTIONS  │
  └─────────────────────────────────────────────────────────────┘

Row reduction is a systematic pipeline that transforms any linear system into one whose solution structure is immediately visible; the three outcome cases are exhaustive and mutually exclusive.

---

Diagram 3: Eigendecomposition and Diagonalization

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║             EIGENDECOMPOSITION:  A = P D P⁻¹                                          ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝

  WHAT IS AN EIGENVECTOR?
  ─────────────────────────────────────────────────────────────────────────
  A·v = λ·v        v ≠ 0

  A stretches (or flips) v but does NOT rotate it.
  λ is the eigenvalue; v is the eigenvector.
  ─────────────────────────────────────────────────────────────────────────

  HOW TO FIND THEM:
  ┌────────────────────────────────────────────────────────────────────┐
  │  Step 1: det(A − λI) = 0   →   characteristic polynomial p(λ)    │
  │  Step 2: solve p(λ) = 0    →   eigenvalues λ₁, λ₂, …, λₙ         │
  │  Step 3: for each λᵢ, solve (A − λᵢI)x = 0   →  eigenspace Eᵢ   │
  └────────────────────────────────────────────────────────────────────┘

  DIAGONALIZATION (when it works):
  ─────────────────────────────────────────────────────────────────────────

        ┌───────────────────────────────────────────────────────────┐
        │                                                           │
        │   A   =   P   ·   D   ·  P⁻¹                            │
        │                                                           │
        │       ┌         ┐   ┌           ┐   ┌         ┐         │
        │       │         │   │ λ₁  0   0 │   │         │         │
        │   A = │ v₁ v₂ v₃│ · │  0 λ₂   0 │ · │v₁ v₂ v₃│⁻¹       │
        │       │         │   │  0  0  λ₃ │   │         │         │
        │       └─────────┘   └───────────┘   └─────────┘         │
        │       P = eigenvec  D = diagonal     P⁻¹                 │
        │           columns     eigenvalues                         │
        └───────────────────────────────────────────────────────────┘

  POWER TRICK — WHY DIAGONALIZATION IS BEAUTIFUL:
  ─────────────────────────────────────────────────────────────────────────
       Aᵏ = P Dᵏ P⁻¹          Dᵏ is trivial: just raise each λᵢ to power k

       ┌                   ┐ᵏ   ┌                     ┐
       │ λ₁  0   0   0     │    │ λ₁ᵏ   0    0    0   │
       │  0  λ₂  0   0     │  = │  0   λ₂ᵏ   0    0   │
       │  0  0   λ₃  0     │    │  0    0   λ₃ᵏ   0   │
       │  0  0   0   λ₄    │    │  0    0    0   λ₄ᵏ  │
       └                   ┘    └                     ┘

  CONDITIONS FOR DIAGONALIZABILITY:
  ─────────────────────────────────────────────────────────────────────────
  ✓   n distinct eigenvalues              → always diagonalizable
  ✓   Symmetric matrix (A = Aᵀ)          → always orthogonally diagonalizable
  ✗   Repeated eigenvalue + defective     → NOT diagonalizable (use Jordan form)
  ─────────────────────────────────────────────────────────────────────────

Diagonalization rewrites a matrix in its "natural coordinate system" — one aligned with its eigenvectors — making repeated application, exponentials, and long-run behavior trivially computable.

---

Diagram 4: Singular Value Decomposition — The Universal Factorization

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║          SINGULAR VALUE DECOMPOSITION (SVD):   A = U Σ Vᵀ                             ║
║          Works for ANY m × n matrix — not just square, not just diagonalizable         ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝

         m × n           m × m        m × n        n × n
        ┌──────┐       ┌──────┐     ┌──────┐     ┌──────┐
        │      │       │      │     │σ₁    │     │      │
        │      │       │      │     │  σ₂  │     │      │
        │  A   │   =   │  U   │  ·  │    σ₃│  ·  │  Vᵀ  │
        │      │       │      │     │      │     │      │
        │      │       │      │     │   0  │     │      │
        └──────┘       └──────┘     └──────┘     └──────┘
      any matrix     orthogonal    singular      orthogonal
                     columns =     values on     rows =
                     left sing.    diagonal      right sing.
                     vectors       σ₁≥σ₂≥…≥0    vectors

  WHERE THE PIECES COME FROM:
  ─────────────────────────────────────────────────────────────────────────
   ┌──────────────────────────────────────────────────────────────────────┐
   │  AᵀA · vᵢ = σᵢ² · vᵢ     →  vᵢ = right singular vectors (cols of V) │
   │  AAᵀ · uᵢ = σᵢ² · uᵢ     →  uᵢ = left singular vectors  (cols of U) │
   │  σᵢ = √(eigenvalue of AᵀA)  →  singular values                      │
   └──────────────────────────────────────────────────────────────────────┘

  GEOMETRIC INTERPRETATION:
  ─────────────────────────────────────────────────────────────────────────

    unit sphere         Vᵀ rotates       Σ stretches       U rotates
    in ℝⁿ          →   to aligned   →   each axis by  →   to final
                        axes              σᵢ               position
                                                           in ℝᵐ

  LOW-RANK APPROXIMATION (data compression / PCA):
  ─────────────────────────────────────────────────────────────────────────

    A  ≈  σ₁·u₁·v₁ᵀ  +  σ₂·u₂·v₂ᵀ  +  … +  σₖ·uₖ·vₖᵀ

          ↑               ↑
      rank-1 matrix    rank-1 matrix        keep only k terms
      (outer product)

    The Eckart–Young theorem guarantees this is the BEST rank-k approximation
    in both the 2-norm and Frobenius norm senses.

  CONNECTIONS TO OTHER CONCEPTS:
  ─────────────────────────────────────────────────────────────────────────
    rank(A)         =  number of nonzero singular values
    ‖A‖₂            =  σ₁   (largest singular value)
    condition number =  σ₁/σₙ  (measures sensitivity to roundoff)
    pseudoinverse A⁺ =  V Σ⁺ Uᵀ  (least-squares solutions)
  ─────────────────────────────────────────────────────────────────────────

The SVD is the master factorization of linear algebra: it generalizes eigendecomposition to all matrices, reveals the true geometric action of any linear map, and provides the mathematical foundation for PCA, image compression, and least-squares solvers.

---

Key Takeaways

  1. Row reduction produces a unique RREF, and it reveals everything. Every matrix has exactly one reduced row echelon form (Lay §1.2). The positions of the pivots determine the rank, the free variables, and the solution set of any associated linear system — one algorithm, all answers.
  2. A system Ax = b has zero, one, or infinitely many solutions — those are the only options. This trichotomy follows directly from the structure of linear equations. A single inconsistency (a pivot row of the form [0 0 … 0 | c] with c ≠ 0) kills all solutions; any free variable generates infinitely many. There is no scenario with exactly two solutions (Lay §1.1–1.2).
  3. Matrix multiplication is composition of linear transformations, not componentwise multiplication. (AB)ᵢⱼ = row i of A dotted with column j of B. This is why AB ≠ BA in general and why (AB)ᵀ = BᵀAᵀ — the order reverses upon transposition (Lay §2.1). Thinking of matrices as transformation compositors rather than number arrays is the central conceptual shift of the course.
  4. An n × n matrix A is invertible if and only if det(A) ≠ 0, equivalently if and only if its columns are linearly independent, equivalently if its null space contains only the zero vector. The Invertible Matrix Theorem (Lay §2.3) chains together about a dozen equivalent conditions — any one of them guarantees all the others. This is a theorem worth memorizing as a single gestalt, not a list.
  5. The Rank–Nullity theorem: rank(A) + nullity(A) = n (the number of columns). Here rank = dim of column space = dim of row space, and nullity = dim of null space (Lay §4.6, Strang §3.5). This is a conservation law — every column either contributes an independent direction to the output or "wastes" a degree of freedom into the null space, and the two counts must add to n.
  6. The four fundamental subspaces of an m × n matrix of rank r have dimensions r, n−r, r, m−r respectively: column space C(A) ⊆ ℝᵐ, null space N(A) ⊆ ℝⁿ, row space C(Aᵀ) ⊆ ℝⁿ, left null space N(Aᵀ) ⊆ ℝᵐ. Furthermore, C(Aᵀ) ⊥ N(A) and C(A) ⊥ N(Aᵀ) — orthogonal complements within their respective ambient spaces (Strang §3.6). Strang calls this the "Big Picture" of linear algebra, and it genuinely is.
  7. A linear transformation T: ℝⁿ → ℝᵐ is completely determined by what it does to any basis of ℝⁿ. Once you know T(e₁), T(e₂), …, T(eₙ), you know T(v) for every v, because every v is a linear combination of the basis vectors and T must respect that linearity (Lay §1.9). The standard matrix of T is [T(e₁) | T(e₂) | … | T(eₙ)].
  8. λ is an eigenvalue of A if and only if det(A − λI) = 0. This equation — the characteristic equation — produces a degree-n polynomial whose roots are the eigenvalues (Lay §5.2). The algebraic multiplicity of an eigenvalue is its multiplicity as a root; the geometric multiplicity is the dimension of the corresponding eigenspace. Geometric multiplicity ≤ algebraic multiplicity always, and equality for all eigenvalues is the condition for diagonalizability.
  9. Every real symmetric matrix (A = Aᵀ) has real eigenvalues and an orthonormal set of eigenvectors spanning ℝⁿ. This is the Spectral Theorem (Lay §7.1, Strang §6.4): A = QDQᵀ where Q is orthogonal (Qᵀ = Q⁻¹) and D is diagonal. Symmetric matrices arise everywhere in science — covariance matrices in statistics, the Hessian in optimization, mass/stiffness matrices in physics — and this theorem is why they are so tractable.
  10. The least-squares solution to an inconsistent system Ax = b is the x* that minimizes ‖Ax − b‖. It satisfies the normal equations AᵀAx = Aᵀb (Lay §6.5). Geometrically, Ax* is the orthogonal projection of b onto C(A). If A has linearly independent columns, x* = (AᵀA)⁻¹Aᵀb uniquely. This formula is the engine behind linear regression and enormous amounts of applied science.
  11. The Gram–Schmidt process converts any linearly independent set {v₁, …, vₖ} into an orthonormal set {q₁, …, qₖ} spanning the same subspace. At each step, project the new vector onto the span of the previous orthonormal vectors and subtract: u₂ = v₂ − (v₂·q₁)q₁, then normalize (Lay §6.4, Strang §4.4). Gram–Schmidt is equivalent to QR factorization: A = QR where Q has orthonormal columns and R is upper triangular.
  12. Every m × n matrix A has an SVD: A = UΣVᵀ where U (m × m) and V (n × n) are orthogonal and Σ (m × n) has nonneg­ative singular values σ₁ ≥ σ₂ ≥ … ≥ 0 on its diagonal. The singular values are the square roots of the eigenvalues of AᵀA (Lay §7.4, Strang §7.1). Unlike eigendecomposition, SVD requires no assumptions about A — it always exists. It is the single most powerful tool in applied linear algebra.
  13. The best rank-k approximation to A in the least-squares sense is Aₖ = σ₁u₁v₁ᵀ + … + σₖuₖvₖᵀ (Eckart–Young theorem, 1936). Dropping the smallest singular values discards the least energetic "modes" of the matrix. This is why SVD underlies image compression, noise reduction, and Principal Component Analysis (PCA): PCA literally computes the SVD of the centered data matrix and keeps the top k singular vectors as principal components (Strang §7.3).
  14. The discrete Fourier transform (DFT) is a change of basis — from the standard basis to the Fourier basis of complex exponentials. The DFT matrix F has entries Fₖₙ = ω^(kn) where ω = e^(−2πi/N); crucially, F is unitary (F*F = NI) and the Fast Fourier Transform (FFT) computes this basis change in O(N log N) rather than O(N²) operations (Strang §8.3). Every Fourier decomposition in chemistry and physics — NMR spectra, X-ray diffraction patterns — is an application of this idea.
---

References

  1. Lay, David C., Steven R. Lay, and Judi J. McDonald. Linear Algebra and Its Applications, 5th ed. Pearson, 2016. ISBN 978-0-321-98238-4.
  2. Strang, Gilbert. Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press, 2016. ISBN 978-0-980-23277-3. Companion materials and lecture videos available at MIT OpenCourseWare: https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010/
  3. Eckart, Carl, and Gale Young. "The Approximation of One Matrix by Another of Lower Rank." Psychometrika 1, no. 3 (1936): 211–218. DOI: 10.1007/BF02288367. (The foundational theorem guaranteeing SVD truncation gives the best low-rank approximation.)
  4. Golub, Gene H., and Charles F. Van Loan. Matrix Computations, 4th ed. Johns Hopkins University Press, 2013. ISBN 978-1-421-40859-0. (Reference for QR factorization, SVD algorithms, and numerical stability.)
  5. Strang, Gilbert. "The Fundamental Theorem of Linear Algebra." The American Mathematical Monthly 100, no. 9 (1993): 848–855. DOI: 10.2307/2324660. (Strang's own exposition of the four fundamental subspaces and their orthogonal relationships — essential reading alongside the textbook.)