Table of contents
Course page: AM301
Singular Value Decomposition
Source video: Lecture: The Singular Value Decomposition (SVD) - AMATH301
linalg: Matrix acts on vector
All what matrix multiplication does is it rotates and stretch the vectors.
Principal Component Analysis
Source video: Lecture: Principal Componenet Analysis (PCA)
Physical example
Considering a physical system as the following, a rod suspends a mass via a spring.
Pull the mass down and let it oscillate up and down.
The displacement (position) of the mass oscillation can be measured by a function f(z). Then f(z) can be solved based on Newton’s second law:
$$ \begin{aligned} F &= ma = m \frac{d²f(z)}{dz²} \\\ \ \\\ -w²f(z) &= m \frac{d²f(z)}{dz²} f(z) = Acos(wz + w₀) \end{aligned} $$Data-driven approach can solve it too.
Suppose the law $F=ma$ is unknown, infer the F=ma from the data alone. First of all, the complexity of this system should be figured out.
Measure this system using cameras
Every camera records the mass coordinates on their projection plane:
Camera 1: (𝐱$_a$, 𝐲$_a$); Camera 1: (𝐱$_b$, 𝐲$_b$); Camera 1: (𝐱$_c$, 𝐲$_c$).
Arrange them into a data matrix 𝐗 (6-rows):
$$ 𝐗 = \begin{bmatrix}𝐱_a \\\ 𝐲_a \\\ 𝐱_b \\\ 𝐲_b \\\ 𝐱_c \\\ 𝐲_c \end{bmatrix} $$Two fundamental issuses associated with these data need to be addressed.
-
Noise
Data with noise on top of it is not a good representation of the system.
-
Redundancy
Measurements are not independent to each other, i.e., x and y are related. Different cameras take the similar infomation just from different angles.
The movement is only one degree of freedom, but the observed data has six sets.
One doesn’t know how to take the perfect observation ahead of time. PCA will reveal which camera at which angle is enough to describe the whole system.
Variance and Covariance
Assumption: big variance score means that vector is chaning a lot. It has a lot stuff happening.
- If the diagonal terms that are big in the covariance matrix , those vectors are matter. Vectors having small variance don’t change much.
MAKE DIAGNOAL (SVD) is remove all the redundancy
PCA for Face Recognition
Source video: Lecture: PCA for Face Recognition
- A 2D image are flattened into a vector.
- So each column in the U returned from SVD is an image.
- So a column essentially contains two directions: x and y.