watch: LA - G.S. 21 | Eigenvalues & Eigenvectors

Source video: 21. Eigenvalues and Eigenvectors - Gilbert Strang - MIT OpenCourseWare

(2023-11-06) The 1st-time recitation note. Haven’t re-check.

Out direction same as in

Matrix multiply vector acting like MLP, where all current dimensions are combined in different weights to produce each dimension in another space.

$$ 𝐀 𝐗 \\\ \begin{bmatrix} y_1 \\\ y_2 \\\ y_3 \end{bmatrix} = 𝐀 \begin{bmatrix} x_1 \\\ x_2 \end{bmatrix} $$

Matrix 𝐀 is like a function that takes as input 𝐗 and output 𝐀𝐗.
Typically, 𝐀𝐗 points in different direction from the input 𝐗.
Eigenvectors are those 𝐗 make 𝐀𝐗 parallels to 𝐗.
Parallel means scaling: 𝐀𝐗 = λ𝐗, where the multiplier λ is eigenvalue.

Given a matrix 𝐀, how to solve its eigenvalues λ and eigenvectors 𝐗?

If λ = 0, 𝐗 solved from 𝐀𝐗 = 0 by elimination is 𝟎, which is useless.

Some hints can be found through the following examples

Projection matrix

Given a projection matrix 𝐏, a 3D vector is projected on to a plane.

Eigenvectors 𝐗 should parallel to the projected vectors 𝐏𝐗. So the vectors located in the plane all are eigenvectors, as $𝐏𝐗=𝐗$, with λ=1.
Vectors perpendicular to the plane statisfy: $𝐏𝐗=0$, i.e., λ=0.
Those two sets of eigenvectors are perpendicular.

Permutation matrix

Given the 2D permutation matrix $𝐀=[^{0 \ 1}_{1 \ 0}]$, which vector can make 𝐀𝐗=𝐗, i.e., identical after permutation.

𝐗 = $[^1_1]$, with λ=1.
As 𝐀 is a 2D matrix, there should be 2 eigenvalues. Which vector can statisfy λ=-1 ?

𝐗 = $[^{-1}_1]$
The 2 sets of eigenvectors 𝐗 are perpendicular as well.
Fact: The sum of eigenvalues equals to the sum of elements on diagonal of 𝐀.

Here, there is 1 + (-1) = 0 + 0.

How to solve λ, 𝐗

Trick: Rearrange to 𝐀𝐗 - λ𝐗 = 0.

If there is $(𝐀 - λ𝐈)𝐗 = 0$ and 𝐗, a non-zero vector, becomes 0 after multiplied by (𝐀 - λ𝐈), the (𝐀 - λ𝐈) must be sigular: determinant is 0.

The formula $𝐀 - λ𝐈 = 0$ doesn’t include 𝐗, so eigenvalues λ can be solved first, as in the example below.

Plus multiple identity

Given a matrix 𝐀 = $[^{3 \ 1}_{1 \ 3}]$, to calculate λ, solve:

$$ \begin{aligned} | 𝐀 - λ𝐈 | = 0 \\\ \begin{bmatrix} 3-λ & 1 \\\ 1 & 3-λ \end{bmatrix} = 0 \\\ (3-λ)^2 -1 = 0 \\\ λ^2 - 6λ + 8 = 0 \\\ (λ-4)(λ-2) = 0 \end{aligned} $$

Two roots: λ₁=4, λ₂=2. Then solve the 2 sets of eigenvectors.

For λ₁=4,
$$ \begin{aligned} (𝐀 - λ𝐈)𝐗 = 0 \\\ \begin{bmatrix} -1 & 1 \\\ 1 & -1 \end{bmatrix} 𝐗 = 0 \\\ \end{aligned} $$
By letting the free variable to 1, 𝐗 can be solved as $[^1_1]$.

Hence, one of eigenvectors is $[^1_1]$
For λ₂=2,
$$ \begin{aligned} (𝐀 - λ𝐈)𝐗 = 0 \\\ \begin{bmatrix} 1 & 1 \\\ 1 & 1 \end{bmatrix} 𝐗 = 0 \\\ \end{aligned} $$
Hence, one of eigenvectors is $[^{-1}_1]$

The sum of the two eigenvalues is the trace of 𝐀, and their product is the determinant of 𝐀.

Comparing the matrix $[^{3 \ 1}\_{1 \ 3}]$ and the permutation matrix $[^{0 \ 1}\_{1 \ 0}]$, there is:

A	$[^{3 \ 1}\_{1 \ 3}]$	$[^{0 \ 1}\_{1 \ 0}]$
λ	4 and 2	1 and -1
𝐗	$[^1\_1]$ and $[^{-1}\_1]$	$[^1\_1]$ and $[^{-1}\_1]$

If $𝐀𝐗 =λ𝐗$, then $(𝐀+3𝐈)𝐗= 𝐀𝐗+3𝐈𝐗 = (λ+3𝐈)𝐗$

That means, if 𝐀 plus 3𝐈, then eigenvalues λ will plus 3𝐈, while eigenvectors doesn’t change.

Can’t generalize

However, th above property (for multiple identity) can’t be generalized to plusing an arbitrary matrix 𝐁.

Matrix addition doesn’t imply eigenvalues addition, because the eigenvector of 𝐁 is typically not 𝐗. Therefore, the following addition can’t be performed:

$$ 𝐀𝐗 =λ𝐗 \\\ 𝐁𝐗 =α𝐗 \\\ (𝐀+𝐁)𝐗 = (λ+α)𝐗 $$

Complex eigenvalues

Considering rotation matrix:

$$ \begin{aligned} \begin{bmatrix} cosθ \\\ sinθ \end{bmatrix} = \begin{bmatrix} cosθ & -sinθ \\\ sinθ & cosθ \end{bmatrix} \begin{bmatrix} 1 \\\ 0 \end{bmatrix} \end{aligned} $$

Let θ be 90°, the rotation matrix is $[^{0 \ -1}\_{1 \ 0}]$.

However, according to the theory “eigenvector outcomes in the direction that it went in”, there seems to be no eigenvector intuitively, as any vector gets in will rotate 90 degree by this matrix.

Solve its eigenvalues:
$$ \begin{aligned} | 𝐀 - λ𝐈 | = 0 \\\ |\begin{bmatrix} 0 & -1 \\\ 1 & 0 \end{bmatrix} - \begin{bmatrix} λ & 0 \\\ 0 & λ \end{bmatrix}| = \begin{vmatrix} -λ & -1 \\\ 1 & -λ \end{vmatrix} = 0 \\\ λ^2 + 1 = 0 \end{aligned} $$
Two roots: λ₁=i, λ₂=-i

Anti-symmetric matrix has imaginary eigenvalues which are always in pairs, as they’re complex conjugate. While a symmetrix matrix’s eigenvalues are all real numbers.
Solve eigenvectors 𝐗:
- For λ₁=i,
  $$ \begin{aligned} (𝐀 - λ𝐈) 𝐗 = 0 \\\ \begin{bmatrix} 0-λ₁ & -1 \\\ 1 & 0-λ₁ \end{bmatrix} \begin{bmatrix} a \\\ b \end{bmatrix} = 0 \\\ \begin{bmatrix} -i & -1 \\\ 1 & -i \end{bmatrix} \begin{bmatrix} a \\\ b \end{bmatrix} = 0 \\\ \end{aligned} $$
  Let the free variable b=1, then $[^a_b] = [^i_1]$
- For λ₁= -i,
  $$ \begin{aligned} \begin{bmatrix} i & -1 \\\ 1 & i \end{bmatrix} \begin{bmatrix} a \\\ b \end{bmatrix} = 0 \\\ \end{aligned} $$
  Let the free variable b=1, then $[^a_b] = [^{-i}_1]$

Repeated eigenvalues

Eigenvalues of a triangular matrix is obvious: the diagonal, as the determinant is directly factorized.

For example, given matrix $𝐀 = [^{3 \ 1}_{0 \ 3}]$:

Solve its eigenvalues:
$$ \begin{aligned} | 𝐀 - λ𝐈 | = 0 \\\ \begin{vmatrix} 3-λ & 1 \\\ 0 & 3-λ \end{vmatrix} = 0 \\\ (3-λ)(3-λ) \end{aligned} $$
Two roots: λ₁=3, λ₂= 3
Solve eigenvectors:
1. For λ₁=3,
  $$ \begin{aligned} (𝐀 - λ𝐈) 𝐗 = 0 \\\ \begin{bmatrix} 3-λ₁ & 1 \\\ 0 & 3-λ₁ \end{bmatrix} \begin{bmatrix} a \\\ b \end{bmatrix} = 0 \\\ \begin{bmatrix} 0 & 1 \\\ 0 & 0 \end{bmatrix} \begin{bmatrix} a \\\ b \end{bmatrix} = 0 \end{aligned} $$
  So $[^a_b]$ can be $[^1_0]$
2. For λ₂= 3, same formulas appear. There isn’t the 2nd (set of) independent eigenvectors.
  
  But there are supposed to be two eigenvalues. It’s incomplete yet.

(2024/06/22)

视频 | 特征向量和特征值的应用 - 遇见数学

Zach-Youtube

Table of contents