🤯 Eigen.. what?

Hey friends! 👋

Today we'll talk about Eigenvalues and Eigenvectors. Sounds scary? Don't worry — I'm scared too 😂 Think of them as the secret sauce behind many AI and Computer Vision tricks.

So what's the deal?

A matrix can be thought of as a transformation of space: rotating, stretching, squishing. But some directions are “special” — if you push a vector in those directions, it won't rotate, only stretch or shrink. These are the eigenvectors. The amount of stretch/shrink is the eigenvalue.

The math definition

We say:

\[ A \mathbf{v} = \lambda \mathbf{v} \]

Where:

• \( \mathbf{A} \) → our matrix (transformation)
• \( \mathbf{v} \) → eigenvector (special direction)
• \( \lambda \) → eigenvalue (scale factor)

Applications

• PCA (Principal Component Analysis) - Reducing data dimensions while keeping the most important info.
• Face recognition - Remember the famous Eigenfaces? That's literally eigenvectors of a face dataset. The computer learns the “main directions of variation” in faces and uses them to compare who’s who. 😎
• Graph analysis - Browsers page rank algorithm uses eigenvectors to figure out which websites are “important”. Yep, the internet itself runs on linear algebra haha 🤭

Symmetric Matrices & Why PCA Loves Them

When working with matrices in data science, a lot of times we deal with covariance matrices. A covariance matrix basically is a square matrix that summarizes how much each pair of variables in your dataset varies together:

• If two variables tend to increase or decrease together, their covariance is positive.
• If one increases while the other decreases, their covariance is negative.
• If they are independent, their covariance is close to zero.

Covariance matrices are symmetric and defined below:

\[ \Sigma = \frac{1}{n} X^\top X \]

Symmetric matrices have some powers:

• ✅ All eigenvalues are real numbers — complex numbers bye bye 👋
• ✅ Eigenvectors are orthonormal — perpendicular and of length 1.

Why is this awesome for PCA?

• Eigenvalues literally tell us how much “variance” is captured in each direction.
• Orthonormal eigenvectors give independent axes — perfect for projecting data without mixing dimensions.

How PCA works

1. Center the data: subtract the mean from each feature.
2. Compute the covariance: \[ \Sigma = \frac{X_c^\top X_c}{n-1} \]
3. Find eigenvalues and eigenvectors of \(\Sigma\).
4. Sort them by descending eigenvalue, keep the top \(k\) vectors — these are the principal components.
5. Project the data: \[ Z = X_c \cdot W_k \]
6. Explained variance ratio: \[ \frac{\lambda_i}{\sum \lambda_j} \]

PCA from scratch in Python

import numpy as np

def pca_from_scratch(X, k):
    """
    Perform PCA using eigen-decomposition of the covariance matrix.
    X: 2D array of shape (n_samples, n_features)
    k: number of principal components to keep
    Returns: (Z, components, explained_variance_ratio)
    """
    X = np.asarray(X, dtype=float)              # ensure float NumPy array
    n_samples = X.shape[0]                      # number of data points
    mean = X.mean(axis=0, keepdims=True)        # feature-wise mean
    Xc = X - mean                               # center the data

    Sigma = (Xc.T @ Xc) / (n_samples - 1)       # covariance matrix

    eigvals, eigvecs = np.linalg.eigh(Sigma)    # eigen-decomposition (stable for symmetric Σ)

    idx = np.argsort(eigvals)[::-1]             # sort eigenvalues in descending order
    eigvals = eigvals[idx]                      # reorder eigenvalues
    eigvecs = eigvecs[:, idx]                   # reorder eigenvectors

    W = eigvecs[:, :k]                          # top-k principal directions
    Z = Xc @ W                                  # project data into low-dim space

    total_var = eigvals.sum()                   # total variance = sum of eigenvalues
    evr = eigvals[:k] / total_var               # explained variance ratio

    return Z, W, evr


# --- Demo with correlated 2D dataset ---
rng = np.random.default_rng(42)                 # reproducible random generator
mean = [0.0, 0.0]                               # 2D mean
cov = np.array([[3.0, 2.0], [2.0, 2.0]])        # covariance (positive semidefinite)
X_demo = rng.multivariate_normal(mean, cov, size=500)  # sample 500 correlated points

Z, components, evr = pca_from_scratch(X_demo, k=2)

print("Principal components (columns):\n", components)
print("Explained variance ratio:", evr)

# Reduce to 1D if you only need the strongest direction
Z1, W1, evr1 = pca_from_scratch(X_demo, k=1)
print("1D EV ratio:", evr1)
                                

Bottom line: eigenvectors give us the “right coordinate system” where the transformation is just simple scaling. PCA turns this into a way of finding directions of maximum variance and compressing data — fast, useful, and super common in computer vision and data science.