Principal
Component
Analysis

What is PCA?

Principal Component Analysis (PCA) is a dimensionality reduction technique that transforms a dataset with many correlated features into a smaller set of uncorrelated variables called principal components.

Each principal component is a linear combination of the original features, ordered by the amount of variance it explains — PC1 always explains the most variance, PC2 the second most, and so on.

Why use PCA?

Modern datasets often have hundreds or thousands of features. This creates several problems that PCA is specifically designed to solve.

The Intuition

Simple Example: Height & Weight

Suppose you have data on people's height and weight. These two features are correlated — taller people tend to weigh more.

Result: You compress 2 features into 1 (PC1) while keeping ~95% of the information. You traded a little accuracy for a huge simplification.

Standardize the Data

Before applying PCA, we must standardize each feature so that it has mean = 0 and standard deviation = 1. This is called z-score normalization.

where μ is the feature mean and σ is the standard deviation.

Example: Student Data

After standardization, all three features have mean=0 and std=1. Now PCA treats them equally.

from sklearn.preprocessing import StandardScaler
import numpy as np

data = {
    'math':    [85, 60, 92, 70, 78],
    'science': [82, 65, 90, 68, 75],
    'hours':   [8,  5,  10, 6,  7 ]
}

scaler = StandardScaler()
X_scaled = scaler.fit_transform(pd.DataFrame(data))

# X_scaled[:,0].mean() ≈ 0.0   X_scaled[:,0].std() ≈ 1.0
print(f"Mean: {X_scaled.mean(axis=0).round(10)}")   # [0. 0. 0.]
print(f"Std:  {X_scaled.std(axis=0).round(2)}")     # [1. 1. 1.]

Compute the Covariance Matrix

The covariance matrix tells us how much each pair of features varies together. It is an n×n symmetric matrix where n is the number of features.

The diagonal contains the variance of each feature. Off-diagonal elements show how two features move together:

• Positive covariance → features rise and fall together
• Negative covariance → one rises when the other falls
• Near zero → features are largely independent

Covariance Matrix — Student Example (3×3)

Color intensity = strength of correlation. Darker blue = stronger positive relationship.

X_cov = np.cov(X_scaled.T)  # transpose: features as rows

print(X_cov.round(2))
# [[1.   0.97 0.98]
#  [0.97 1.   0.96]
#  [0.98 0.96 1.  ]]

Eigenvalues & Eigenvectors

This is the mathematical core of PCA. We decompose the covariance matrix to find its eigenvalues and eigenvectors.

where A = covariance matrix, v = eigenvector, λ = eigenvalue

Eigenvalues for Student Dataset

eigenvalues, eigenvectors = np.linalg.eigh(X_cov)

# Sort by eigenvalue descending
idx = np.argsort(eigenvalues)[::-1]
eigenvalues  = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]

variance_explained = eigenvalues / eigenvalues.sum()
cumulative         = np.cumsum(variance_explained)

print("Eigenvalues:",         eigenvalues.round(3))  # [2.94  0.05  0.01]
print("Variance explained:",  variance_explained.round(3))
print("Cumulative variance:", cumulative.round(3))

Selecting k Components

After computing the eigenvalues, we choose how many principal components (k) to keep. This is one of the most important decisions in PCA.

Three Rules for Choosing k

from sklearn.decomposition import PCA

# Option A: specify exact number of components
pca = PCA(n_components=2)

# Option B: specify variance threshold (auto-selects k)
pca = PCA(n_components=0.95)  # keep 95% of variance

pca.fit(X_scaled)
print(f"Components kept: {pca.n_components_}")
print(pca.explained_variance_ratio_)  # per-component %
print(pca.explained_variance_ratio_.cumsum())  # cumulative %

Transform the Data

Once we've selected k components, we project the original data onto the new principal component axes. This is a simple matrix multiplication.

where W is the matrix of k eigenvectors (shape: n_features × k)

The result is a new dataset with k columns instead of the original n columns, where each column is uncorrelated with all others.

Transformed Student Data (k=1)

To reconstruct the original data (approximately), use inverse_transform:

pca = PCA(n_components=1)
X_pca = pca.fit_transform(X_scaled)        # compress: (5,3) → (5,1)
X_back = pca.inverse_transform(X_pca)     # reconstruct: (5,1) → (5,3)

# X_back ≈ X_scaled (not exact — we lost the 2% in PC2+PC3)
reconstruction_error = np.mean((X_scaled - X_back)**2)
print(f"Reconstruction MSE: {reconstruction_error:.4f}")

Complete Python Implementation

Here is a complete, production-ready PCA implementation using scikit-learn with all best practices.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline

# ─── 1. Load and prepare data ───────────────────────────────
df = pd.read_csv('dataset.csv')
X = df.drop('target', axis=1)

# ─── 2. Build PCA pipeline ──────────────────────────────────
pipeline = Pipeline([
    ('scaler', StandardScaler()),        # always standardize first!
    ('pca',    PCA(n_components=0.95))  # keep 95% variance
])

X_pca = pipeline.fit_transform(X)
pca   = pipeline.named_steps['pca']

# ─── 3. Examine results ─────────────────────────────────────
k = pca.n_components_
print(f"Original features:  {X.shape[1]}")
print(f"PCA components kept: {k}")
print(f"Variance retained:  {pca.explained_variance_ratio_.sum():.1%}")

# ─── 4. Scree plot ───────────────────────────────────────────
plt.figure(figsize=(8, 4))
plt.bar(range(1, k+1), pca.explained_variance_ratio_, color='#4f8ef7')
plt.plot(range(1, k+1), pca.explained_variance_ratio_.cumsum(), 'r--')
plt.xlabel('Principal Component')
plt.ylabel('Variance Explained')
plt.title('Scree Plot')
plt.show()

# ─── 5. Use in ML model ─────────────────────────────────────
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

full_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('pca',    PCA(n_components=0.95)),
    ('clf',    LogisticRegression())
])

scores = cross_val_score(full_pipeline, X, df['target'], cv=5)
print(f"CV Accuracy: {scores.mean():.1%} ± {scores.std():.1%}")

How to Choose k (Explained Variance)

The right value of k depends on your goals. Here is a practical decision guide:

Understanding Loadings

PCA components are linear combinations of original features. The coefficients are called loadings (or component weights). They tell you which original features each principal component is "made of".

Example: Student Loadings

pca = PCA(n_components=2)
pca.fit(X_scaled)

# pca.components_ shape: (n_components, n_features)
loadings = pd.DataFrame(
    pca.components_.T,
    index   = ['math', 'science', 'hours'],
    columns = ['PC1', 'PC2']
)

print(loadings.round(3))
#          PC1    PC2
# math     0.578 -0.401
# science  0.575 -0.448
# hours    0.579  0.800

Advantages & Disadvantages

When to Use PCA

PCA Variants & Alternatives

Standard PCA has limitations. Here are specialized variants for different situations.

Key Terms

Test Your Understanding

Click an answer to check if you are right. Explanations appear after answering.