Principal Component Analysis (PCA) is a statistical technique to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. It captures the maximum variability in the data in decreasing order of significance.
Given a data matrix X of dimension N x M, where N is the number of samples and M is the number of features, PCA aims to find a few orthogonal linear combinations of the M features that capture most of the variability.
We first center the data by subtracting the mean:
Where
We then calculate the covariance matrix:
The eigen decomposition of S gives us:
Where
We rank eigenvalues from largest to smallest significance. The K principal axes defined by eigenvectors corresponding to the top K eigenvalues represent the directions of maximum variability in the data.
To project a point x onto the new K dimensional space:
Where
The projected data xPCA has a lower dimension while preserving most information about the variability in X. We have reduced the dimensionality from M to K principal components.
The proportion of total variance captured by selecting K components is given by: