Principal Component Analysis

Principal component analysis (PCA) is a method for replacing a collection of possibly correlated variables by uncorrelated linear combinations called principal components. The components are ordered so that the first accounts for the greatest possible variance, the second accounts for the greatest remaining variance subject to being orthogonal to the first, and so on.

Let the rows of the matrix be centered observations and let

(1)

The matrix is the sample covariance matrix. If its orthonormal eigenvectors and eigenvalues are ordered so that

			(2)
			(3)

then the th principal-component score of the th observation is

(4)

The variance of the th component is , and the fraction of total variance that it explains is

(5)

Keeping only the first components gives a lower-dimensional representation that maximizes retained variance and, equivalently, minimizes the sum of squared orthogonal reconstruction errors among -dimensional linear subspaces. PCA can also be performed on standardized variables using their correlation matrix instead of their covariance matrix. The method originated with Pearson's closest-fitting subspaces and was developed and named by Hotelling (Pearson 1901, Hotelling 1933).