PLS vs OLS
Can a smaller model make better predictions? Change the data to see when partial least squares helps—and when ordinary least squares does just as well or better.
| Model | Training data | 200 new observations |
|---|---|---|
| PLS | ||
| OLS |
Predictions on new observations
How many components?
Where PLS improves on OLS
Ordinary least squares finds coefficients that minimize training error. When many predictors are strongly correlated, those coefficients can be sensitive to noise. A close fit to the training data can still predict new observations poorly.
Partial least squares combines the predictors into a few components chosen using their relationship with the response. Using fewer components can reduce prediction variance, trading some flexibility for stability. It is useful when predictors overlap or outnumber observations.
PLS is not automatically better. Too few components can underfit; too many can fit noise. With enough clean training data, OLS can match or outperform it. Try the “Plenty of training data” example, then increase the PLS component count.
How the comparison works
The data are synthetic and reproducible from the sample seed. Predictors form two independent groups; correlation applies within each group. The response is a known linear combination of the predictors plus Gaussian noise, with an intercept of 5. The signal variance is held constant when predictor count or correlation changes; the noise control sets the response-noise standard deviation.
Both methods learn centering and sample-standard-deviation scaling from training observations only. Each validation fold fits its own scaling. The PLS implementation supports one or multiple responses; this experiment displays one. The suggested component count uses only the training folds, never the test responses. Exploring component counts after looking at test errors is an illustration; a final real-world evaluation needs fresh held-out data.
OLS uses a singular-value decomposition, including a minimum-norm least-squares solution in standardized predictor coordinates when coefficients are not uniquely determined. No ridge penalty is added. The PLS path stops when its usable rank or response covariance is exhausted. The interface explores up to 12 components, subject to the training-fold limits.
Implementation, verification, and sources
This is a TypeScript port of my Scala PLS project, implementing Dayal and MacGregor’s linear kernel algorithm. It is not a nonlinear kernel model. Single- and multiple-response predictions are checked against scikit-learn’s independent NIPALS implementation; OLS is checked against NumPy’s least-squares solver. Tests also cover constant and duplicate columns, intercepts, rank exhaustion, and validation without test-data leakage.
Regression source · Experiment source · Verification record · License notices
- Dayal and MacGregor, Improved PLS Algorithms, 1997.
- scikit-learn: cross decomposition and PLSRegression.