Chapter 5 · Model Selection & Regularisation

ISLR Chapter 6 — what to do when you have more variables than judgement

Least squares has no opinion about how many predictors you feed it, and that is the problem. As \(p\) grows towards \(n\), variance grows with it; past \(n\) the estimator stops existing. Modern economic data — administrative registers, text, high-frequency series — arrives wide by default, so this chapter is less an advanced topic than a survival skill.

Three strategies. Subset selection keeps some variables and discards the rest. Shrinkage keeps all of them but pulls the coefficients toward zero, trading a little bias for a large reduction in variance; ridge shrinks smoothly, the lasso shrinks some coefficients exactly to zero and so selects as it shrinks, and elastic net splits the difference. Dimension reduction replaces the predictors with a smaller number of constructed directions.

What you should take away

  • Best subset, forward and backward stepwise selection, and their computational limits
  • \(C_p\), AIC, BIC and adjusted \(R^2\) as analytic stand-ins for test error
  • Ridge regression: the \(\ell_2\) penalty, and why standardising predictors is not optional
  • The lasso: the \(\ell_1\) penalty, sparsity, and the geometry that produces it
  • Elastic net, and why correlated predictors need it
  • Choosing \(\lambda\) by cross-validation, and the one-standard-error rule
  • Principal components regression and partial least squares

Slides

Labs

  • ch6.r — subset selection, ridge and lasso with leaps and glmnet
  • ch6.rmd — the same lab written up in R Markdown
  • ch6-lab.txt — lab transcript

Rendered output: Chapter 6 lab

Readings

TipStandardise first, always

Both penalties are scale-dependent. A predictor measured in rupees and one measured in millions of rupees receive very different amounts of shrinkage for no statistical reason. glmnet() standardises by default; sklearn.linear_model.Ridge does not. Know which one you are using.

Exercises

Fit ridge and lasso to the same wide dataset and plot both coefficient paths against \(\log \lambda\). Then answer, in a paragraph: at the cross-validated \(\lambda\), how many coefficients has the lasso set to exactly zero, and would you defend that selection to a colleague who believes those variables matter?

Back to top