Chapter 3 · Classification

ISLR Chapter 4 — when the outcome is a label, not a number

Default or not. Approved or declined. Poor or non-poor. A great deal of applied economics lands on a categorical outcome, and squared error stops making sense the moment it does. This chapter builds four ways of drawing a boundary through the predictor space, and one discipline for deciding which boundary to trust.

Logistic regression models the log-odds directly and stays interpretable. Linear and quadratic discriminant analysis run the argument backwards through Bayes’ rule, assuming Gaussian predictors within each class. \(k\)-nearest neighbours assumes nothing at all and pays for that freedom in variance. Which wins depends entirely on how the true boundary is shaped, which is exactly what you do not know in advance.

What you should take away

• Why linear probability models leak outside \([0, 1]\) and why it matters
• The logistic function, maximum likelihood, and interpreting odds ratios
• LDA and QDA as Bayes classifiers under Gaussian assumptions
• The bias–variance trade-off in \(k\)-NN as \(k\) moves from 1 to \(n\)
• Confusion matrices, sensitivity, specificity, and the ROC curve
• Why accuracy is a poor headline number under class imbalance

Slides

• Logistic Regression (PPTX)
• LDA and QDA (PPTX)

Labs

R R

• ch4.r — the full Chapter 4 lab
• logistic-regressio-ch4.r — logistic regression on Smarket
• lda-ch4.r — linear discriminant analysis
• qda.r — quadratic discriminant analysis
• knn.r — \(k\)-nearest neighbours, training versus test error
• knn-application-to-canvas-data.r — \(k\)-NN applied to class data

Comparison write-ups: logistic vs LDA vs QDA · version 2 · alternate

Python Python

• ch4-lab-py.qmd — the Chapter 4 lab in Python, as Quarto
• knn.py — \(k\)-NN with scikit-learn
• ch04-classification-lab.ipynb — the ISLP notebook

Rendered output: Python Chapter 4 lab · ISLP classification notebook

Readings

• Classification (PDF)
• k-NN: training versus test error (PDF)

Data

• who.csv — WHO country indicators, used in the classification exercises
• hmda-aer.dta · hmda-sw.dta — mortgage application data

TipThe threshold is a policy choice

predict() hands you a probability; turning it into a decision requires a cutoff, and 0.5 is a convention rather than a result. Ask what a false positive costs relative to a false negative in the setting you care about, then choose the threshold that follows. Most of the damage done by classifiers in policy comes from skipping this step.