Chapter 4 · Resampling Methods

ISLR Chapter 5 — spending data to buy an honest error estimate

You have one sample and two competing needs: to fit the model and to judge it. Use the same rows for both and the judgement flatters the fit. Resampling is the machinery for holding those needs apart.

Cross-validation splits the data repeatedly, fits on part and evaluates on the rest, and averages. The bootstrap does something different: it resamples with replacement to approximate the sampling distribution of any statistic, including ones with no closed-form standard error. Between them they cover most of what you need to say honestly how good a model is and how uncertain a number is.

What you should take away

• The validation-set approach, and why its estimate is both variable and pessimistic
• Leave-one-out cross-validation, and the shortcut formula for linear models
• \(k\)-fold cross-validation, and why \(k = 5\) or \(10\) is the usual compromise
• The bias–variance trade-off in the error estimate itself, not just the model
• Cross-validation for tuning versus cross-validation for reporting performance
• The bootstrap for standard errors when theory is unavailable or unbelievable

Slides

• Resampling Methods (PPTX)

Labs

R R

• ch5.r — the Chapter 5 lab
• ch5-lab.r — an expanded version
• cross-validation.r · cross-validation.qmd — LOOCV and \(k\)-fold, written up
• boot-statlearning.r — the bootstrap with the boot package
• bootstrapping.r · bootstrapping.qmd — bootstrap from first principles
• psx-cv.r · pxs-cv.qmd — cross-validation on Pakistan Stock Exchange returns
• psx-bootstrapping.qmd — bootstrapping PSX returns
• psx-clean-code.r — the data preparation behind the PSX labs

Rendered output: Bootstrapping · PSX cross-validation · PSX bootstrapping

Python Python

scikit-learn puts all of this behind cross_val_score, KFold and LeaveOneOut. The danger is that convenience: it is easy to cross-validate the wrong thing. Fit the pre-processing inside the fold, not before it.

• model-selection-code.py — cross-validation and model comparison in scikit-learn
• psx.py — the PSX example in Python

Readings

• Cross-validation and the bootstrap (PDF)
• Cross-validation and model selection (PDF)
• Week 5 solutions (PDF)

Data

• psx.csv · psx-102025.csv — Pakistan Stock Exchange series
• kmi30-clean.xlsx — KMI-30 index

ImportantThe most common mistake in this course

Selecting variables, tuning a parameter or standardising columns using the full dataset, and only then cross-validating. The held-out fold has already seen that information, and your error estimate is now fiction. Every choice made from the data belongs inside the resampling loop.