Chapter 6 · Trees & Ensembles

ISLR Chapter 8 — recursive partitioning, and the wisdom of averaging many bad models

A regression tree does something a regression cannot: it splits the predictor space into boxes and predicts a constant in each. Interactions come free, non-linearity comes free, and the result draws on a whiteboard in thirty seconds. That interpretability is the selling point, and it is also the trap, because a single tree is unstable — shift a few observations and the whole structure rearranges.

The fix is to stop trusting any one tree. Bagging averages hundreds of them fitted to bootstrap samples. Random forests go further and hide most of the predictors at each split, which decorrelates the trees and cuts the variance of the average. Boosting grows trees slowly and sequentially, each one fitting what the last got wrong. You lose the whiteboard picture and gain, very often, the best predictions in the course.

What you should take away

• Recursive binary splitting, and why it is greedy rather than optimal
• Cost-complexity pruning, and choosing the penalty by cross-validation
• Gini index and cross-entropy as impurity measures for classification trees
• Bagging, out-of-bag error, and why it is nearly free cross-validation
• Random forests: the role of mtry and why decorrelation helps
• Boosting: learning rate, depth, number of trees, and how to stop early
• Variable importance, and why it should be read with suspicion

Labs

R R

• tree-ch8.r — the Chapter 8 lab: trees, bagging, forests, boosting
• cart.qmd · cart.rmd — CART written up with rpart
• cart-models.qmd — comparing tree-based models
• heart-disease-cart.qmd — classification trees on the heart disease data
• cart-hmda.r — mortgage approval, a fairness-adjacent application
• hmda-cart-analysis-fixed.r — the fuller HMDA analysis

Rendered output: CART · CART model comparison

Python Python

The scikit-learn equivalents are DecisionTreeRegressor, RandomForestRegressor and GradientBoostingRegressor, with xgboost for the tuned version. Work through the ISLP Chapter 8 notebook alongside the R lab; plot_tree gives you the whiteboard picture and permutation_importance gives you a more defensible importance measure than the built-in one.

Readings

• Trees (PDF)
• Chapter 8 lab output (HTML)
• Additional notes (TXT)

Data

• hmda-aer.dta · hmda-sw.dta — Home Mortgage Disclosure Act data
• ict-women.dta — ICT access and gender

WarningVariable importance is not a causal claim

A random forest will happily rank a variable highly because it proxies for something else, or because it has many possible split points. Importance tells you what the algorithm used, not what matters in the world. Say so explicitly whenever you present one of these plots to a policy audience, because they will read it as causal unless stopped.