Introduction to Statistical Learning - Final Examination

Chapters 5, 6, 7, 8, and 10

Author

Your Name

Published

Invalid Date

```{css, echo=FALSE} .pagedjs_page { padding: 1in; }

h1 { color: #2C3E50; border-bottom: 2px solid #3498DB; padding-bottom: 10px; }

h2 { color: #34495E; margin-top: 20px; }

.mcq-question { margin-bottom: 20px; page-break-inside: avoid; }

.short-question { margin-bottom: 30px; page-break-inside: avoid; background-color: #f8f9fa; padding: 15px; border-left: 4px solid #3498DB; } ```

Instructions

Time Allowed: 180 minutes
Total Marks: 100

This examination consists of two sections:

  • Section A: Multiple Choice Questions (20 questions × 3 marks = 60 marks)
  • Section B: Short Questions (3 questions × variable marks = 40 marks)

Answer all questions. Show all working for Section B questions.

Section A: Multiple Choice Questions (60 marks)

Instructions: Select the best answer for each question.

Question 1. Which of the following best describes the purpose of k-fold cross-validation?

  1. To increase the size of the training dataset
  2. To estimate the test error of a statistical learning method
  3. To reduce the variance of predictions
  4. To eliminate the need for a validation set

Question 2. In the bootstrap method, approximately what fraction of observations are NOT included in a given bootstrap sample?

  1. 1/n
  2. 1/e ≈ 0.368
  3. 1/2
  4. 1/3

Question 3. The bias-variance trade-off in k-fold cross-validation suggests that:

  1. Larger k leads to higher bias and lower variance
  2. Larger k leads to lower bias and higher variance
  3. Larger k leads to both lower bias and lower variance
  4. k does not affect bias or variance

Question 4. Ridge regression addresses multicollinearity by:

  1. Removing highly correlated predictors
  2. Setting some coefficients exactly to zero
  3. Shrinking all coefficients toward zero
  4. Orthogonalizing the predictor matrix

Question 5. Which statement about Lasso regression is TRUE?

  1. It can only shrink coefficients, not set them to zero
  2. It performs variable selection by setting some coefficients to zero
  3. It requires all predictors to be on the same scale
  4. Both b and c are correct

Question 6. In ridge regression, as the tuning parameter λ increases:

  1. Model flexibility increases and training RSS decreases
  2. Model flexibility decreases and training RSS increases
  3. Coefficient estimates approach their least squares values
  4. The bias-variance trade-off does not change

Question 7. Principal Components Regression (PCR) differs from ordinary least squares by:

  1. Using only the first M principal components as predictors
  2. Performing variable selection automatically
  3. Requiring standardization of predictors
  4. Both a and c are correct

Question 8. A natural cubic spline with K knots is represented by:

  1. K basis functions
  2. K + 1 basis functions
  3. K + 2 basis functions
  4. K + 4 basis functions

Question 9. Which method would be most appropriate for modeling a non-linear relationship when you suspect different predictors have different functional forms?

  1. Polynomial regression
  2. Regression splines
  3. Generalized Additive Models (GAMs)
  4. Step functions

Question 10. A key advantage of regression splines over polynomial regression is:

  1. They are always more flexible
  2. They provide better behavior at the boundaries
  3. They require fewer basis functions
  4. They automatically select the degree of the polynomial

Question 11. In decision tree construction, which measure is NOT typically used for node splitting in classification trees?

  1. Gini index
  2. Cross-entropy
  3. Mean squared error
  4. Classification error rate

Question 12. Bagging (Bootstrap Aggregation) reduces:

  1. Bias only
  2. Variance only
  3. Both bias and variance equally
  4. Neither bias nor variance

Question 13. Random forests improve upon bagging by:

  1. Using a larger number of trees
  2. Randomly selecting a subset of predictors at each split
  3. Using deeper trees
  4. Pruning each tree more aggressively

Question 14. In boosting, each new tree is fit to:

  1. The original response values
  2. The residuals from the current model
  3. A bootstrap sample of the data
  4. A random subset of predictors

Question 15. Which tree-based method typically uses shallow trees (stumps)?

  1. Bagging
  2. Random forests
  3. Gradient boosting
  4. Single decision trees

Question 16. In a neural network, the activation function serves to:

  1. Initialize the weights
  2. Introduce non-linearity into the model
  3. Reduce overfitting
  4. Select important features

Question 17. Dropout is a regularization technique that:

  1. Removes less important input features
  2. Randomly sets a fraction of neurons to zero during training
  3. Reduces the number of hidden layers
  4. Decreases the learning rate over time

Question 18. Convolutional Neural Networks (CNNs) are particularly well-suited for:

  1. Time series forecasting
  2. Text classification
  3. Image recognition
  4. Structured tabular data

Question 19. The vanishing gradient problem in deep neural networks refers to:

  1. Gradients becoming too large during backpropagation
  2. Gradients becoming very small in early layers
  3. The loss function having multiple local minima
  4. The training data having missing values

Question 20. Recurrent Neural Networks (RNNs) differ from standard neural networks by:

  1. Having connections that form cycles
  2. Using convolutional layers
  3. Requiring less training data
  4. Being limited to binary classification

Section B: Short Questions (40 marks)

Question 1. (15 marks)

Suppose we have a dataset with n = 100 observations and p = 10 predictors. We want to compare three approaches: least squares regression, ridge regression, and lasso regression.

  1. (5 marks) Explain conceptually why we might prefer ridge or lasso regression over least squares regression in this setting.

  2. (5 marks) Describe a procedure to select the optimal tuning parameter λ for ridge regression using cross-validation. Include specific details about the steps involved.

  3. (5 marks) Suppose after fitting lasso regression with the optimal λ, you find that only 4 of the 10 predictors have non-zero coefficients. How would you interpret this result? What does this tell you about the relationship between the predictors and the response?

Space for answer:

Question 2. (12 marks)

Consider fitting a generalized additive model (GAM) to predict a continuous response Y using three predictors X₁, X₂, and X₃.

  1. (4 marks) Write down the general form of a GAM with these three predictors. Explain what each component represents.

  2. (4 marks) Describe two advantages and one disadvantage of using a GAM compared to a standard linear regression model.

  3. (4 marks) Suppose diagnostic plots suggest that the relationship between Y and X₁ is approximately linear, while the relationships with X₂ and X₃ are highly non-linear. How would you modify your GAM to account for this? Write the modified model.

Space for answer:

Question 3. (13 marks)

You are building a classification model to predict customer churn (binary outcome) using a dataset with 20 predictor variables. You are considering using random forests and boosting.

  1. (5 marks) Explain the key algorithmic difference between random forests and boosting. How does each method combine the predictions from multiple trees?

  2. (4 marks) Both methods have tuning parameters that need to be selected. For random forests, identify two key tuning parameters and explain their role. For boosting, identify two key tuning parameters and explain their role.

  3. (4 marks) After fitting both models, you find that the random forest has a test error rate of 12% and the boosting model has a test error rate of 10%, but the boosting model takes much longer to train. Discuss the trade-offs you would consider when choosing between these two models for deployment in a production environment.

Space for answer:

Answer Key (For Instructor Use Only)

Section A - Multiple Choice Answers

  1. b - Cross-validation estimates test error
  2. b - Approximately 1/e observations are excluded
  3. b - Larger k → lower bias, higher variance
  4. c - Ridge shrinks coefficients toward zero
  5. d - Lasso performs variable selection and requires scaling
  6. b - Higher λ → less flexibility, higher training RSS
  7. d - PCR uses M principal components and requires standardization
  8. d - Natural cubic spline with K knots has K + 4 basis functions
  9. c - GAMs allow different forms for different predictors
  10. b - Splines have better boundary behavior
  11. c - MSE is used for regression trees, not classification
  12. b - Bagging reduces variance
  13. b - Random forests randomly select predictors at each split
  14. b - Boosting fits to residuals
  15. c - Gradient boosting typically uses shallow trees
  16. b - Activation functions introduce non-linearity
  17. b - Dropout randomly zeros neurons during training
  18. c - CNNs excel at image recognition
  19. b - Vanishing gradient means gradients become small in early layers
  20. a - RNNs have cyclic connections

Section B - Marking Guidelines

Question 1: - Part (a): Award marks for mentioning bias-variance trade-off, handling multicollinearity, feature selection (for lasso) - Part (b): Award marks for describing data splitting, grid of λ values, computing CV error, selecting λ with minimum error - Part (c): Award marks for discussing variable selection, model interpretability, identifying important predictors

Question 2: - Part (a): Correct GAM form: Y = β₀ + f₁(X₁) + f₂(X₂) + f₃(X₃) + ε, explanation of smooth functions - Part (b): Advantages: flexibility, interpretability; Disadvantage: additive assumption - Part (c): Modified model using linear term for X₁ and smooth functions for X₂ and X₃

Question 3: - Part (a): Random forests: parallel independent trees with averaging; Boosting: sequential trees with weighted combination - Part (b): RF: number of trees (B), mtry; Boosting: number of trees (B), learning rate (λ), tree depth (d) - Part (c): Discussion of accuracy vs. computational cost, model interpretability, maintenance, real-time requirements

END OF EXAMINATION