🎯 Logistic Regression vs LDA vs QDA

Logistic Regression

What it is:

Models the probability of class membership directly using a logistic function. It's a discriminative classifier.

Decision Boundary:

Linear (straight line/plane)

Key Assumptions:

None about feature distributions. Assumes linear relationship between features and log-odds.

✅ Pros:

  • No distributional assumptions
  • Flexible and robust
  • Easy to interpret
  • Works well with large datasets

❌ Cons:

  • Only linear boundaries
  • Can underperform if data follows Gaussian distribution

LDA (Linear Discriminant Analysis)

What it is:

Models the distribution of features for each class, then uses Bayes' theorem. It's a generative classifier.

Decision Boundary:

Linear (straight line/plane)

Key Assumptions:

Features are normally distributed. All classes share the same covariance matrix.

✅ Pros:

  • More stable with small samples
  • Efficient when assumptions hold
  • Can handle multiple classes naturally
  • Good for dimensionality reduction

❌ Cons:

  • Strong assumptions about data
  • Sensitive to outliers
  • Only linear boundaries

QDA (Quadratic Discriminant Analysis)

What it is:

Like LDA but allows each class to have its own covariance matrix. Also a generative classifier.

Decision Boundary:

Quadratic (curved)

Key Assumptions:

Features are normally distributed. Each class has its own covariance matrix.

✅ Pros:

  • Can model curved boundaries
  • More flexible than LDA
  • Better when classes have different spreads

❌ Cons:

  • Needs more parameters (more data required)
  • Risk of overfitting with small datasets
  • Still assumes normality

Quick Comparison

Feature Logistic Regression LDA QDA
Type Discriminative Generative Generative
Boundary Linear Linear Quadratic
Covariance N/A Shared across classes Different per class
Parameters Fewest Moderate Most
Best when... No clear distribution or large n Gaussian data, equal variance Gaussian data, different variance
Small sample? Good Better Risky