🎓 Deep Learning in Economics I

Stanford Graduate School of Business

📚 Course Structure

🌅 Part 1 (Morning): Machine Learning

Click to expand details...

Deep learning tools for unstructured data
Text, image, and audio processing
Practical implementation techniques
Real-world economic applications

🌆 Part 2 (Afternoon): Econometrics

Click to expand details...

Measurement error handling
Statistical inference with ML predictions
Proper econometric methodology
Research best practices

🎯 Focus: Practical examples with online technical resources

🚀 The Deep Learning Revolution

💡 Core Innovation

Deep learning revolutionizes processing of unstructured data that is:

📏

Too High-Dimensional

Raw text, images with millions of features

🧩

Too Complex

Intricate patterns humans can't easily code

❓

Uninterpretable

Raw pixel data, audio waveforms

🔄 Transformation Process

📄 Raw Data → 🤖 Deep Learning → 📊 Computable Info

🎯 Applications

Economics: Sentiment analysis, topic extraction
General: Self-driving cars, medical diagnosis
Impact: Large-scale accurate extraction

🤔 Quick Check: What's the main challenge with raw unstructured data?

It's too small to analyze

It's too high-dimensional and complex for direct analysis

It doesn't contain useful information

📊 Types of Unstructured Data

📝

Text Data

News articles, social media, documents

Largest source of information

🖼️

Image Data

Historical documents, satellite imagery, social media images

🎵

Audio/Video

Video = images + audio
Podcasts, interviews, speeches

🔗 Unified Methodology

Key Insight: Similar neural network architectures work across all data types!

🏗️ Transformer Architecture

Data Type	Architecture	Example
📝 Text	Text Transformers	GPT, BERT
🖼️ Images	Vision Transformers	ViT, DALL-E
🎵 Audio	Audio Models	Pre-trained on images!

🎯 Goal: Learn mapping from complex, diverse data to processable format

🔄 Data Representation & Dimensionality Reduction

⚠️ The Challenge

📝 Raw Text

High-dimensional sparse vectors
(vocabulary size: 50,000+)

🖼️ Raw Images

RGB pixel data
(width × height × 3)

❌ Difficult to compute with directly!

✅ Deep Learning Solution

🎯 Continuous Vector Space

Maps complex objects into embeddings

Lower-dimensional vectors (e.g., 512 dimensions)
Continuous numbers in each position
Much easier to work with

🔬 Core Concept: Learning continuous vector representations from examples

No explicit programming required! 🤖

🔍 Example: Word to Vector

Click to see transformation...

Input	Traditional	Deep Learning
"economics"	[0,0,0,1,0,0...] (sparse)	[0.2, -0.5, 0.8, 0.1...] (dense)

⚡ Why Neural Networks are Powerful

🔄

1. Transfer Learning

Leverage massive-scale training data

Models see trillions of text tokens
Internet compressed into billions of parameters
Question: Is information loss relevant to your problem?

🧠

2. Contextual Understanding

Context changes everything!

"Bank" in "river bank" vs "investment bank"
Transformer architecture excels at context mapping
Words/pixels get different representations based on surroundings

📈

3. Scalability

Unprecedented computational scale

Raw data is unwieldy at scale
Optimized tools for vector computations
Modern GPUs enable massive parallel processing

🏗️ The Architecture Advantage

Transformers unify diverse tasks:

📝 Text → 🧠 Transformer → 🎯 Task-specific output

🖼️ Images → 🧠 Transformer → 🎯 Task-specific output

🤔 Which is NOT a key advantage of neural networks?

Transfer learning from massive datasets

Contextual understanding

Perfect accuracy on all tasks

Computational scalability

🔤 Context vs. Bag-of-Words

📚 Traditional NLP (Bag-of-Words)

❌ Limitations

Each word mapped independently
Context completely ignored
Same word = same representation

Example:
"Bank" always represented the same way in:
• "River bank" 🏞️
• "Investment bank" 🏦

🧠 Neural Networks (Deep Learning)

✅ Advantages

Raw text fed into complex function
Context of each term considered
Dynamic representations

Example:
"Bank" gets different vectors in:
• "River bank" → [0.1, 0.8, -0.2...] 🏞️
• "Investment bank" → [0.7, -0.1, 0.9...] 🏦

🎯 Key Takeaway

Deep neural networks excel at estimating really complicated functions

That's essentially what a neural network is! 🧮

🔬 Interactive Demo

Click to see context in action!

💻 Computation Revolution in Economics

📅 Historical Perspective
                        🖥️ 1990s Revolution
                        Personal computing emergence
Economics: Theoretical → Empirical
Spreadsheets and databases

                    

                        🚀 Today's Revolution
                        GPU computing power
Cloud platforms (Azure, AWS)
New types of questions possible

                    

💰

Cost-Effectiveness

Thousands of CPUs for small cost

📊

Scale

Process data at unprecedented scales

🔬

New Research

Questions unimaginable 15 years ago

💡 Example Transformation

What used to require expensive supercomputers now runs on:

Only university clusters

Commodity hardware and cloud platforms

Personal laptops only

🎯 Neural Networks in Economic Research

🎯 Primary Application

Making predictions from unstructured data

📰 Framework Example

Click to see detailed example...

Component	Details
📊 Dataset	50 million newspaper articles
🎯 Goal	Identify economic policy uncertainty discussions
🔢 Output	Binary classification (0/1 value)
🤖 Method	Deep neural networks impute missing information

🎯 Main Focus

📏

Measurement

Measuring specific attributes within unstructured data (text/images)

🔮 Other Applications

Approximating solutions to NP-hard problems
Applications in economic theory
Optimization and game theory

📋 Three Types of Measurement Problems

📈

1. Regression Problems

Task: Predict continuous numerical outcomes

Example: Locating street vendors in satellite images

Predict GPS coordinates
Measure economic activity
Output: (latitude, longitude)

🏷️

2. Classification Problems

Task: Assign to known, discrete categories

Example: Economic policy uncertainty detection

Classes are prespecified
Multi-class or multi-label variants
Output: Yes/No or Category A/B/C

🕸️

3. Embeddings/Unsupervised

Task: Discover relationships without prespecified classes

Use case: Exploring large datasets

Find patterns and topics
No predetermined categories
Work with vector representations

🤔 Which Problem Type?

The choice depends on:

Known categories: Classification
Continuous values: Regression
Unknown patterns: Embeddings

🏗️ Neural Network Architecture

🔄 Basic Process

📄 Unstructured Data → 🧠 Neural Network → 🎯 Embeddings

Low-dimensional, dense, contextualized vector representations

🎯 Supervised Learning

+ Classification Layer

Maps embeddings → class probabilities
For regression: continuous outputs
Generative AI: predicts next token

Flow: Text → Embeddings → Classifier → Prediction

🕸️ Unsupervised Learning

Direct Embeddings

Work directly with vector representations
No classification layer added
Often more advantageous

Flow: Text → Embeddings → Analysis

🤔 When might you choose embeddings over classification?

When you have clearly defined categories

When exploring data without knowing what patterns exist

When you need yes/no answers

🤔 Classification Decision Framework

❓ Q1: Are classes known a priori?

❌ NO → Use Embeddings

Can't classify unknown classes!

✅ YES → Continue to Q2

❓ Q2: How many classes?

📊 Many (100,000+) → Consider Embeddings

Computationally unwieldy to train classifier

🔢 Few → Continue to Q3

❓ Q3: Data domain and granularity?

💡 Recommendation

Always experiment with generative AI first - it's cheap to try! 💰

🔧 Traditional Classification with Transformers

⚙️ Process Overview
                        1️⃣ Text → Vectors (per token)
                    
                        2️⃣ + Classifier Layer
                    
                        3️⃣ → Class Probabilities

🤖 BERT Example

Click to explore BERT architecture...

🧠 Backbone

Same pre-trained model for multiple tasks
Token embeddings with context
Dense vectors for each word/subword

🎯 Task Layers

Text Classification: Classifier on "class token"
Relationship: Two texts + "SEP" token
NER: Token-level classification

🔑 Key Advantage: Unified architecture for diverse tasks!

📝

Text Classification

Entire text → single prediction

🔗

Relationship Classification

Two texts → relationship type

🏷️

Token Classification

Each word → individual label

⚠️ Classifier Probability Interpretation

⚠️ Important Caveat

Classifier "probabilities" are NOT statistical probabilities!

🎯 Model Behavior

Trained to assign highest weight to most likely class
Changing normalization alters "probabilities"
Goal is relative ranking, not absolute probability

🔬 Research Implications

Critical limitation for econometric analysis
Proper statistical inference requires care
Part 2: How to account for measurement error

🤔 Why is this important for economists?

The probabilities are always wrong

We need to account for measurement error in further analysis

Neural networks don't work for economics

📊 Interactive Demo

Click to see probability interpretation in action!

Model Output	Interpretation
[0.8, 0.2]	Class A more likely than Class B
[0.6, 0.4]	Class A still more likely, but less confident
NOT: 80% chance	True statistical probability

🎓 Thank You!