USAID’s Khyber Pakhtukhwa Revenue Mobilization (KPRM) Activity

Author

Zahid Asghar

Published

September 12, 2026

About Me

  • Ph.D (Economics) — Pakistan Institute of Development Economics, 2007
  • Director, School of Economics (2019-Jan-22)
  • Registrar(2020-2021), Quaid-i-Azam University, Islamabad
  • Specializations:
    • Applied Econometrics
    • Data Analyst
    • Development Economics
  • Research interests : data for policy, data and analytical skill development ,data for policy

Data Science

  • You go into data analysis with the tools you know, not the tools you need

  • The next few days are all about giving you the tools you need

    • Admittedly, a bit before you know what you need them for

  • We will extend them as we learn specific models

R

  • Free and open source

  • A very large community

    • Written by statisticians for statistics
    • Most packages are written for R first

  • Can handle virtually any data format

  • Makes replication easy

  • Can integrate into documents (with Quarto)

  • R is a language so it can do everything

    • A good stepping stone to learning other languages like Python

Excel (or Stata) Can’t Do This

ggplot(data = gapminder, 
       aes(x = gdpPercap,
           y = lifeExp,
           color = continent))+
  geom_point(alpha=0.3)+
  geom_smooth(method = "lm")+
    scale_x_log10(breaks=c(1000,10000, 100000),
                  label=scales::dollar)+
    labs(x = "GDP/Capita",
         y = "Life Expectancy (Years)")+
  facet_wrap(~continent)+
  guides(color = F)+
  theme_light()

Or This

The average GDP per capita is ` r dollar(mean(gapminder$gdpPercap)) ` with a standard deviation of ` r dollar(sd(gapminder$gdpPercap)) `.

The average GDP per capita is $7,215.33 with a standard deviation of $9,857.45.

Or This

library(leaflet)
leaflet() %>%
  addTiles() %>%
  addMarkers(lng = 73.0874, lat =33.6604 ,
             popup = "Avari Xpress Hotel, Islamabad")

Meet R and R Studio

R and R Studio

  • R is the programming language that executes commands

  • Could run this from your computer’s shell

    • On Windows: Command prompt
    • On Mac/Linux: Terminal

R and R Studio

  • R Studio1 is an integrated development environment (IDE) that makes your coding life a lot easier
    • Write code in scripts
    • Execute individual commands & scripts
    • Auto-complete, highlight syntax
    • View data, objects, and plots
    • Get help and documentation on commands and functions
    • Integrate code into documents with Quarto

R Studio — Four Panes

Ways to Use R Studio: Using the Console

  • Type individual commands into the console pane (bottom left)

  • Great for testing individual commands to see what happens

  • Not saved! Not reproducible! Not recommended!

Ways to Use R Studio: Writing a .R Script

  • Source pane is a text-editor

  • Make .R files: all input commands in a single script

  • Comment with #

  • Can run any or all of script at once

  • Can save, reproduce, and send to others!

Ways to Use R Studio: Quarto Documents

Getting to Know Your Computer

  • R assumes a default (often inconvenient) “working directory” on your computer

    • The first place it looks to open or save files

  • Find out where R this is with getwd()

  • Change it with setwd(path/to/folder)2

Avoid this Hassle with R Projects

  • A R Project is a way of systematically organizing your R history, working directory, and related files in a single, self-contained directory
  • Can easily be sent to others who can reproduce your work easily
  • Connects well with version control software like GitHub
  • Can open multiple projects in multiple windows

Avoid this Hassle with R Projects

  • In almost all cases, you simply want a New Project

  • For more advanced uses, your project can be an R Package or a Shiny Web Application

  • If you have other packages that create templates installed (as I do, in the previous image), they will also show up as options

Avoid this Hassle with R Projects

  • Enter a name for the project in the top field

    • Also creates a folder on your computer with the name you enter into the field

  • Choose the location of the folder on your computer

  • Depending on if you have other packages or utilities installed (such as git, see below!), there may be additional options, do not check them unless you know what you are doing

  • Bottom left checkbox allows you to open a new instance (window) of R just for this project (and keep existing windows open)

An Intro to Coding

Learning…

  • You don’t “learn R”, you learn how to do things in R

  • In order to do learn this, you need to learn how to search for what you want to do

Building Good Workflow Habits

  • I will show you the tools to make your workflow:
    • Reproducible
    • Computer- and Human-Readable (!)
    • Automated
    • All in one program

For Example

library(gapminder)
library(gganimate)
gapminder %>%
  filter(continent != "Oceania") %>%
ggplot(aes(x = gdpPercap,
           y = lifeExp,
           color = country,
           size = pop))+
  geom_point(alpha=0.3)+
    scale_x_log10(breaks=c(1000,10000, 100000),
                  label=scales::dollar)+
  scale_size(range = c(0.5, 12)) +
  scale_color_manual(values = gapminder::country_colors) +
    labs(x = "GDP/Capita",
         y = "Life Expectancy (Years)",
         caption = "Source: Hans Rosling's gapminder.org",
         title = "Income & Life Expectancy - {frame_time}")+
  facet_wrap(~continent)+
  guides(color = F, size = F)+
  theme_minimal(base_family = "Fira Sans Condensed")+
  transition_time(year)+
  ease_aes("linear")

Agenda : 5 Important verbs for handling data

View, glimpse, structure, head, tail

select() for column selection

filter() for data filtering

arrange() Data Ordering

mutate() Creating Derived Columns

summarise() Calculating Summary Statistics

group_by()

Gapminder dataset

library(tidyverse)
library(knitr)
library(kableExtra)
#install.packages("gapminder")
library(hrbrthemes)
library(viridis)
library(kableExtra)
options(knitr.table.format = "html")
library(plotly)
library(gridExtra)
library(ggrepel)

The dataset is provided in the gapminder library

library(gapminder)

gapminder %>% filter(country=="Sweden")%>%
  mutate(gdpPercap=round(gdpPercap,0), lifeExp=round(lifeExp,2))%>%kable()%>%
  kable_styling(bootstrap_options = "striped", full_width = F)
country continent year lifeExp pop gdpPercap
Sweden Europe 1952 71.9 7124673 8528
Sweden Europe 1957 72.5 7363802 9912
Sweden Europe 1962 73.4 7561588 12329
Sweden Europe 1967 74.2 7867931 15258
Sweden Europe 1972 74.7 8122293 17832
Sweden Europe 1977 75.4 8251648 18856
Sweden Europe 1982 76.4 8325260 20667
Sweden Europe 1987 77.2 8421403 23587
Sweden Europe 1992 78.2 8718867 23880
Sweden Europe 1997 79.4 8897619 25267
Sweden Europe 2002 80.0 8954175 29342
Sweden Europe 2007 80.9 9031088 33860

Information in gapminder data

View command opens data in new worksheet while glimpse lists nature of variables (numeric/character/factor…) and total number of rows and columns.

glimpse(gapminder) # We see that there are 1704 rows for 6 columns and also tells nature of variable
Rows: 1,704
Columns: 6
$ country   <fct> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanist…
$ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, A…
$ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 1…
$ lifeExp   <dbl> 28.8, 30.3, 32.0, 34.0, 36.1, 38.4, 39.9, 40.8, 41.7, 4…
$ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 1488037…
$ gdpPercap <dbl> 779, 821, 853, 836, 740, 786, 978, 852, 649, 635, 727, …
#View(gapminder)    # This opens up full data in a new window

Summary

summary(gapminder) 
##         country        continent        year         lifeExp    
##  Afghanistan:  12   Africa  :624   Min.   :1952   Min.   :23.6  
##  Albania    :  12   Americas:300   1st Qu.:1966   1st Qu.:48.2  
##  Algeria    :  12   Asia    :396   Median :1980   Median :60.7  
##  Angola     :  12   Europe  :360   Mean   :1980   Mean   :59.5  
##  Argentina  :  12   Oceania : 24   3rd Qu.:1993   3rd Qu.:70.8  
##  Australia  :  12                  Max.   :2007   Max.   :82.6  
##  (Other)    :1632                                               
##       pop             gdpPercap     
##  Min.   :6.00e+04   Min.   :   241  
##  1st Qu.:2.79e+06   1st Qu.:  1202  
##  Median :7.02e+06   Median :  3532  
##  Mean   :2.96e+07   Mean   :  7215  
##  3rd Qu.:1.96e+07   3rd Qu.:  9325  
##  Max.   :1.32e+09   Max.   :113523  
##

dplyr features

filter() to keep selected observations

select() to keep selected variables

arrange() to reorder observations by a value

mutate() to create new variables

summarize() to create summary statistics

group_by() for performing operations by group

Select()

Column Selection

For example PDHS files have more than 5000 columns in some files and maybe 40 or 50 or even fewer than that are needed for your analysis. Select() function of R’s dplyr is used to select columns of your interest

gapminder %>% select(country, pop, lifeExp)

select most of the variables and drop one or two

gapminder %>% select(-gdpPercap)

Data Filtering

filter() funtion

gapminder_07<- gapminder %>% filter(year==2007)
kbl(gapminder_07[1:10,])%>%kable_styling(fixed_thead=T)
country continent year lifeExp pop gdpPercap
Afghanistan Asia 2007 43.8 3.19e+07 975
Albania Europe 2007 76.4 3.60e+06 5937
Algeria Africa 2007 72.3 3.33e+07 6223
Angola Africa 2007 42.7 1.24e+07 4797
Argentina Americas 2007 75.3 4.03e+07 12779
Australia Oceania 2007 81.2 2.04e+07 34435
Austria Europe 2007 79.8 8.20e+06 36126
Bahrain Asia 2007 75.6 7.09e+05 29796
Bangladesh Asia 2007 64.1 1.50e+08 1391
Belgium Europe 2007 79.4 1.04e+07 33693

##>Have we accidently deleted all other rows? Answer is no.

Nope: If you don’t believe me try entering gapminder at the console.

gapminder %>% filter(year==2007)

Filtering with respect to two variables

One can apply multiple filters 

gapminder %>% filter(year==2007,country=="Sri Lanka")

gapminder %>% filter(year==2007, country=="Pakistan")

Now we are selecting multiple countries for year 2007.

gapminder %>% filter(year==2007, country %in% c("India", "Pakistan","Bangladesh", "Afghanistan", "Iran"))

Filtering data for South Asia countries

gapminderSA<-gapminder %>% filter(country %in% c("Bangladesh","India","Pakistan","Sri Lanka","Nepal", "Afghanistan","Bhutan", "Maldives"))
gapminderSA

Sort data with arrange() 

gapminderSA %>% arrange(gdpPercap)

arrange() sorts in ascending order. Sort in descending order, desc().

gapminderSA %>% arrange(desc(gdpPercap))

Life Expectancy in South Asia in 2007

What is the lowest and highest life expectancy among South Asian countries?

gapminderSA %>% filter(year==2007) %>%  arrange(lifeExp)

What was it in 1952?

mutate() to change existing or create new variable

gapminderSA %>% mutate(pop=pop/1000000)

If we want to calculate GDP, we need to multiply gdpPercap by pop.

But wait! Didn't we just change pop so it's expressed in millions?

No: we never stored the results of our previous command, we simply displayed them. Just as I discussed above, unless you overwrite it, the original gapminder dataset will be unchanged. With this in mind, we can create the gdp variable as follows:

gapminderSA %>% mutate(gdp = pop * gdpPercap)

How to calculate new variables

As mentioned above, mutate is used to calculate new variable. Here we have calculated a new variable gdp and then arrange() data and selected top_n(10) countries to see whether higher lifeExpectancy and higher gdp are linked or not?

gapminder %>% filter(year==2007) %>% 
  mutate(gdp=gdpPercap*pop) %>% 
  arrange(desc(gdp)) %>% 
  top_n(10)

transmute() keeps only the derived column. Let’s use it in the example from above:

gapminder %>% filter(year==2007) %>% 
  transmute(gdp=gdpPercap*pop) %>% 
  arrange(desc(gdp)) %>% 
  top_n(10)

Ordering

arrange data by life expectancy, we use arrange() function

gapminder %>% 
  select(country, year,lifeExp) %>% 
  filter(year==2007) %>% 
  arrange(lifeExp)

top to bottom, then use arrange(desc()) command as follows:

gapminder %>% 
  select(country, year,lifeExp) %>% 
  filter(year==2007) %>% 
  arrange(desc(lifeExp))

Top 5

gapminder %>% 
  select(country, year,lifeExp) %>% 
  filter(year==2007) %>% 
  arrange(desc(lifeExp)) %>% 
  top_n(5)

Summarising data

Another feature of dplyr is summarise data

gapminder %>% filter(year==2007) %>% group_by(continent) %>% summarise(mean=mean(lifeExp),min=min(lifeExp),max=max(lifeExp))
gapminder %>% 
  summarise(avglifeExp=mean(lifeExp))

Summarising data by groups

gapminder %>%
  filter(year == 2007, continent == "Asia") %>%
  summarize(avgLifeExp = mean(lifeExp)) 

gapminder %>% 
  group_by(continent) %>% 
  filter(year==2007) %>% 
  summarize(avglife=mean(lifeExp))

mean, maximum, and mean life expectancy

gapminder %>%
  summarise(
    lifeExp_min = min(lifeExp),
    lifeExp_max = max(lifeExp),
    lifeExp_mean = mean(lifeExp)
  )

LE for each year

gapminder %>%
  group_by(year) %>%
  summarise(
    lifeExp_min = min(lifeExp),
    lifeExp_max = max(lifeExp),
    lifeExp_mean = mean(lifeExp)
  )

##dplyr tutorial0

if_else command alongwith mutate

gapminder %>%
  filter(year == 2007) %>%
  group_by(continent) %>%
  summarize(avgLifeExp = mean(lifeExp)) %>%
  mutate(over75 = if_else(avgLifeExp > 70, "Y", "N"))

Total Population by Continets in 2007

gapminder %>% 
  filter(year==2007) %>% 
  group_by(continent) %>% 
  summarize(tot_pop=sum(pop))

Percentiles

gapminder %>% select(country,year, lifeExp, gdpPercap) %>% 
  filter(year == 2007) %>%
  mutate(percentile = ntile(lifeExp, 100)) %>%
  arrange(desc(gdpPercap))

Bottom side

So it makes sense that higher the GDP, higher the lifeExp. This is not formal testing but exploratory data makes lot of sense here.

gapminder %>% select(country,year, lifeExp, gdpPercap) %>% 
  filter(year == 2007) %>%
  mutate(percentile = ntile(lifeExp, 100)) %>%
  arrange(gdpPercap)

Advanced Analysis

Filtering data as done in introductory analysis seems quite difficult if you are not familiar with these simple things. But if you are working with dplyr for quite sometime, there is not anything very advanced or difficult.

For example, let’s say you have to find out the top 10 countries in the 90th percentile regarding life expectancy in 2007. You can reuse some of the logic from the previous sections, but answering this question alone requires multiple filtering and subsetting:

gapminder %>% filter(year==2007) %>% 
  mutate(percentile=ntile(lifeExp,100)) %>% 
  filter(percentile>90) %>% 
  arrange(desc(percentile)) %>% 
  top_n(10,wt=percentile) %>% 
  select(country,continent,lifeExp,gdpPercap)

In case you are interested in bottom 10 (worst lifeExp countries from the bottom), use top_n with -10.

gapminder %>% filter(year==2007) %>% 
  mutate(percentile=ntile(lifeExp,100)) %>% 
  filter(percentile<10) %>% 
  arrange(percentile) %>% 
  top_n(-10,wt=percentile) %>% 
  select(country,continent,lifeExp,gdpPercap)

Visualizing data to get data insight

Visualizing data is one of the most important aspect of getting data insight and may provide a better data insight than a complicated model. Visualizing large data sets were not an easy task, so researchers relied on mathematical and core econometric/regression models. ggplot2 which is a set of tidyverse package is probably one of the greatest tool for data visualization used in R. In the following sections we are going to visualize gapminder data.

Stat graphics is a mapping of variable to aesthetic attributes of geometric objects.

3 Essential components of ggplot2

  • data: dataset containing the variables of interest
  • geom: geometric object in question line, point, bars
  • aes: aesthetic attributes of an object x/y position, colors, shape, size

Scatter plot


gapminder2007<-gapminder %>% filter(year==2007)
p1<-ggplot(data=gapminder2007,mapping = aes(x=gdpPercap,y=lifeExp,color=continent,size=pop))+geom_point()
p1+facet_wrap(~continent)

p1+  labs(x = "GDP Per Capita", y = "Life Expectancy in Years",
       title = "Economic Growth and Life Expectancy",
       subtitle = "Data points are country-years",
       caption = "Source: Gapminder.")

Bubbleplot

##This is a table of data about a large number of countries, each observed over several years. Let's make a scatterplot with it.
P<-ggplot(data=gapminder,mapping = aes(x=gdpPercap,y=lifeExp))  

P+geom_point()+geom_smooth()


P+geom_point()+geom_smooth(method = "lm")


P+geom_point()+geom_smooth(method = "gam")+scale_x_log10()



P+geom_point()+geom_smooth(method = "gam")+scale_x_log10(labels=scales::dollar)



P<-ggplot(data=gapminder,mapping = aes(gdpPercap,y=lifeExp,color="purple"))
P+geom_point()+geom_smooth(method = "loess")+scale_x_log10()

With proper title

P<-ggplot(data=gapminder,mapping = aes(gdpPercap,y=lifeExp))
P+geom_point(alpha=0.3)+
  geom_smooth(method = "gam")+
  scale_x_log10(labels=scales::dollar)+
  labs(x = "GDP Per Capita", y = "Life Expectancy in Years",
       title = "Economic Growth and Life Expectancy",
       subtitle = "Data points are country-years",
       caption = "Source: Gapminder.")

p + geom_point(mapping = aes(color = log(pop))) +
  scale_x_log10()

Comparison of 2007 vs 1952 continentwise

p1+labs(x = "GDP Per Capita", y = "Life Expectancy in Years",
       title = "Economic Growth and Life Expectancy",
       subtitle = "Data points are country-years",
       caption = "Source: Gapminder.")

gapminder %>%  
  select(-pop, -gdpPercap) %>%  
  filter(year == 2007) %>%  
  group_by(continent) %>%  
  summarise(mean_life_exp =  
              mean(lifeExp),median_life_exp=median(lifeExp)) 

gapminder %>%  
  select(-pop, -gdpPercap) %>%  
  filter(year == 2007) %>%  
  group_by(continent) %>%  
  summarise(median_life_exp =  
              median(lifeExp)) %>%  
  ggplot() +  
  aes(x = continent) +  
  aes(y = median_life_exp) +  
  geom_point(color = "blue", size = 3,  
             alpha = .4) +  
  scale_y_continuous(limits = c(0,85)) +  
  labs(title = "Median life expectency by continent in 2007") +  
  labs(subtitle = "Data Source: Gapminder package in R") +  
  labs(x = "") +  
  labs(y = "Median life expectancy") +  
  labs(caption = "Zahid Asghar for 'QM4SSH'") +  
  theme_minimal()

gapminder %>%  
  filter(year == 1997) %>%  
  select(country, continent, lifeExp) %>%  
  mutate(  
    life_cats =  
      case_when(lifeExp >= 70 ~ "70+",  
                lifeExp < 70 ~ "<70")) %>%  
  ggplot() +  
  aes(x = continent, y = life_cats) +  
  geom_jitter(width = .25, height = .25)+  
  aes(col = continent) +  
  scale_color_discrete(guide = FALSE) +  
  theme_dark() +  
  labs(x = "", y = "") +  
  labs(title = "Life expectency beyond 70 in 1997")

gapminder %>%  
  mutate(gdp_billions =  
           gdpPercap *  
           pop/1000000000) %>%  
  ggplot() +  
  aes(x = year) +  
  aes(y = gdp_billions) +  
  geom_line() +  
  aes(group = country) +  
  scale_y_log10() +  
  aes(col = continent) +  
  facet_wrap( ~ continent) +  
  scale_color_discrete(guide = F) +  
  theme_minimal()

gapminder %>%  
  filter(year == 2007) %>%  
  ggplot() +  
  aes(x = continent, y = lifeExp) +  
  geom_boxplot() +  
  geom_jitter(height = 0, width = .2) +  
  stat_summary(fun.y = mean,
               geom = "point",
               col = "goldenrod3",
               size = 5)

Pakistan

gapminder %>%  
  filter(country == "Pakistan") %>%  
  ggplot() +  
  aes(x = year, y = lifeExp) +  
  geom_point() +  
  geom_line() +  
  aes(alpha = year) +  
  aes(col = year) +  
  scale_color_viridis_c() +  
  theme_classic()

gapminder %>%  
  filter(year == 2007) %>%  
  ggplot() +  
  aes(x = gdpPercap) +  
  aes(y = lifeExp) +  
  geom_point(alpha = .5) +  
  geom_rug(size = 1) +  
  aes(col = continent) +  
  aes(col = lifeExp) +  
  scale_x_log10() +  
  aes(size = gdpPercap) +  
  aes(size = pop) +  
  geom_point(col = "darkgreen", size = 1) +  
  facet_wrap(~ continent)

Back to top

Footnotes

  1. The company R Studio recently announced they will be rebranding later this fall as Posit.↩︎

  2. Note the path is OS-specific. For Windows it might be C:/Documents/. For Mac it is often your username folder.↩︎