---
title: "Dr. Hu's R Workshop"
output: 
  learnr::tutorial:
    progressive: true
    allow_skip: true
runtime: shiny_prerendered
---

```{r setup, include=FALSE}
library(learnr)
library(gapminder)
library(car)
library(arm)
library(summarytools)
library(performance)
library(nnet)
library(tibble)
library(tidyverse)
```

# Lecture III: Data Analysis

## Skills

1. Descriptive statistics:
    + Moments of a distribution
    + Cross-variable spreadsheet
1. Analytic statistics:
    + Binary analysis (t-test, correlation, ANOVA)
    + Multivariate analysis (OLS, MLE)

## Abilities You'll Achieve

1. Get whatever info about a variable and sparkling descriptive tables.
1. Run the common models and learn how to get more complex models.


## Toy Data

Demographic statistics popularized by [Hans Rosling's TED talks](https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen?utm_campaign=tedspread--b&utm_medium=referral&utm_source=tedcomshare).

```{r toy, exercise = TRUE}
library(tibble)
library(gapminder)

glimpse(gapminder)
```

## Let the Data Tell

Ah, rare data!

How JUICY they are!!

How...to let it tell?


## Descriptive Statistics

### Moments

1. Mean
1. Variance
1. Skewness
1. Kurtosis


```{r moments, exercise = TRUE, exercise.eval = TRUE}
library(tidyverse)
library(moments)

gapminder %>% 
  summarise(mean_life = mean(lifeExp),
            median_life = median(lifeExp),
            iqr_life = IQR(lifeExp), 
            #  IQR = quantile(x, 3/4) - quantile(x, 1/4)
            skew_life = skewness(lifeExp),
            kurtosis_life = kurtosis(lifeExp))

```

```{r moments-solution}
library(broom)

sum_OneVar <- summary(gapminder$lifeExp) %>% tidy
```

### Summary Table

```{r descriptive, exercise = TRUE}
library(summarytools)

freq(gapminder)
```

```{r descriptive-solution}
dfSummary(gapminder)
```

###

```{r descriptivePretty, eval = FALSE, include = FALSE}
print(dfSummary(gapminder, 
                graph.magnif = 0.3,
                trim.strings = TRUE,
                plain.ascii = FALSE,
                valid.col = FALSE,
                na.col = FALSE),
      method = 'render')
```

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/analyzing/images/dfSummary.png' width = '100%'>

## Analytic Statistics

### Binary analysis

How does one variable relate to another?

1. Are the averages close?
1. Are the two variables independent to each other?
1. Does variance matter?

### Cross Table

```{r crossTable}
library(summarytools)

gapminder$coldWar <- gapminder$year <= 1992
gapminder$asia <- gapminder$continent == "Asia"
ctable(gapminder$asia, gapminder$coldWar)
```


### Difference in mean

Q: Does the average life expectancy changes before and after the Cold War? 

$H_{0}: \bar{LifeExpctancy}_{prio 1991} = \bar{LifeExpctancy}_{post 1991},\ \alpha = .05$

```{r ttest, exercise = TRUE}
gapminder$coldWar <- gapminder$year <= 1992
gapminder %>% 
  group_by(coldWar) %>% 
  summarise(mean(lifeExp))
```

```{r ttest-solution}
t.test(gapminder$lifeExp[gapminder$year <= 1991], gapminder$lifeExp[gapminder$year > 1991])

t.test(gapminder$lifeExp[gapminder$year <= 1991], gapminder$lifeExp[gapminder$year > 1991], alternative = "greater", conf.level = .99)
```

### Correlation

Q: Does a country's average life expectancy associate with its GDP per capital? (How about with populations?)

$H_{0}: \rho_{(LifeExpectancy, GDP)} = 0,\ \alpha = .05$

```{r corr, exercise = TRUE}
cor.test(gapminder$lifeExp, gapminder$gdpPercap)
```

### More than Two Variables

```{r corrplot}
select(gapminder, year, lifeExp, gdpPercap, pop) %>% 
  cor %>% 
  corrplot::corrplot.mixed()
```

### ANOVA

Q: Does the average life expectancy vary across continents? (And over year?)

$H_{0}: \mu_{Africa} = \mu_{Americas} = ... = \mu_{Oceania},\ \alpha = .05$

```{r anova, exercise = TRUE}
aov(lifeExp ~ continent, data = gapminder) %>% 
  summary()
```

ANC(ovariate)OVA?

```{r anova-solution}
aov(lifeExp ~ continent + as.factor(year), data = gapminder) %>% 
  summary()
```


### OLS

Q: How does a country's average life expectancy change across continents and over time?

```{r ols, exercise = TRUE}
gapminder %>% count(lifeExp, continent, year)
```

```{r ols-solution}
lm(lifeExp ~ continent + year, data = gapminder) %>%
  summary()
```

### Variable transformation

Controlling for time dependency?

$t + t^2$

```{r time, exercise = TRUE}
lm(lifeExp ~ gdpPercap + year, data = gapminder) %>%
  summary()
```


```{r time-solution}
lm(lifeExp ~ gdpPercap + year + I(year^2), data = gapminder) %>%
  summary()
```

Other transformations

`sqrt(var)`, `log(var)`

### Standardization

```{r standardized}
library(arm)

lm(lifeExp ~ gdpPercap + year + I(year^2), data = gapminder) %>%
  standardize() %>% 
  summary()
```


### Conditional effect

Does the contribution of GDP per captia to a country's life expectancy vary with different sizes of population?

```{r interact, exercise = TRUE}
lm(lifeExp ~ gdpPercap + year, data = gapminder) %>%
  summary()
```


```{r interact-solution}
lm(lifeExp ~ gdpPercap * pop + year, data = gapminder) %>%
  summary()
```

Hint: You must include both base terms in the model.

### Post-estimate diagnoses

+ Residual
+ Outlier
+ Heteroscedasticity
+ Multicollinearity
+ Autocorrelation

......

### Residual

```{r residual, exercise = TRUE}
fit <- lm(lifeExp ~ gdpPercap + year, data = gapminder)
res <- resid(fit)
```

```{r residual-solution}
plot(fit, which  = 1)
```

### Outlier

```{r outlier, exercise = TRUE}
library(car)
outlierTest(fit)
# car::qqPlot(fit)
```

### Heteroscedasticity

```{r heter, exercise = TRUE}
ncvTest(fit) 
```

### Multicollinearity

```{r multi, exercise = TRUE}
vif(fit)
```

### Autocorrelation

```{r durbin, exercise = TRUE}
durbinWatsonTest(fit)
```

### Cool shortcut

```{r performance, exercise = TRUE}
library(performance)

model_performance(fit)
check_normality(fit)
check_outliers(fit)
```


### Logit

* Logit: Does GDP per capita determine if a country's average life expentacy is above the global average after controlling for the population size?

```{r logit, exercise = TRUE}
glimpse(mtcars)

glm(vs ~ gear + wt + mpg, data = mtcars, family = "binomial") %>%
  summary()
```

### Interpretation

```{r margin, exercise = TRUE}
glm(vs ~ gear + wt + mpg, data = mtcars, family = "binomial") %>% 
  ggeffects::ggpredict()
```

### Ordered Logit

```{r ologit, exercise = TRUE}
polr(as.factor(gear) ~ wt + mpg, data = mtcars) %>% 
  summary()
```

```{r ologit-solution}
library(ordinal)

clm(as.factor(gear) ~ wt + mpg, data = mtcars) %>% 
  summary()
```


### Multinomial 

```{r mlogit, exercise = TRUE}
library(nnet)

multinom(as.factor(cyl) ~ wt + mpg, data = mtcars) %>% 
  summary()
```


### Poisson

```{r poisson, exercise = TRUE}
df_poisson <- data.frame(counts = c(18, 17, 15, 20, 10, 20, 25, 13, 12),
                         outcome = gl(3, 1, 9), 
# Generate factors by specifying the pattern of their levels.
                         treatment = gl(3, 3))

glm(counts ~ outcome + treatment, 
               family = poisson(), 
               data = df_poisson) %>% 
  summary()
```


## Wrap Up

1. Descriptive: 
    + `ctable`
    + `cor`
    + `aov`
2. Analytic: `lm/glm/clm/multinom(Y ~ Xs, data)`

## Thank you!

<i class="fa fa-envelope fa-lg"></i>&nbsp; [yuehu@tsinghua.edu.cn](mailto:yuehu@tsinghua.edu.cn)

<i class="fa fa-globe fa-lg"></i>&nbsp; https://sammo3182.github.io/

<i class="fab fa-github fa-lg"></i>&nbsp; [sammo3182](https://github.com/sammo3182)