---
title: "Dr. Hu's R Workshop"
output: 
  learnr::tutorial:
    progressive: true
    allow_skip: true
runtime: shiny_prerendered
---


```{r setup, include=FALSE}
library(learnr)
library(tidyverse)
library(gapminder)
```

# Lecture II: Data Munging

## Skills

1. Generalization: Arrange, count, summarise
1. Extraction: Filter, select, mutate

## Abilities You'll Achieve

1. Be the "family doctor" to your dataset
1. Store the valuable data easily! And quickly!

## Your "Patient" Today

Demographic statistics popularized by [Hans Rosling's TED talks](https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen?utm_campaign=tedspread--b&utm_medium=referral&utm_source=tedcomshare).

```{r toy, exercise = TRUE}
library(gapminder)
gapminder
```

## Generalization

### Knowing about the Data

```{r glimpse, exercise = TRUE}
head(gapminder, n = 6)

## Systemic view

str(gapminder)
```

### Knowing professionally

Hey, I'm a professional~ I wanna see the data in a systematic way, such as finding out 

* Size of the observations
* Number and names of the variables included
* Or the structure of the entire data frame

```{r systemView, exercise = TRUE}
gapminder
```

```{r systemView-solution}
nrow(gapminder)
ncol(gapminder)
names(gapminder)
str(gapminder)
```


### Knowing about the Variable

Q: Tell me something about the population variable in the dataset, like, how many countries' population we have, what the average, who has the largest and smallest population, and many other things! Btw, what type the `pop` is stored?

```{r variable, exercise = TRUE}
head(gapminder$year, n = 10)
```

```{r variable-solution}
mean(gapminder$year, na.rm = TRUE)
median(gapminder$year)
min(gapminder$year)
max(gapminder$year)
length(gapminder$year)
summary(gapminder$year)

class(gapminder$gdpPercap)
typeof(gapminder$gdpPercap)
```

## Fancier Moves 

Welcome to the Tidyverse

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/tidyverseHive.png' width = '100%'>

Prevalent toolkit for data manipulation 

+ A Hadley package
+ A growing [set](https://www.tidyverse.org/packages/) of packages, actually! 

Installation:

```{r loadTidy, exercise = TRUE}
## install.packages("tidyverse")
library("tidyverse")
```

We focus on `dplyr` today.

## Five Guns of `dplyr`

They do one thing, but they do it well.

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/simple.png' width = '95%'>

### Composability: Make Everything Different
<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/composable.png' width = '95%'>

Making codes more readable.

Shortcut for `%>%`: 

* Ctrl + Shift + M (Win)
* Cmd + Shift + M (Mac)

## Data Overview

### Beyond the base 

You still remember `str()`, right?

```{r overview, exercise = TRUE}
str(gapminder)
glimpse(gapminder)
```

### View in Order

Q: Which countries have the largest populations? And the smallest?

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/arrange.png' width = '95%'>

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/desc.png' width = '95%'>

```{r ex_arrange, exercise = TRUE}
gapminder
```

```{r ex_arrange-solution}
gapminder %>% 
  arrange(pop)

arrange(gapminder, desc(pop))
```

### "Give me some numbers"!

Q: How many observations do we have in each continent? Do we have same number of observations in each countries in the same continent?

```{r ex_count, exercise = TRUE}
gapminder %>% 
  count(continent)

# gapminder %>% 
#   add_count(continent)
```

```{r ex_count-solution}
gapminder %>% 
  count(continent, country)
```

What does `count()` give?

### Stats to variables

Q: What was the average GDP per capita and median life expectancy?

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/summarise.png' width = '95%'>


```{r ex_summary, exercise = TRUE}
gapminder %>% 
  summarise(mean_gdp = mean(gdpPercap), median_life = median(lifeExp))
```

### Summarise in groups

Q: What was the average GDP per capita and median life expectancy *in each continent*?

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/group_by.png' width = '95%'>


```{r ex_summaryG, exercise = TRUE}
gapminder %>% 
  group_by(continent) %>% 
  summarise(mean_gdp = mean(gdpPercap), median_life = median(lifeExp))
```


## Extraction

### Rows

Q: Which countries had the largest population in *2007*?

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/filter.png' width = '95%'>


```{r ex_filter, exercise = TRUE, exercise.eval = TRUE}
gapminder %>% 
  arrange(desc(pop))
```

```{r ex_filter-solution}
gapminder %>% 
  filter(year == 2007) %>% 
  arrange(desc(pop))
```

How about which country had the largest population in the decade ending with 2007? (Tip: using `%in%` as a condition)

### Columns

Q: If I want

1. Only country, year, and population
2. Everything but not continent
3. Variables starting with "co"

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/select.png' width = '95%'>

```{r ex_select, exercise = TRUE}
gapminder %>% 
  select(country, year, pop)
```

```{r ex_select-solution}
gapminder %>% 
  select(-continent)

gapminder %>% 
  select(starts_with("co"))
```

### Combo Attack

Q: What's the life expectancy of the country that had the largest population in 2007---showing the country name, population, and life expectancy together, please?

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/comboAttack.gif' width = '95%'>

```{r ex_combo, exercise = TRUE}
gapminder
```

```{r ex_combo-solution}
gapminder %>% 
  filter(year == 2007) %>% 
  arrange(desc(pop)) %>% 
  select(country, pop, lifeExp)
```

### Modification

Q: What's the total GDP of each country?

<img src='https://raw.githubusercontent.com/sammo3182/drhur/master/inst/tutorials/munging/images/mutate.png' width = '95%'>

```{r ex_mutate, exercise = TRUE}
gapminder %>% 
  mutate(gdp = pop * gdpPercap) %>% 
    select(country, pop, gdpPercap, gdp)
```

### Batch Modification

Q: How do we only keep the integers for all the numeric variables?

```{r ex_batch, exercise = TRUE}
gapminder %>% 
  mutate_if(is.double, round, digits = 0)
```

### Important note

When doing `gapminder %>% ...`, you are <span style="color:red">NOT</span> adding or changing anything of the `gapminder`.
If you want to save the changes, send the result to an object.

```{r eval = FALSE}
gapminderNew <- gapminder %>% ...
```

## Take-Home Points

1. Be clear about the logic <span style="color:red">before</span>  the moves;
1. Use the `dplyr` functions wisely and in combo;
    + Generalization: `arrange`, `count`, `summarise`
    + Extraction: `filter`, `select`, `mutate`
1. Don't forget `group_by` and `mutate_if`

## Bonus: Combining variables

```{r df_coalesce, echo = FALSE}
df_toy <- data.frame(x = sample(c(1:2, NA, NA, NA)),
                     y = c(1, 2, NA, NA, 5),
                     z = c(NA, NA, 3, 4, 5))
```

Q: I want to fill the missing in the `x`, and combine `y` and `z` to one variable? 

```{r coalesce, exercise = TRUE}
df_toy %>% 
  mutate(x = coalesce(x, 0L),
         yz = coalesce(y, z))
```


## Thank you!

<i class="fa fa-envelope fa-lg"></i>&nbsp; [yuehu@tsinghua.edu.cn](mailto:yuehu@tsinghua.edu.cn)

<i class="fa fa-globe fa-lg"></i>&nbsp; https://sammo3182.github.io/

<i class="fab fa-github fa-lg"></i>&nbsp; [sammo3182](https://github.com/sammo3182)
