---
title: "eiCompare: Parallel Processing"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{eiCompare: Parallel Processing}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  warning = FALSE,
  comment = "#>"
)

library(eiCompare)

data("corona")
```

This vignette aims to highlight the parallel processing capabilities within eiCompare. Functions that include this option are:

- `ei_iter()`

- `ei_rxc()` (only for diagnostic)

- `run_geocoder()`

Prior to attempting to run these functions in parallel, it is advised you check your computer or server for the following properites:

- You should more than 4 physical cores

- You should have at least 16 GB of RAM


## Introduction to parallel processing

Building off of exisiting parallel processing packages such as `foreach` and `doSNOW`,  this package includes parallel processing capabiltiies to speed up ecological inference analyses.

_Parallel processing_ decreases the time needed for your processes by splitting the job amongst your computer's CPU cores. We recommend 16 GB of RAM so that R can store the data you are currently working on. Furthermore, if you are using multiple cores, the minimum RAM needed is the product of the number of cores you're using and the size of your data.So if you are working on 4 cores and your dataset is 1 GB, you'll be using at least 4 GB of RAM.

In order to make this functionality more accessible to users, eiCompare's functions that include parallel processing include a check for the number of cores you have available to you. If you have less than 4 cores, our functions will not let you proceed with parallelization. Even with exacltly 4 cores, the functions will return a warning that parallelization is not recommended. 

There are many resources online if you'd like to learn more about parallelization in general.

## Walking through an example: `ei_iter()`

In this vignette, we'll be focusing on `ei_iter()` (the functionalities and work flow are described in detail in the Ecological Inference tutorial). We recommend that you review this vignette prior to attempting parallelization for this function. 

The data we'll be using for this example is from 2014 elections in California, specifically looking at voting results and racial demogrphiacs for Corona by precinct.

```{r}
head(corona)
```

We have a row for every precinct, if we check the dimensions of our dataset you will see that this is 46. We also have 12 variables included in this dataset.
```{r}
print(dim(corona))
names(corona)
```

The variables are as follows:
  - `precinct`: Precinct ID number     
  
  - `totvote`: Total number of votes cast
  
  - `pct_husted`: Percent of voting precinct population who voted for Husted
  
  - `pct_spiegel`: Percent of voting precinct population who voted for Spiegel
  
  - `pct_ruth`: Percent of voting precinct population who voted for Ruth   
  
  - `pct_button`:  Percent of voting precinct population who voted for Button 
  
  - `pct_montanez`: Percent of voting precinct population who voted for Montanez
  
  - `pct_fox`: Percent of voting precinct population who voted for Fox     
  
  - `pct_hisp`: Percent of voting precinct population who identify as Hispanic 
  
  - `pct_asian`: Percent of voting precinct population who identify as Asian  
  
  - `pct_white`: Percent of voting precinct population who identify as White 
  
  - `pct_non_lat`: Percent of voting precinct population who identify as Non-Latino
  
  

Non-Latino encompasses the Asian and White voting population.
```{r}
corona$pct_hisp + corona$pct_non_lat == 1
```

So for this analysis there are 6 candidates (Husted, Spiegel, Ruth, Button, and Montanez) and 3 racial groups (Hispanic/Latino, Asian, and White). With that, let's set up the inputs we need for the function and time it to see how long it takes to complete the iterative ei analysis without parallelization.

Note: Some lines of code below are commented out to limit testing time. To run this demo, uncomment them.

```{r eval=FALSE}
cand_cols <- c("pct_husted", "pct_spiegel", "pct_ruth", "pct_button", "pct_montanez", "pct_fox")
race_cols <- c("pct_hisp", "pct_asian", "pct_white")
totals_col <- "totvote"

# Run without parallization
#start_time <- Sys.time()
#results_test <- ei_iter(corona, cand_cols, race_cols, totals_col)
#(end_time <- Sys.time() - start_time)
```

To run in this parallel, all you need to do is toggle `par_compute` to be TRUE.
```{r eval=FALSE}
# Run with paralleization
#start_time <- Sys.time()
#results_test <- ei_iter(corona, cand_cols, race_cols, totals_col, par_compute = TRUE, n_cores = 2)
#(end_time <- Sys.time() - start_time)
```
This saves us a about a minute for this specific data set (on the computer we ran this on, but times will vary). With larger datasets and more candidate and racial demographic comparisons, the process will take longer and parallelization will become more beneficial.

## Expectations for parallelization

Depending on your dataset, the number of races and candidates you're analyzung, the amount of RAM you have, and the number of physical and logical cores you have the amount of time it takes to run eiCompare functions will differ. Furthermore, its important to uderstand parallelizing requires overhead time and relationships between sample size and run time are not necessarily linear. In the plot below, you'll be able to see the average run time of a dataset for a sample of 100, 200, and 300 precincts. It is apparent here that less samples does not equate a shorter run time. Nonetheless, parallelization can save multitudes of the ~4 minutes saved here, especially if you repeat function calls for analyses such as a boostrap.

```{r echo=FALSE, out.width='100%'}

knitr::include_graphics("para_benchmark_box_nsamples.png")
```

## Summary

`eiCompare` provides capabilities to parallelize operations for iterative ei in `ei_iter()`, diagnostic tests in `ei_rxc()`,  and geocoding in `run_geocoder()`. By setting the parallelization toggle to `TRUE`, as well as having the proper set up with more than 4 cores and more than 16 GB of RAM, the user should be able to run these operations multitudes faster.
