---
title: "`rddapp` `shiny` manual"
author: "Felix Thoemmes"
#date: "January 20, 2018"
output:
  html_document:
    toc: true
    toc_depth: 3
    toc_float: true
    theme: "cosmo"
    css: help_page.css
bibliography: help_page.bib
csl: apa-old-doi-prefix.csl
---

`rddapp` is an [`R` package](https://cran.r-project.org/web/packages/rddapp/index.html) with a `shiny` [web app](https://shiny.rstudio.com/) for the analysis of [regression discontinuity designs](https://en.wikipedia.org/wiki/Regression_discontinuity_design) (RDDs). It offers both parametric and non-parametric estimation of causal effects in RDDs, with one or two assignment variables. It also provides numerous assumption checks, and allows the estimation of statistical power.

**Note:** If any of the equations or mathematical symbols do not align properly on your web page, right click on the disrupted text and select "Math Settings" in the dropdown menu. Then, navigate to the "Math Renderer" option and select "Common HTML".

# Use

## Online
The `rddapp` `shiny` web app can be accessed [online](https://rddapp.shinyapps.io/shinyrdd/). Untested features will first be introduced to the beta [version](https://rddapp.shinyapps.io/shinyrdd_beta/).
The `shiny` environment is fully point-and-click; no programming in `R` is required. Data can be uploaded directly to the `shiny` server for analysis. For sensitive data that must be kept on secure, the use of an offline client is available.

## Offline
To use the `shiny` app offline, `R` must be [installed](https://cran.r-project.org/). Next, open an `R` session and download [the offical rddapp R package from CRAN](https://cran.r-project.org/web/packages/rddapp/index.html). Load the package using the `library(rddapp)` command in the `R` prompt, and then execute the function `shiny_run()`. This command will load the point-and-click `shiny` environment in a local browser. No further interaction with the `R` console is required and no data will be transferred over the Internet.

# Menu
At the very top of the `shiny` app are three menu options, labeled [**Model**](#model_page), [**Power**](#power_page), and **More**. The **Model** page is for statistical estimation, the **Power** page is for power analysis, and the **More** menu contains this manual, as well as a simple **About** page.

# Model page {#model_page}
The **Model** page is subdivided into a data panel and a dynamic results panel which appears after the data are loaded.

## Data upload panel
Data are uploaded using the pull-down menu in the **Data** panel. The choices are to upload either a `.sav` file (native file format of `SPSS`), or a `.csv` file (comma separated values).

The user can further fine-tune the data upload by clicking on the small **gear** button at the top right of the panel to reveal additional options depending on the file format chosen.

The **Browse** button appears automatically when the user chooses a `.sav` or `.csv` file and opens a file explorer from which to select the data for upload.

Once the data are loaded, additional panels will appear for the [**Outcome**](#outcome_panel) variable and [**Treatment design**](#treatment_design_panel).

### `.sav` files {#sav_files}

If the user selects an `.sav` file for upload, the **gear** button in the top right of the **Data** panel can be used to display further options.

If the **Use Value Labels** option is selected, value labels from SPSS will be used for the uploaded data. For instance, a user may have coded a categorical variable `0` and `1`, and included corresponding value labels `male` and `female`. When **Use Value Labels** is activated, the actual labels will be displayed; otherwise, the numeric codes will be used.

If the **Use User's Missing Value** option is selected, any user-specified missing values in `SPSS` will be treated as [`NA`](https://www.statmethods.net/input/missingdata.html) in the `R` program.

Finally, the user can indicate that the uploaded file has been multiply imputed (typically to deal with missing values) by selecting a **Multiple Imputation ID** variable from the data. This variable indicates which imputation event each observation corresponds to. All analyses will be performed on each imputation, estimates from each imputation will be averaged, and standard errors will be derived using Rubin's rules (1987), which accounts for the variation both within and between imputations.

### `.csv` files

If the user selects a `.csv` file, the options under the **gear** button allow the user to specify exactly how the `.csv` file is structured.

The user can specify:

* whether any values are enclosed in **quotes** (double `"`, single `'`, or none),
* what kind of **separator** is used between values (comma `,`, semi-colon `;`, tab, or whitespace),
* whether **missing values** are blank by default or if a special character is used (e.g., `NA`),
* whether the file includes a **header** in the first line with variable names. 

Just as for `.sav` files, the user can specify that the file is multiply imputed, and if so which variable is the **Multiple Imputation ID** (see [above](#sav_files) for details).

### CARE example data

The application includes an example data set, named **CARE**. The data are sampled from the Carolina Abecedarian Project and the Carolina Approach to Responsive Education (CARE) study, with the full data set hosted by [ICPSR](http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/4091). The example data are described briefly in the `rddapp` package [manual](https://cran.r-project.org/web/packages/rddapp/rddapp.pdf), and in detail in the [vignette](https://cran.r-project.org/web/packages/rddapp/vignettes/rddapp.html).

## Outcome panel {#outcome_panel}
The **Outcome** menu allows the user to define the outcome variable: the variable that the researcher believes is affected by the treatment. Additional options for analyzing the outcome variable appear by clicking the **gear** button and include the **Kernel for local linear fitting**, **Type of SE** (standard error), and **Auxiliary Variables**.

The **kernel** option specifies the type of kernel that should be used for the non-parametric (local linear fitting) analysis, described further in the [**Estimates**](#estimates_tab) section. By default, a triangular kernel is chosen, in line with recommendations of Imbens and Kalyanaraman [-@ImbenKalya09; -@ImbenKalya12]. However, the user can change this to a variety of other kernels, defined by the following functions:

  1. Triangular:   $K(u) = (1 - |u|)$,
  2. Rectangular (uniform):  $K(u) = \frac{1}{2}$,
  3. Epanechnikov: $K(u) = \frac{3}{4} (1 - u^2)$,
  4. Quartic (biweight):      $K(u) = \frac{15}{16} (1 - u^2)^2$,
  5. Triweight:    $K(u) = \frac{35}{32}(1 - u^2)^3$,
  6. Tricube:      $K(u) = \frac{70}{81}(1 - |u|^3)^3$,
  7. Gaussian:     $K(u) = \frac{1}{\sqrt{2\pi}} e^{-u^2/2}$,
  8. Cosine:       $K(u) = \frac{\pi}{4}\cos(\frac{\pi}{2}u)$,

where $|u|\leq 1$, except for with the Gaussian kernel, where $u$ can be any real number.

The second option under the outcome tab is the **type of standard error (SE)** that should be used. `rddapp` supports a "regular" standard error, that assumes constant variance ($\sigma^2 (X'X)^{-1}$), as well as all of the types of standard errors that are included in the `sandwich` package ([Zeileis, 2017](https://cran.r-project.org/web/packages/sandwich/sandwich.pdf)). By default, a [heteroskedasticty-robust standard error](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors) (HC1) is chosen. If the user chooses the "One-way Cluster" option, an additional pull-down menu appears, prompting the user to select the **cluster ID** variable.

The **auxiliary variables** option allows users to specify the use of auxiliary variables in the RDD. These variables are integrated in the regressions of the parametric RDD. All auxiliary variables must be numeric, including ordinal and nominal variables (e.g., factors). Non-numeric variables should be manually recoded into dummy-indicator variables prior to performing the analysis.

Purely linear terms are used for all auxiliary variables; there is currently no implementation for adding non-linear terms. To add non-linear terms, a user would have to create them by hand first (e.g., polynomials) and enter them as regular auxiliary variables. By default, the auxiliary variables will not be entered as interactive terms (i.e., interacting with the treatment assignment). In a properly executed RDD, it should not be necessary to include auxiliary variables, but in some instances, including them can increase precision of parameter estimates, and thus statistical power.

For some models, in particular the [frontier approach](#frontier_approach) for multiple assignment variables, an additional option will be displayed that enables the user to define the **number of bootstrap samples** that should be performed for estimation of standard errors and confidence intervals. 

## Treatment design panel {#treatment_design_panel}
The **treatment mechanism** of the RDD is a (series of) rules used to determine whether an individual is *selected* for treatment, but it does not necessarily imply that the individual *actually* received treatment. In a **sharp RDD**, all individuals that were assigned treatment actual received treatment. In a **fuzzy RDD**, not all individuals that were assigned the treatment received it and/or some individuals who were *not* assigned the treatment received it.

The mechanism is described by (a series) of "if" statements, in the **Treatment design** panel, under the heading **Assign treatment if**. The user selects the **_assignment_ variable** in the drop-down menu labeled **primary assignment**. The box underneath is used to input the threshold for being assigned to the treatment. 
By default, every value of the assignment variable that is **greater than or equal to** the threshold leads to an assignment to the treatment condition. Clicking on the $\geq$ sign switches the condition to **less than or equal to** ($\leq$), meaning that values of the assignment variable less than or equal to the threshold lead to assignment. To exclude treatment at the threshold level, click the **gear** button and deselect the **Treatment at the cutoff(s)** option. This will convert the inequality to a **strict** greater than symbol ($>$) or less than symbol ($<$).

For example, choosing an assignment variable and then selecting greater than or equal to ($\geq$) with treatment threshold value 40, would mean that treatment was assigned to those individuals whose assignment variable was equal to or greater than 40. On the other hand, choosing strictly less than ($<$) a value of 10 would mean that individuals whose assignment variable was *strictly* less than 10 were assigned to the treatment.

Clicking the **plus** button in the top right corner of the **Treatment design** panel allows the user to select a **secondary assignment** variable, to accommodate **multiple-assignment RDDs**. Currently, the software only supports up to two assignment variables. One restriction that is in place for these designs is that *both* assignment rules must be either strict inequalities or not (i.e., either both $>$ and/or $<$ or both $\geq$ and/or $\leq$). Combinations of inequalities where one is strict but the other is not can lead to designs in which it is at least theoretically possible to have undefined cases for some analytic approaches.

Under the **Treatment Receipt** heading, the user selects the **_treatment_ variable** in pull-down menu labeled **actual treatment**. The treatment variable indicates whether the individual *actually* received treatment.

# Data analysis tab
The **Data** tab displays a preview of the data upon upload, mainly for the purposes of validating that the data were read into the program correctly, in the **Raw data** panel. Variable names are displayed as table column headers. Users can sort observations by each of the variables by either clicking the **up arrow** next to the variable name for **ascending order** or the **down arrow** for **descending order**. 

Upon specifying the [**Outcome** ](#outcome_panel) variable and the [**Treatment design**](#treatment_design_panel), a  **Model Data** button appears at the top right corner of the **Raw data** panel. Clicking this button restricts the preview table to just the outcome and treatment design variables.

Specifying the [**Outcome** ](#outcome_panel) variable and the [**Treatment design**](#treatment_design_panel) also generates two new result tables.

## Table 1.1

**Table 1.1** shows descriptive statistics of the treatment variable (T), the assignment variable (A1 - and A2, if a second assignment was chosen), and the outcome variable (O). For each variable, three columns are reported: the sample size (**N**), the mean (**M**), and the standard deviation (**SD**). In addition, a correlation matrix is shown on the side, whose columns are treatment (**T**) and outcome (**O**), and whose values are the Pearson correlation coefficient between variables. Clicking on the **notepad** icon in the top right corner of the Table 1.1 panel downloads the data as a `.csv` file.

## Table 1.2

**Table 1.2** summarizes the design of the RDD, and can serve to validate that the design was correctly specified in the [**Treatment design**](#treatment_design) panel. Entries of this table are sample sizes, **n**. For an RDD with a single **assignment variable**, rows are the two different groups given by the treatment design (i.e., having the assignment variable below or above the specified threshold). The columns specify whether the treatment was *actually* received within each group, based on the specified **treatment receipt** variable.

In a **sharp RDD** (a design in which assigned and received treatment always coincide), the off-diagonal cell counts are always zero. In a **fuzzy RDD**, the off-diagonal elements will contain sample sizes of incorrectly classified individuals (i.e., treatment assignment and received treatment do not coincide). In this case, the probability of receiving treatment, $\pi$, conditional on being on either side of the threshold is also displayed. This quantity is computed as a simple conditional probability, $P(T=1|A1)$; no parametric model is being used for this estimate.

If a **second assignment variable** is specified, an additional column is added for this variable. All combinations of treatment thresholds are displayed as single rows, with their respective sample sizes, **n**, and the estimated $\pi$ parameter.

`rddapp` reports further details of the treatment design under the design summary table. It determines whether the design is sharp or fuzzy, and whether one or two assignment variables were used. If the second assignment variable does not alter any of the treatments assigned based on the first assignment variable, the software notes that the second assignment variable is redundant, and calls this an **ineffective assignment**.

# Assumptions tab
RDDs rely on several assumptions, which are summarized in the **Assumptions** tab. Depending on whether the design is sharp or fuzzy, and relies on one or two assignment variables, the content of the assumption tab will vary slightly.

## Figure 2.1

**Figure 2.1** shows the result of **McCrary's** (2008) **sorting test**, which relies on the following idea: in a proper RDD, there should be **no discontinuity at the threshold** for treatment if participants were neutrally selected based on the assignment rule. The presence of a discontinuity could imply that there was self-selection for or against the treatment by some participants or others involved.

As an example, suppose that students are put on the dean's list if their GPA is 3.75 or higher. Teachers who (either implicitly or explicitly) want to help their students may round up a grade that is very close to the treatment threshold. A teacher may grade the final exam of a student with a GPA of 3.74 more leniently, so that the student can obtain a final GPA of 3.75 and make it on the dean's list. If such behavior was wide-spread, we would find fewer scores of 3.74 and just below, and more scores of 3.75 and just above. This effect would manifest itself as a **small discontinuity** in the distribution of GPA, right at the treatment threshold.

### McCrary's sorting test

The sorting test is implemented as described by McCrary (2008). The user specifies the assignment variable and threshold of interest in the pull-down menu labeled **Assignment [Cutoff]**. The data are aggregated with a default bin size and a non-parametric kernel smoother is applied to both sides of the treatment threshold using the midpoints of bins and the observed count in each bin as data. The test measures the vertical distance of the two edges on both sides of the treatment threshold with a log metric. 

Because the sorting test relies on a non-parametric estimation of the kernel density estimate, it is possible to change parameters of the test. Any changes to the parameters automatically update both the plot and the [test summary](#test_summary).

### Bin size

The **bin size** determines the width of bins for the kernel density estimate in the sorting test, which is performed on the means of each bin of raw data. As the bin size **decreases**, the number of bins with frequency at or close to 1 will increase, and so the density estimate will look **very flat** (near 0). As the bin size **increases**, the number of bins decreases, and the kernel density estimate will tend to be **under-smoothed** if it relies on too few bins.

### Band width

The **bandwidth**, also known as the smoothing parameter, determines how many data points are considered at once when computing the smooth approximation to the data. As the bandwidth **increases**, more data points are considered and so the smoother goes **from flat to jagged**. The choice of bandwidth will affect the test statistic, and possibly different inferential decisions. Generally speaking, a bandwidth parameter should be chosen so that **no over- or under-smoothing occurs**. To aid in the selection of the bandwidth parameter, Figure 2.1 will update every time the smoothing parameter is changed.

When changing the parameters, be mindful that it it is considered **bad statistical practice** to change the bin size or bandwidth to maximize the p-value in order to make the test look good. Instead either the default bin size should be used, or adjusted in accordance with a properly smoothed plot. The software defaults are often good choices, but sometimes minor adjustments are needed. <!-- IP: Citation for the default values and why they are good choices?-->

## Test summary {#test_summary}

The software derives a test statistic, **theta**, which quantifies the vertical distance, and a standard error (**se**). These quantities are tested using a $z$-test. The **z** value and the **p** value are also reported. Ideally, this test should *not* be significant. A significant test statistic would indicate that the assumption of a neutrally implemented RDD (assignment only due to treatment threshold) has been violated. The **test summary** is displayed in the lower right corner of the Figure 2.1 panel.

Both the plot and the test summary can be downloaded using the buttons in the top right corner of the Figure 2.1 panel. The plot can be downloaded as a `.png`, `.svg`, or `.pdf`. The test data displayed in the plot can be downloaded as a `.csv` file. 

## Table 2.1

**Table 2.1** displays the results of an attrition analysis. In each row, it provides a count, **N**, of missing data and corresponding percentages of the total sample size for each variable (treatment, outcome, and assignment). The three columns give these counts and percentages:

1. for **all** individuals (**Overall**);
2. restricted to individuals in the control group (**Control**);
3. restricted to individuals who received treatment (**Treatment**).
<!-- IP: The above is writen with the assumption that "control" and "treatment" reflects whether the subject *actually* received treatment (i.e. based on the *treatment* variable and not the *assignment* variable. Is that right? -->

This information can help to determine whether any differential attrition occurred. The table can be downloaded as a `.csv` file using the button in the top right corner of the Table 2.1 panel.

# Estimates tab {#estimates_tab}

The **Estimates** tab is where the user will find point estimates and inferential statistics for the treatment effect. We begin by discussing the **basic estimates interface** in detail in the section on [RDD with one assignment rule](#rdd_one_asst). We then discuss specifics for the **multivariate case** in the [RDD with two assignment rule](#rdd_two_asst) section. We recommend that users read through the single variable section *before* moving on to the section on multivariate analysis. 

## RDD with one assignment rule {#rdd_one_asst}

### Table 3.1

**Table 3.1** contains **all results**, **treatment effect estimates**, and **inferential statistics** for both the parametric and non-parametric models. The user can expand or reduce the amount of information that is being displayed using several toggle switches at the top right of the table, including those for **Parametric** (global) and **Non-parametric** (local) models.

#### Parametric models {#parametric_models}

The **parametric** model results are subdivided into three rows, where each row gives results of the outcome variable modeled as a function of the assignment variable, but with different functional forms in each case. The available functional forms are:

* **linear**, which only has a linear term;
* **quadratic**, which in addition to the linear term from above also has a quadratic term;
* **cubic**, which in addition to the linear and quadratic terms from above also has a cubic term.

Each term listed above has an associated regression coefficient. Moreover, these coefficients are allowed to differ before and after the assignment threshold. As a result, the model always includes an **interaction effect** between the assignment variable and the treatment status. <!-- IP: I don't quite understand what this last sentence means, so it might be worth trying to clarify, perhaps with an example? -->

<!--
IP: I'm trying to understand what the value in the "Est." column is for the parametric models. It might be worth being a bit more specific about how this quantity is calculated.

Also, in trying to understand how the estimates are computed, I've come up with a few questions. If each term type (linear, quadratic, cubic) has an associated regression coefficient as described above:
* is the single estimate reported in the table for each row just the coefficient of the highest order term?
* do the coefficients on the lower order terms change when we add higher order terms or can we also read these off the table from the previous rows?
* if these coefficients are allowed to vary before and after the assignment threshold, which single number is reported as the estimate for "the" coefficient? Is it the difference between the coefficients before and after the treatment?
-->

By default, all parametric models used in the estimation of the treatment effect are performed using **two-stage least-squares** (2SLS) estimation, with **robust standard errors**. The first stage involves a model that uses the **assignment variable** to predict the **treatment variable**. Note that in sharp RDDs, this prediction of treatment assignment will *always* be perfect by definition, which renders this first step unnecessary; the whole process reduces down to a single-stage least-squares regression model. In the second stage, the predicted values from the first stage are used as predictors of the **outcome variable**. The treatment effect is captured in the regression coefficient that is associated with the **treatment  variable** in the *second stage* of estimation.

For each parametric model, several **inferential statistics** are reported in labeled columns:

* **N**: the total <u>sample size</u>;
* **Est.**: the <u>treatment effect estimate</u>;
* **SE**: the <u>standard error</u> associated with the treatment effect estimate;
* **z**: the <u>z-score</u> associated with the treatment effect estimate;
* **p**: the <u>p-value</u> for the estimate;
* **95% CI**: the <u>95% confidence interval</u> for the treatment effect estimate;
* **Cohen's d**: the <u>Cohen's d effect size</u>, which is computed as the mean difference at the treatment threshold divided by the standard deviation of all observed outcome values (same across all models to make direct comparisons of Cohen's d meaningful). 

#### Non-parametric models

The **non-parametric** models use a **weighted least-squares** linear regression. A bandwidth function determines how to weigh each data point in the regression, which determines both the **smoothness** of the regression, as well as the **window** around the treatment threshold from which to use data (points are excluded by being assigning a weight of 0).

The non-parametric model results are subdivided into three rows, where each row gives results of the outcome variable modeled as a function of the assignment variable, but with a different bandwidth function. The bandwidth functions include:

* **optimal**: the "optimal" bandwidth, as defined by Imbens and Kalyanaraman [-@ImbenKalya09; -@ImbenKalya12];
* **half-optimal**: half the optimal bandwidth;
* **double-optimal**: double the optimal bandwidth.

Note that, unlike parametric models, these non-parametric models *differ in the sample size* being used in each regression (since the bandwidth function can assign 0 weight to some points).

The three different non-parametric models are run by default to allow the researcher to see whether results hold under a variety of bandwidths. Additional robustness checks are performed in the [sensitivities](#sensitivities_tab) tab.

In the case of a fuzzy RDD, the user may also request the **intent-to-treat** estimate, using the **ITT** toggle switch in the top right of the Table 3.1 panel. This estimate  ignores the fuzziness of the design, and simply assigns each individual the treatment that they were *supposed to receive* based on the treatment assignment rule. Note that ITT model is only useful for fuzzy designs since in sharp designs, treatment assignment and actual treatment always coincide.

If any covariates are specified in the **auxilliary variables** field found in the additional options of the [outcome](#outcome_panel) panel, an additional toggle button labeled **COV** is shown at the top right of the table. It is deselected by default, but if the user selects it, additional rows are added to Table 3.1 specifying the **regression coefficient** that describes the (linear) relationship between the covariate and the outcome is displayed, along with all **inferential statistics**. Note that even though in the table the regression coefficient is displayed under the same column heading (**Est.**), it does *not* mean that this is a treatment effect estimate. It is simply the linear relationship between covariate and the outcome variable. The regression coefficient for the covariate is obtained from the same model that is used to estimate the RDD treatment effect. That means, the treatment and assignment variables are *also* predictors in this model. It is currently not possible to model the relationship between the covariate and the outcome variable in a non-linear fashion (e.g., polynomial regression, interacted with the treatment, etc.).

The table can be downloaded as a `.csv` file using the button in the top right corner of the Table 3.1 panel.

### Figure 3.1

**Figure 3.1** provides a visual display of the treatment effect summarized in Table 3.1, typically through some combination of a **scatterplot** of the assignment vs. outcome variables along with a plot of one or more fitted models and their associated confidence intervals over top. 

The user can customize this plot in three different ways: by changing the **model type**, **predicted lines**, and **data points**.

#### Model type

<!-- IP: Perhaps briefly describe the difference between compiler average and intention-to-treat here. Although ITT is briefly discussed in the section on non-parametric models,  "compiler average" is never explained. -->

The **model type** option is only useful for fuzzy designs as there is no difference between the **compiler average** and the **intention-to-treat** options for sharp design; in these cases, treatment assignment and actual treatment always coincide.

#### Predicted lines {#predicted_lines}

The user can toggle which of the models summarized in Table 3.1 are plotted as **predicted lines** in Figure 3.1 by selecting their labels from the drop-down menu or by deleting their labels from this field by clicking the label and hitting the backspace key on the keyboard. By default, the non-parametric optimal model is overlaid.

Furthermore, the user can specify a percentage for the confidence interval(s) plotted in the **CI** input box, and toggle whether or not the area between the confidence interval boundaries should be colored in (by selecting the **fill area** button).

#### Data points {#data_points}

The user can choose whether to scatterplot all **data points** (with the **raw** option), or to bin them **evenly** or by **quartile**. The latter option may be preferred if there are many data points to prevent over-plotting. In each bin, an average of the outcome variable is computed and plotted as a single point.

Binning **evenly** means that the assignment variable is sub-divided into bins of equal width, while binning by **quantiles** uses quantiles of the assignment variable. In either case, the binning algorithm will *always* avoid spanning a bin over the treatment threshold value, which can sometimes lead to slight distortions in the bin width and sample size. For instance, there are situations in which even binning does not result in bins of equal width, or quantile binning does not result in the same sample size across all bins, as one might expect.

The user can specify the number of bins in the input box labeled **bins**. The point size can be scaled with respect to the number of points in each bin using the **scale point size** switch.

A (normal) confidence interval is plotted around the estimate of the mean, at whatever percentage specified in the **CI** field. The user can toggle whether or not to **hide error bars** with the eponymous button.

Plot labels (e.g., regression line labels, point legends, axis labels) are generated automatically and cannot be customized by the user at this time.

The plot can be downloaded as a `.png`, `.svg`, and a `.pdf` using the buttons in the top right of the Figure 3.1 panel.

## RDD with two assignment rules {#rdd_two_asst} 

An **RDD with two assignment variables** requires a more complicated analysis because several estimates <!-- IP: For what? Treatment effects? --> have been suggested in the literature. For an overview, see the paper by Wong, Steiner, and Cook (2013), which describes a variety of designs and estimates. The three estimate approaches implemented in this software are directly derived from the Wong et al. (2013) paper, as well as the Stata code that was published by Wong.

The top right of the **Table 3.1** panel features toggle buttons for the three multivariate **approaches**: [Centering](#centering_approach), [Univariate](#univariate_approach), and [Frontier](#frontier_approach). For each of these approaches, the user can also the toggle parametric and non-parametric models described in [the previous section](#rdd_one_asst), and the same results are displayed as previously discussed.

#### Centering approach {#centering_approach}
In the **centering approach**, the multiple assignment RDD is transformed into a single assignment RDD through the use of a **centering function**. In particular, the program centers all assignment variables around the treatment threshold value by subtracting the value of the treatment threshold from each assignment variable; these differences are called **deviation scores**. For each observation, we take **minimum** of the two deviation scores calculated from the values of each of the two assignment variables. In designs in which treatment is assigned to units above or below the treatment threshold (or any combination of those), sometimes the maximum is taken. <!-- IP: Maybe provide one example of why the maximum might taken? I *think* the min makes sense intuitively because you want to get as close to the treatment threshold to make sure you're comparing similar subjects (apples to apples) to reduce confounding factors. If that's the case, I'm not sure why you would ever want to take the max! -->

Following this transformation, each observation has a single (centered) assignment variable (the **minimum deviation score**), which is then used to compute a [single assignment RDD](#rdd_one_asst). Note that the treatment effect estimate is **sensitive to the metric** used for each assignment variable. Wong et al. (2013) show that even standardization cannot typically mitigate this issue, and advise that the centering approach should *only* be used when the assignment variables share the **same metric**. 

The centering approach computes what Wong et al. call the **frontier average treatment effect**, which is essentially a treatment effect along the whole treatment threshold of the two-dimensional response (outcome) surface defined by the two assignment variables. <!-- IP: Based on the above section, I thought the centering approach is just a single assignment RDD after the centering transformation is applied, so wouldn't the estimated treatment effect be computed in the same way as in the one variable case? Where does the frontier average treatment effect come in, which sounds like a multivariate computation? -->

The plot for the centering approach can be displayed in **Figure 3.1** by selecting **Centering** in the **Model Type** drop-down menu of that panel. The options for [**Predicted lines**](#predicted_lines) and [**Data points**](#data_points) remain the same as in the case of an RDD with a single assignment variable.

#### Univariate approach {#univariate_approach}
The **univariate approach** focuses on treatment effects that are **exclusively attributable** to one of the two assignment variables. Wong et al. (2013) call these the **frontier-specific effects**. In a design with two assignment variables, two frontier-specific effects will be computed: one for each assignment variable.

To estimate these effects, the observations are divided into **subsets** based on which rule assigned them to treatment. For example, to compute the frontier-specific effect of the *first* assignment variable, the only observations used are those assigned to treatment based *solely* on the *first* assignment variable. **Table 3.1** shows the sample sizes for each subset in the **N** column.

These subsets of the data for each assignment variable, labeled **A1** and **A2**, are then analyzed using an [RDD with one assignment variable](#rdd_one_asst).

There is an **alternative** to the univariate approach, which retains *all* subjects and then treats the resulting design as a *fuzzy* RDD with a single assignment variable. Wong et al. (2013) demonstrated that this approach generally has **no advantages** with respect to increasing statistical **power** or decreasing **bias** when compared to simply excluding the mis-classified observations from the analysis. 

The plot for the univariate approach can be displayed in **Figure 3.1** by selecting **Univariate (A1)** or **Univariate (A2)** in the **Model Type** drop-down menu of that panel. The options for [**Predicted lines**](#predicted_lines) and [**Data points**](#data_points) remain the same as in the case of an RDD with a single assignment variable.

#### Frontier approach {#frontier_approach}
The **frontier approach** estimates the treatment effect along the whole discontinuity frontier of the response (outcome) surface. As described by Wong et al. (2013), this approach is performed by first estimating the response surface (e.g., using a parametric model), and then using **kernel density estimates** for numerical integration of the treatment effect.

The frontier approach yields three different estimates of the average treatment effect:

* the **average frontier effect**, computed for the whole frontier
* the **frontier 1** effect, computed for the frontier along the first assignment variable, **A1**;
* the **frontier 2** effect, computer for the frontier along the second assignment variable, **A2**.

All of these effects are estimated **simultaneously** using the full sample.

This method usually results in **higher statistical power** than other approaches, but requires the assumption that the response surface is correctly modeled by the linear *parametric* model used in the computation. Non-parametric models are not supported at this time and so its toggle button is inactive in this case. 

There are three linear parametric models available: **Unconstrained**, **Heterogeneous effects**, and **Constant Effect**. Note that non-linear parametric models are not supported at this time.

The **constant effect** model is the most restrictive, as it imposes two big constraints:

1. the heights of the discontinuities at the treatment thresholds must be equal across both dimensions given by the assignment variables;
2. the regression slope on all parts of the response surface (away from the discontinuity) must be equal.

Despite these constraints, the average frontier treatment effect and the frontier-specific effects are **not necessarily identical** due to the fact that the raw data may not be identically distributed along the regression surface, and so the kernel density estimate for the numerical integration is not weighted equally on all parts of the frontier. <!-- IP: I added this last bit with my limited understanding of this procedure, so please double check it! -->

The **heterogeneous effects** model relaxes constraint 1 of the **constant effect** model. In particular, the heights of the discontinuities are allowed to differ in different segments of the frontier. However, the slope of the regression surface must still be equal everywhere (away from the discontinuities).

The fully **unconstrained** model eschews both constraints 1 and 2, which means that it is the most complex of these linear parametric models. Wong et al. (2013) does not provide closed form (analytic) solutions for standard errors for the frontier treatment effects. We use **bootstrapping** to compute the inferential statistics in **Table 3.1**.

In order to perform these computations, the number of bootstrap samples must be specified in the additional options of the [**Outcome**](#outcome_panel) panel (in the [**Data**](#data_tab) tab), which is displayed using the **gear button** in the top right corner of the panel. The input field for the number of bootstrap samples is labeled **SE and CI for frontier estimate**. Once the number of samples is specified and the user navigates back to the [**Estimates**](#estimates_tab) tab, the computation begins and **Table 3.1** is populated when the calculations are complete. Using a large number of bootstrap samples is computationally expensive, and it make take a long time to complete the calculations.

**Figure 3.1** displays a three-dimensional plot of the response surface when the **Frontier** option is selected in the **Model type** pull-down menu. New pull-down menus appear below, replacing the **Predicted lines** and **Data points** options.

The first new menu is labeled **Model Specification**, where the user can select any of the three linear parametric models implemented for the frontier approach (**Unconstrained**, **Heterogenous effect**, **Constant effect**). These options help the user visualize each model's constraints (or lack thereof).

The remaining new menus enable the user to modify the appearance of the plot. The **View** pull-down menu has four options: **Perspective**,which displays a three-dimensional plot of the response surface, as well as **Top**, **Front**, and **Side**, which show various two-dimensional projections of the surface (flattening one dimension). For example, the **Top** view shows the response surface directly from above,  flattening the outcome variable dimension, which can help the user visualize the treatment thresholds.

The three-dimensional perspective plot can be **rotated** using the **sliders** labeled with arrows. The graph is regenerated every time these sliders are adjusted, and it takes for the plot to refresh.

The **panel** buttons allow the user to **color** all surfaces that received the treatment assignment, add a **grid** to the response surface, and **shade** the surface slightly.

Finally, **data** buttons allow the user to scale the axes to the **original scale** (by default, the axes are scaled to the same length), and to add the **raw data** points to the plot. 

### Fuzzy RDD
In the case of an **RDD with multiple assignments** that is **fuzzy**, the design becomes slightly more complicated. However, the overall interface and outputs are identical to the sharp counterpart of this design. All of the models that were described [above](#rdd_two_asst), are now being estimated using the **two-stage least squares** approach, in which the fuzzy treatment assignment is estimated in a first step, and then the *estimated* treatment assignments are used for all of the models. 

# Sensitivities tab {#sensitivities_tab}
In this tab, the user can perform various **sensitivity analyses**, including those for the **treatment threshold** and the **bandwidth**. Note that neither of Figures 4.1 or 4.2 will be generated automatically; they will only appear after the corresponding blue circular arrow button is pressed (found in the lower right corner of each figure panel).

## Figure 4.1

Figure 4.1. performs a sensitivity analysis for the **treatment threshold** by plotting the treatment effect estimate as a function of a range of hypothetical treatment thresholds for various models.

These tests of treatment effects at wrong treatment threshold values are sometimes also called **placebo tests**. The assumption is that in a properly-run RDD, the treatment effect should *only* be visible at the actual treatment threshold. Hence, seeing many non-significant results at treatment thresholds away from the actual treatment threshold value is desirable. 

The user can change the span of treatment thresholds to test by defining the endpoints using the **min** and **max** boxes below the plot, and the number of estimates to compute between these endpoints using the **steps** box. The treatment effect at the true treatment threshold is always estimated and denoted with a dashed vertical line labeled **designed cutoff**. By default, a 95% confidence interval is shown around the estimates, but the user can change this percentage in the **CI** box at the top right of the Figure 4.1 panel. 

The user can **select the models** with which to perform this analysis (e.g., the parametric linear model, the non-parametric optimal model etc.) in the box directly under the plot. Clicking this box will open a drop-down menu featuring all models that can be added to the plot. To delete a model from the plot, click on its label and press the keyboard's backspace button. The labels in this menu correspond to the ones used in the [Estimates](#estimates_tab) tab.

Both the plot and the data being plotted can be downloaded using the buttons in the top right corner of the Figure 4.1 panel. The plot can be downloaded as a `.png`, `.svg`, or `.pdf`. The test data displayed in the plot can be downloaded as a `.csv` file. 

## Figure 4.2

Figure 4.2. is a robustness check for the non-parametric model; it shows how the treatment effect estimate changes with different **bandwidths**. Ideally, the graph should show a somewhat flat line that indicates that over a wide range of bandwidth choices, the estimate does not change dramatically.

The user can change the span of bandwidths to test by defining the endpoints using the **min** and **max** boxes below the plot, and the number of estimates to compute between these endpoints using the **steps** box. The treatment effect at the optimal bandwidth is always estimated and denoted with a dashed vertical line labeled **optimal bandwidth**. By default, a 95% confidence interval is shown around the estimates, but the user can change this percentage in the **CI** box at the top right of the Figure 4.2 panel. 

Both the plot and the data being plotted can be downloaded using the buttons in the top right corner of the Figure 4.2 panel. The plot can be downloaded as a `.png`, `.svg`, or `.pdf`. The test data displayed in the plot can be downloaded as a `.csv` file. 

# `R` code tab
In this tab, the current analysis that the user selected via the drop-down menu is translated into `R` code. <!-- IP: I don't see where the drop-down menu is; is this an outdated sentence (after updates to the GUI)? --> Users can copy and paste this code into an `R` script for customization, automation, and for conducting reproducible research.

# Power page {#power_page}
Users may wish to perform a formal (frequentist) **power analysis**, to determine a priori sample sizes, or to construct power curves for various sample sizes. All power analyses are performed using [**Monte Carlo simulations**](https://en.wikipedia.org/wiki/Monte_Carlo_method). The user defines population parameters, which are used to simulate a model. The treatment effect is estimated from the simulation and its significance is calculated. This procedure is repeated many times, and a **mean** and **variance** are calculated for the treatment effects. The **statistical power** is estimated as the proportion of significant results across all Monte Carlo simulations.

The success of this process hinges on properly defining the **population parameters**, which is often challenging. We attempt to guide the user in this process using various graphical displays. These displays change depending on the particular design that is chosen (i.e., whether the RDD has a single or multiple assignment variable(s)).

## Assignment panel

The **Assignment** panel is where the user defines the assumed distribution of the assignment variable. The plot shows a histogram with kernel density estimate. The shading of the histogram bars indicates group membership (**treatment** or **control**).

The settings under the **Distribution** label allow the user to specify the distribution type and parameters for the assignment variable, labeled **A1** by default. The distribution types available are **normal** and **uniform**, and the user can toggle between them using the eponymous toggle button. For a normal distribution, a mean $\mu$ and a standard deviation $\sigma$ must be provided. For a uniform distribution, a **min**imum value and a **max**imum value must be provided. 

Under the **Treatment design** label, the user can set the assignment rule, i.e., the values of the assignment variable **A1** at which treatment is administered. The assignment rule changes the proportion of treated and untreated, which affects power estimates. To change the assignment rule, the user can click on the **inequality** symbol to toggle between greater than or equal to and less than or equal to and change the value of the assignment threshold in the adjacent text box. Note that there is no setting to exclude treatment at the threshold (i.e., to remove the equality from the inequality). If the assignment variable is discrete and the desired treatment assignment is **A1** $> x$, then the user could use the equivalent assignment rule **A1** $\geq x+1$. 

The user may also change the amount of **fuzziness** in the design, which is defined as the proportion of misclassified cases around the treatment threshold. The user can set this proportion (as a decimal) below and above the threshold independently, using the text boxes adjacent to the **fuzziness** label. The left box is for the proportion of control cases that were mis-classifieds and instead treated, while the right box is for cases assigned treatment but actually left untreated. Note that the left box is *always* associated with control cases and the right box is *always* associated with treatment cases even if the assignment rule flips the plot so that cases *under* (i.e., to the left of) the threshold are treated.

The **sample size** can be specified in the box found in the top right of the **Assignment** panel.

Clicking the **plus** button in the top right corner of the **Assignment** panel allows the user to add a second assignment variable (labeled **A2**). The **distribution** and **treatment design** for **A2** can be set in the same way as they were for **A1**. All parameters (including distribution, and fuzzyness) of the second variable can be modified in the same way. The plot will update to show an annotated scatterplot with each assignment variable on one axis, along with a sketch of their distributions.

## Parametric model for outcome panel

Once the assignment is specified, the user must specify the **parametric model** for the **outcome** variable as a function of the assignment and treatment.

Currently, only **linear** models are implemented, though there is a fully **unconstrained parametric model** with respect to **interactive effects**. In this case, every segment of the regression surface can have a different slope.

The plot in this panel shows the **simulated outcome** variable as a function of the assignment and will update when simulation parameters are changed. 

In the case of an **RDD with one assignment variable**, a scatterplot with a regression line is shown. The user can modify the treatment effect (labeled **Treatment**), the intercept of the regression slope (labeled **Constant**), the slope on the control side of the treatment threshold (labeled **Slope**), and the difference in slopes on the treatment side of the threshold (labeled **Interaction**). Lastly, the user can define the **Partial $\eta^2$**, which measures the effect size of the treatment. As the partial $\eta^2$ increases, the simulated data will lie closer to the regression lines. Defining the effect size is an indirect way of changing the standard error of the treatment effect. 

In the case of an **RDD with two assignment variables**, the parametric model is substantially more complex. It now involves treatment effects at both **frontiers**, and slopes that are defined for every piece of the regression surface divided by the threshold for each assignment variable. <!-- IP: I'm not 100% sure what this last bit means, but I also haven't edited the Estimates section for two variable RDDs yet...-->

There are a total of 13 regression coefficients that need to be defined by the user:

* the **intercept**, i.e., the predicted outcome when both assignment variables at their respective treatment thresholds (labeled **constant**);
* the two **treatment effects** (labeled **T1** and **T2** for the assignment variables **A1** and **A2** respectively), which set the effect size of the treatment at the threshold of each assignment variable;
* the two **regression coefficients** that relate the assignment variables to the outcome, while holding treatments constant at the control group (labeled **A1** and **A2**); <!-- IP: Not sure what the second part about the control group means... -->
* a total of seven **interaction effects** are included, that define how the slopes of the assignment variables change with different values of the treatment and control values for the two cut-offs. <!-- IP: An example would be helpful here, and I'm not really sure what to suggest because I don't fully understand these fields. --> 

## Table 5.1

Once the parametric models are specified in the **Assignment** and **Parametric Model for Outcome** panels, the user can generate Monte Carlo simulations in the **Table 5.1 Powers of RDD estimates** panel.

The user can specify the number of Monte Carlo simulations to be performed in the field labeled **Iterations** in the top right corner of the panel. The default number of iterations is 500. In general, as the number of iterations increases, the power estimate becomes more precise; however, a large number of iterations may take a very long time to complete.

The simulations are executed by pressing the **blue circular arrow button** in the top right corner of the panel. Once the simulations are complete, Table 5.1. is populated with results.

For an **RDD with one assignment variable**, the rows of the table give the results for the **linear parametric** model and the **non-parametric** model with **optimal bandwidth**.

For an **RDD with two assignment variables**, the rows of the table give the results for three linear parametric models (**centering**, and two **univariate** models), as well as three non-parametric models (same as the parametric ones but with an **optimal bandwidth**). Other possible effects (e.g., the frontier average effect) are not estimated as they take prohibitively long to compute.

In both cases, the columns of the table are:

* **S**: the number of successful Monte Carlo simulations; typically, this value will be equal to the number of iterations, but for some usual models (e.g., lots of fuzziness in the assignment), it is possible that some simulations do not yield a valid estimate;
* **Treatment effects**: the sample **Mean** and **Variance** of the treatment effect from the simulations of each model; for a large enough number of iterations, the user-defined treatment effect should be recovered;
* **Power**: the estimated statistical power (Type II error rates) for various significance levels ($\alpha = .05, .01, .001$). 

The tab above the Table 5.1 panel, labeled **R Code** gives the code used to perform the simulation and populate the table. 

## Figure 5.1

Figure 5.1. allows the user to investigate how statistical power is affected by sample size. Inside the panel, the user can set the **minimum sample size** and the **maximum sample size**, while the number of sample sizes to take in this interval can be set in the **Steps** box, in the top right corner of the panel.

The number of **Iterations in each step** can be set in the top right corner of the panel as well. The default number of iterations is 500. In general, as the number of iterations increases, the power estimate becomes more precise; however, a large number of iterations may take a very long time to complete.

The simulations are executed by pressing the **blue circular arrow button** in the top right corner of the panel. Figure 5.1. is populated with results as the simulations for each sample size are complete.

Once the simulations are fully complete, the user can adjust what is plotted in the figure. The **Group lines by** drop-down menu reveals two main options:

1. **Model type**: the results for each model are plotted as a different curve;
2. **Alpha level**: the results the three levels of significance ("alpha levels") are plotted as different curves.

Depending on the **Group lines by** option selected, a second drop-down menu appears below it. If **Model type** is selected, the menu **Show alpha level** appears, where the user chooses the significance level for which to plot the estimated power. If **Alpha leve** is selected, the menu **Show model type** appears, where the user chooses the model for which to plot the estimated power.

Both the currently displayed plot and the full simulation data can be downloaded using the buttons in the top right corner of the Figure 5.1 panel. The plot can be downloaded as a `.png`, `.svg`, or `.pdf`. The simulation data can be downloaded as a `.csv` file.

# About
This software was developed by [Felix Thoemmes](https://psychology.cornell.edu/felix-j-thoemmes), [Wang Liao](https://wliao229.github.io/), [Ze Jin](https://www.linkedin.com/in/ze-jin-08045b31), [Wenyu Zhang](http://zwenyu.github.io/), and [Irena Papst](https://github.com/papsti) all at Cornell University. We are happy to receive feedback! Please send it to
![](../www/fjt_email.png)


The project was supported by the [Institute of Education Sciences](https://ies.ed.gov/), [U.S. Department of Education](https://www.ed.gov/), through Grant R305D150029. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

# Citations
