# Cross-validation with single algorithm

When users choose to estimate and evaluate ITR under cross-validation, the package implements Algorithm 1 from Imai and Li (2023) to estimate and evaluate ITR. For more information about Algorithm 1, please refer to the this page.

Instead of specifying the `split_ratio` argument, we choose the number of folds (`n_folds`). We present an example of estimating ITR with 3 folds cross-validation. In practice, we recommend using 10 folds to get a more stable model performance.

Input | R package input | Descriptions
--- | --- | ---
Data \(\mathbf{Z}=\left\{\mathbf{X}_i, T_i, Y_i\right\}_{i=1}^n\) | `treatment = treatment, form = user_formula, data = star_data` | `treatment` is a character string specifying the treatment variable in the `data`; `form` is a formula specifying the outcome and covariates; and a dataframe `data`
Machine learning algorithm \(F\) | `algorithms = c("causal_forest")` | a character vector specifying the ML algorithms to be used
Evaluation metric \(\tau_f\) | PAPE, PAPD, AUPEC, GATE | By default
Number of folds \(K\) | `n_folds = 3` | `n_folds` is a numeric value indicating the number of folds used for cross-validation
… | `budget = 0.2` | `budget` is a numeric value specifying the maximum percentage of population that can be treated under the budget constraint

    library(evalITR)
    # estimate ITR
    set.seed(2021)
    fit_cv <- estimate_itr(
                   treatment = treatment,
                   form = user_formula,
                   data = star_data,
                   algorithms = c("causal_forest"),
                   budget = 0.2,
                   n_folds = 3)
    #> Evaluate ITR with cross-validation ...

The output will be an object that includes estimated evaluation metric \(\hat{\tau}_F\) and the estimated variance of \(\hat{\tau}_F\) for different metrics (PAPE, PAPD, AUPEC).

    # evaluate ITR
    est_cv <- evaluate_itr(fit_cv)
    #> Cannot compute PAPDp

    # summarize estimates
    summary(est_cv)
    #> -- PAPE ------------------------------------------------------------------------
    #>   estimate std.deviation     algorithm statistic p.value
    #> 1     0.49          0.65 causal_forest      0.76    0.45
    #>
    #> -- PAPEp -----------------------------------------------------------------------
    #>   estimate std.deviation     algorithm statistic p.value
    #> 1        3          0.77 causal_forest       3.9 8.8e-05
    #>
    #> -- PAPDp -----------------------------------------------------------------------
    #> Cannot compute PAPDp
    #>
    #> -- AUPEC -----------------------------------------------------------------------
    #>   estimate std.deviation     algorithm statistic p.value
    #> 1      1.3           1.6 causal_forest       0.8    0.42
    #>
    #> -- GATE ------------------------------------------------------------------------
    #>   estimate std.deviation     algorithm group statistic p.value upper lower
    #> 1      -56            59 causal_forest     1     -0.95    0.34    59  -171
    #> 2       32            67 causal_forest     2      0.48    0.63   163   -99
    #> 3       16            59 causal_forest     3      0.27    0.79   131   -99
    #> 4       10            76 causal_forest     4      0.14    0.89   159  -138
    #> 5       16            98 causal_forest     5      0.16    0.87   209  -177