---
title: "Data cleaning function in conquestr"
author: "Dan Cloney, Dave Jeffries"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Data cleaning function in conquestr}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library("conquestr")
```

# Introduction

This vignette demonstrates how to use _conquestr_ to document data, report on 
the quality of data, clean data, and construct item bundles or derived variables 
based on several variables.

# Document data

conquestr has a built in system file that we will use for this example.

The function `getCqItanal` will return a list of lists, each list relating to 
one generalised item from an 'ACER ConQuest' `itanal` output. 
The list for each item contains the following information: (1) the item name 
according to the item label, (2) a table of item category statistics for the 
item, and (3) the item-total and item-rest correlations for the item.

Note that you must use `matrixout` in your 'ACER ConQuest' call to `itanal` to 
ensure that these objects are available in the system file from your analysis.

```{r}

# get default sys file
myEx1Sys <- ConQuestSys()

# get itanal lists
myEx1Sys_itanal <- getCqItanal(myEx1Sys)

# show unformatted list objects for first item
print(myEx1Sys_itanal[[1]][[1]])

```

Following the item-specific list objects, the last element of the list returned 
by `getCqItanal` contains summary statistics for the full set of items. 
The summary statistics include raw and latent score distribution statistics and 
Cronbach's coefficient $\alpha$.

# Create formatted itanal tables for a report

So far, we have shown how to access the test and item analysis statistics that 
are available through the itanal command in 'ACER ConQuest' and we have shown 
these without any formatting. 
One of the many benefits of integrating 'ACER ConQuest' output into a markdown 
document is to permit automated conditional formatting of item analysis output. 
In this section we show how this conditional formatting can be set up.

## Set up criteria for conditional formatting

Pre-specifying criteria for conditionally formatting item analysis output is a 
key step in an automated workflow. Any number of metrics from the item analysis 
can be specified for conditional formatting. 
Several of these can be passed to _conquestr_ functions as will be illustrated 
in the following sections.

```{r}

# set statistical criteria for conditional formatting

easyFlag <- 85 # highlight if facility is GREATER than this value
hardFlag <- 15 # highlight if facility is LESS than this value
irestFlag <- 0.2 # highlight if item-rest r is LESS than this value
underfitFlag <- 1.2 # highlight if weighted MNSQ is GREATER than this value
overfitFlag <- 0.8 # highlight if weighted MNSQ is LESS than this value
ptBisFlag <- 0.0 # highlight if non-key ptBis r is MORE than this value

```


## An example of an html itanal table for item categories

The function `fmtCqItanal` will return a formatted version of the itanal object 
that we read in earlier.  
Presently this function will apply coloured text to any distractor point biserial 
correlation that is larger than `r ptBisFlag`. 
The following example shows the output for the fourth item in the current item analysis.

```{r}

# return a conditionally formatted item category statistics table for the fourth item
myEx1Sys_itanal_f <- fmtCqItanal(myEx1Sys_itanal, ptBisFlag = ptBisFlag, textColHighlight = "red")

# print table
myEx1Sys_itanal_f[[1]][[4]]

# print summary
myEx1Sys_itanal_f[[1]][[length(myEx1Sys_itanal_f[[1]])]] # the last object is always the summary

```

# Conclusion

This short vignette has illustrated how to access and display itanal output from an 
'ACER ConQuest' analysis using _conquestr_. 
Future vignettes will demonstrate basic and advanced plotting and the production 
of publication quality item analysis technical reports.


