# Introducing OutlierTree

#### David Cortes

#### 2022-07-27

This short vignette illustrates basic usage of the OutlierTree library for outlier detection, using the hypothyroid dataset which is bundled with it.

This is a library which flags suspicious values within an observation, contrasting them against the normal values in a human-readable format and potentially adding conditions within the data that make the observation more suspicious; and does so in a similar way as one would do it manually, by checking extreme values in sorted order and filtering observations according to the values of other variables (e.g. if some other variable is `TRUE` or `FALSE`).

For a full description of the procedure see Explainable outlier detection through decision tree conditioning.

### A look at the dataset

This is a dataset about hospital patients who might potentially have hypo- or hyperthyroidism problems. The observations are about anonymous people whose demographic characteristics, drug intake, and hormone indicators were recorded, along with the judgement about their condition.

It contains many interesting outliers which have something obviously wrong when examined visually, but which would nevertheless be missed by other outlier detection methods.

    library(outliertree)
    data(hypothyroid)
    summary(hypothyroid)
          age           sex       on.thyroxine    query.on.thyroxine
     Min.   :  1.00   F   :1817   Mode :logical   Mode :logical
     1st Qu.: 36.00   M   : 849   FALSE:2442      FALSE:2733
     Median : 54.00   NA's: 106   TRUE :330       TRUE :39
     Mean   : 51.75
     3rd Qu.: 67.00
     Max.   :455.00
     NA's   :1
     on.antithyroid.medication    sick          pregnant       thyroid.surgery
     Mode :logical             Mode :logical   Mode :logical   Mode :logical
     FALSE:2738                FALSE:2663      FALSE:1882      FALSE:2734
     TRUE :34                  TRUE :109       TRUE :41        TRUE :38
                                               NA's :849

     I131.treatment  query.hypothyroid query.hyperthyroid  lithium
     Mode :logical   Mode :logical     Mode :logical      Mode :logical
     FALSE:2724      FALSE:2611        FALSE:2600         FALSE:2758
     TRUE :48        TRUE :161         TRUE :172          TRUE :14

       tumor           goitre        hypopituitary     psych
     Mode :logical   Mode :logical   Mode :logical   Mode :logical
     FALSE:2747      FALSE:2701      FALSE:2771      FALSE:2638
     TRUE :25        TRUE :71        TRUE :1         TRUE :134

          TSH                T3             TT4             T4U
     Min.   :  0.005   Min.   : 0.05   Min.   :  2.0   Min.   :0.3100
     1st Qu.:  0.430   1st Qu.: 1.60   1st Qu.: 88.0   1st Qu.:0.8800
     Median :  1.400   Median : 2.00   Median :104.0   Median :0.9800
     Mean   :  4.509   Mean   : 2.03   Mean   :109.1   Mean   :0.9986
     3rd Qu.:  2.600   3rd Qu.: 2.40   3rd Qu.:125.0   3rd Qu.:1.0800
     Max.   :478.000   Max.   :10.60   Max.   :430.0   Max.   :2.1200
     NA's   :277       NA's   :576     NA's   :179     NA's   :291
     referral.source       diagnosis         FTI
     other:1615      compensated: 154   Min.   :  2.381
     STMW :  91      negative   :2553   1st Qu.: 92.857
     SVHC : 275      primary    :  63   Median :107.175
     SVHD :  31      secondary  :   2   Mean   :110.791
     SVI  : 760                         3rd Qu.:124.272
                                        Max.   :394.495
                                        NA's   :292

#### Finding outliers

    otree <- outlier.tree(hypothyroid, nthreads=1)
    Reporting top 8 outliers [out of 8 found]

    row [531] - suspicious column: [hypopituitary] - suspicious value: [TRUE]
        distribution: 99.964% different [norm. obs: 2772]

    row [623] - suspicious column: [age] - suspicious value: [455.00]
        distribution: 99.964% <= 94.00 - [mean: 51.60] - [sd: 18.98] - [norm. obs: 2770]

    row [2230] - suspicious column: [T3] - suspicious value: [10.60]
        distribution: 99.951% <= 7.10 - [mean: 1.98] - [sd: 0.75] - [norm. obs: 2050]
        given:
            [query.hyperthyroid] = [FALSE]

    row [1138] - suspicious column: [age] - suspicious value: [75.00]
        distribution: 95.122% <= 42.00 - [mean: 31.46] - [sd: 5.28] - [norm. obs: 39]
        given:
            [pregnant] = [TRUE]

    row [2211] - suspicious column: [age] - suspicious value: [73.00]
        distribution: 95.122% <= 42.00 - [mean: 31.46] - [sd: 5.28] - [norm. obs: 39]
        given:
            [pregnant] = [TRUE]

    row [1438] - suspicious column: [FTI] - suspicious value: [394.50]
        distribution: 99.618% <= 232.08 - [mean: 132.68] - [sd: 28.23] - [norm. obs: 261]
        given:
            [TT4] > [123.00] (value: 430.00)
            [referral.source] != [other] (value: STMW)

    row [745] - suspicious column: [TT4] - suspicious value: [239.00]
        distribution: 98.571% <= 177.00 - [mean: 135.23] - [sd: 12.57] - [norm. obs: 69]
        given:
            [FTI] between (97.96, 128.12] (value: 112.74)
            [T4U] > [1.12] (value: 2.12)
            [age] > [55.00] (value: 87.00)

    row [1412] - suspicious column: [TT4] - suspicious value: [430.00]
        distribution: 99.762% <= 230.00 - [mean: 111.88] - [sd: 31.88] - [norm. obs: 420]
        given:
            [T3] is NA

(i.e. it’s saying that it’s abnormal to be pregnant at the age of 75, or to not be classified as hyperthyroidal when having very high thyroid hormone levels)

### A closer look at some of those outliers

A look at the distributions within the clusters in which some outliers were flagged:

    pregnant <- hypothyroid[hypothyroid$pregnant,]
    hist(pregnant$age, breaks=50, col="navy",
         main="Age distribution among pregnant patients",
         xlab="Age")

    non.hyperthyr <- hypothyroid[!hypothyroid$query.hyperthyroid,]
    hist(non.hyperthyr$T3, breaks=50, col="darkred",
         main="T3 hormone levels\n(Non-hyperthyroidal patients)",
         xlab="T3 blood concentration")

### Handling results

The identified outliers, along with all the relevant information, are returned as a list of lists, which can be inspected manually and the exact conditions extracted from them (see documentation for more details).

They are nevertheless returned as a class of its own in order to provide pretty-printing and slicing:

    outliers <- predict(otree, hypothyroid, outliers_print=FALSE)
    outliers[1:700]
    Reporting top 2 outliers [out of 2 found]

    row [531] - suspicious column: [hypopituitary] - suspicious value: [TRUE]
        distribution: 99.964% different [norm. obs: 2772]

    row [623] - suspicious column: [age] - suspicious value: [455.00]
        distribution: 99.964% <= 94.00 - [mean: 51.60] - [sd: 18.98] - [norm. obs: 2770]

    outliers[1138]
    Reporting top 1 outliers [out of 1 found]

    row [1138] - suspicious column: [age] - suspicious value: [75.00]
        distribution: 95.122% <= 42.00 - [mean: 31.46] - [sd: 5.28] - [norm. obs: 39]
        given:
            [pregnant] = [TRUE]

    outliers[[1138]]
    $suspicous_value
    $suspicous_value$column
    [1] "age"

    $suspicous_value$value
    [1] 75

    $suspicous_value$decimals
    [1] 0

    $group_statistics
    $group_statistics$upper_thr
    [1] 42

    $group_statistics$pct_below
    [1] 0.9512195

    $group_statistics$mean
    [1] 31.46154

    $group_statistics$sd
    [1] 5.28078

    $group_statistics$n_obs
    [1] 39

    $conditions
    $conditions[[1]]
    $conditions[[1]]$column
    [1] "pregnant"

    $conditions[[1]]$value_this
    [1] TRUE

    $conditions[[1]]$comparison
    [1] "="

    $conditions[[1]]$value_comp
    [1] TRUE

    $tree_depth
    [1] 1

    $uses_NA_branch
    [1] FALSE

    $outlier_score
    [1] 0.01297346