# Cleaning Quadrat Data from CoralNet

# Introduction

This vignette will walk you through cleaning up quadrat data from CoralNet to produce an easy to analyze data frame.

A little about the data:

The data being cleaned will be the `softcoral_LQuads` and this vignette will build off of the cropping vignette as well. Data was collected by the Baum Lab and Kiritimati Field Teams. The `softcoral_LQuads` data are from photo quadrats (1m by 1m) which were randomly annotated with 100 random points each. At each of these annotated points, the substrate was identified. Photo quadrats were collected on Kiritimati Island in the Republic of Kiribati and document coral cover over time and space. The annotations and output of the data frame were produced using CoralNet and all annotations were done manually, by trained researchers.

# Cleaning the Data

First lets load the package, the dplyr package, tidyr package and the data, plus a few extras used to create this vignette.

    library(quadcleanR)
    library(dplyr)
    library(tidyr)
    library(shiny)
    library(knitr)
    library(kableExtra)

    data("softcoral_LQuads")

Now let me point out some unique aspects of this data:

    tail(softcoral_LQuads)

Image.ID

Image.name

Annotation.status

Points

AcCor

AcDig

Acr_arb

Acrop

AcroTab

Astreo

B_Acr_arb

B_Acro

B_Astre

BAT

B_Cosc

B_Echin

B_FavHal

B_Favia

B_FaviaM

B_FaviaS

B_FaviaSt

B_Favites

B_FavPent

B_Fung

BGard

B_GonEd

B_Herpo

B_HYDNO

B_HyExe

BlAcro.Cor

B_Lepta

B_Lepto

Blisop

B_Lobo

BlTurbFol

B_MOEN

B_MOFO

B_Monta

B_Monti

B_Oxyp

B_Paly

B_PaveDUER

B_Pavona

B_PEYDO

B_Plat

B_PMEAN

B_Pocillo

B_Porit

B_Psam

B_PVAR

B_Sando

B_UnkCoral

Cirr

COSC

ECHIN

Fav

FavHal

Favia

FaviaM

FaviaS

FaviaSt

FavPent

Fung

Gardin

GonEd

Herpo

HYDNO

HyExe

Isopora

Lepta

Lepto

Lobo

X.MOEN

X.MOFO

Monta

Monti

Oxyp

Paly

PaveDUER

Pavon

PEYDO

Plat

Plero

PMEAN

Pocill

Porit

Psam

PVAR

Sando

Tuba

TURB

UnkCoral

ANEM

B_Clad

B_Sinu

Clad

EncBry

EUR

HYDCO

Hydra

Mille

MOBI

Sarco

SECO

Sinu

Sponge

Stylas

UnkTUN

XmasW

ZOAN

B_Sarco

Sand

Sediment

SCRO

B_Loph

CYAN

Loph

Rubble

SHAD

Trans

Unc

AVRA

Caul

CCA

Dict

DICTY

Hali

Lobph

Macro

Mdict

Pad

Peysson

Turf

TURFH

Unidentified

4101

1368736

KI2019_site8_Q6.1.jpg

Confirmed

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

6

0

0

0

0

0

0

0

0

0

0

0

0

0

0

94

4102

1368737

KI2019_site8_Q6.jpg

Confirmed

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

100

4103

1368738

KI2019_site8_Q8.6.jpg

Confirmed

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

2

0

0

0

0

0

0

0

0

0

0

0

0

0

0

98

4104

1368739

KI2019_site8_Q9.7.jpg

Confirmed

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

1

0

0

0

0

0

0

0

0

0

0

0

0

0

98

4105

1368740

KI2019_site8_Q9.9.jpg

Confirmed

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

99

4106

ALL IMAGES

NA

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

100

This data as an `Image.ID` column which was arbitrarily added to this data set, so we are going to remove that as it holds no scientific value. There is also a final row which sums all quadrats but since we will be removing quadrats and points to clean these data up, we will remove that final row as well. The `Image.name` column is the unique ID for each photo quadrat, but it is very messy and not easy to use, so we will make this into new columns and add more information. `Annotation.status` is a column from CoralNet which tells if the annotations in each photo quadrat have been confirmed by human researchers or are only based on AI. The `Points` column tells us how many randomly annotated points there are in each quadrat, and since they are all 54, we know these data are from the smaller quadrats. The rest of the columns are the different coral and substrate IDs and how many points where annotated for each tag in each photo.

So first, lets remove unneeded columns and make sure we are only working with “Confirmed” annotations.

    LQuad_confirmed <- softcoral_LQuads %>% filter(Annotation.status == "Confirmed") %>% select(-c(Image.ID, Points, Annotation.status))

Now we will separate the `Image.name` column into more descriptive columns.

    LQuad_separated <- separate(LQuad_confirmed, Image.name, sep="_", into=c("Field.Season", "Site","Quadrat"))

But if you notice, there are still .jpg and .jpeg in the quadrat names, so lets remove those, and change the naming of siteT19 to site40. At the same time I will remove site8.5, any DEEP sites and any MPQs (mega photo quadrat) from the data set.

    LQuad_nojpg <- rm_chr(LQuad_separated, c(".jpg", ".jpeg"))

    LQuad_site40 <- change_values(LQuad_nojpg, "Site", "siteT19", "site40")

    LQuad_noDEEP_site8.5 <- keep_rm(LQuad_site40, c("DEEP", "site8.5"), select = "row", exact = FALSE, colname = "Site", keep = FALSE)

    LQuad_noMPQ <- keep_rm(LQuad_noDEEP_site8.5, c("MPQ"), select = "row", exact = FALSE, colname = "Quadrat", keep = FALSE)

Now lets look at the levels of some of these columns.

    unique(LQuad_noMPQ$Field.Season)

    ##  [1] "KI2013"  "KI2014"  "KI2015a" "KI2015b" "KI2015c" "KI2015d" "KI2016a"
    ##  [8] "KI2016b" "KI2017"  "KI2018"  "KI2019"

    unique(LQuad_noMPQ$Site)

    ##  [1] "site14" "site15" "site19" "site1"  "site20" "site22" "site23" "site24"
    ##  [9] "site25" "site26" "site27" "site30" "site32" "site34" "site35" "site3"
    ## [17] "site6"  "site8"  "site9"  "site38" "site5"  "site10" "site12" "site33"
    ## [25] "site37" "site40" "site13" "site18" "site31" "site36" "site7"

Next lets update the column names for this data frame. The column names are currently set as the tag shorthand used during the annotation process, but now I want them to better reflect the actual substrate names.

    data("coral_labelset")
    head(coral_labelset)

short_name | full_name | taxonomic_name | functional_group | life_history
--- | --- | --- | --- | ---
Rubble | Broken_coral_rubble | Broken_coral_rubble | Abiotic_Substrate | Not_Coral
SCRO | Consolidated_hard_rock | Consolidated_hard_rock | Abiotic_Substrate | Not_Coral
Sand | Sand | Sand | Abiotic_Substrate | Not_Coral
Sediment | Sediment | Sediment | Abiotic_Substrate | Not_Coral
CCA | Crustose_coralline_algae | Crustose_coralline_algae | Crustose_Algae | Not_Coral
Peysson | Peyssonnelia | Peyssonnelia | Crustose_Algae | Not_Coral

This is what the label set document looks like, but you could also make this in R by joining a series of vectors. Now lets fix the column names.

    LQuad_colnames <- change_names(LQuad_noMPQ, coral_labelset, "short_name", "full_name")
    names(LQuad_colnames)[1:16]

    ##  [1] "Field.Season"                  "Site"
    ##  [3] "Quadrat"                       "Acropora_corymbose"
    ##  [5] "Acropora_digitate"             "Acropora_arborescent"
    ##  [7] "Acropora"                      "Acropora_tabulate"
    ##  [9] "Astreopora"                    "Bleached_Acropora_arborescent"
    ## [11] "Bleached_Acropora"             "Bleached_Astreopora"
    ## [13] "Bleached_Acropora_tabulate"    "Bleached_Coscinarea"
    ## [15] "Bleached_Echinophyllia"        "Bleached_Favites_halicora"

Much better.

Now for these photo quadrats, the Shadow, Transect_hardware and Unclear tags need to be removed and not used when we calculate percent cover. If I was going to use this data to do a diversity analysis with hard corals, I would also include unknown_hard_coral, and Bleached_unknown_hard_coral to this list, but we are going to clean this data for soft coral analyses, so we will leave them in.

    LQuad_colnames <-  mutate_at(LQuad_colnames, c(4:134), as.numeric)

    LQuad_usable <- usable_obs(LQuad_colnames, c("Shadow", "Transect_hardware", "Unclear"),
                                max = TRUE, cutoff = 10)

    LQuad_removed <- usable_obs(LQuad_colnames, c("Shadow", "Transect_hardware", "Unclear"),
                                 max = TRUE, cutoff = 10, above_cutoff = TRUE)

By identifying how many usable points there are in each quadrat, and removing any quadrats that had over 10% of the identified points unusable, we have removed 0 quadrats from analysis, which you could view with the `LQuad_removed` data frame but as we have no quadrats to removed, it will be an empty data frame.

Now we know how many usable annotations for each tag there are in each photo quadrat. Lets convert this into a proportion cover now. This may take a minute as there are many rows.

    LQuad_cover <- cover_calc(LQuad_usable, names(LQuad_usable[,4:131]), prop = TRUE)

This data frame is now nicely formatted and could be used for many community based analyses. This might be a great stopping point for some analyses, but to further clean this up I am going to convert this into long format data.

    LQuad_long <- LQuad_cover %>% select(-c(unusable)) %>% pivot_longer(cols = names(LQuad_cover[,4:131]), names_to = "Tag_Name", values_to = "prop_cover")

One thing you may notice by looking at the `Tag_Name` column, is that these species names are not unique species, but there are duplicates of the same species, categorized into bleaching and non bleaching forms. For any kind of diversity analysis, this would inflate the number of different species, so it is important to combine different forms of the same species if diversity analyses are being done.

For this clean up, we will walk through 3 ways of dealing with this based on what you want to accomplish.

Option A. Categorizing rows.

If you want to use your data in this long format, want to just categorize everything and you will use these various categories based on your different research questions, you could just add a bunch of category columns like so:

    A_LQuad_Bleach <- categorize(LQuad_long, "Tag_Name", values = c("Bleach"), name = "Bleached", binary = TRUE, exact = FALSE)

This categorizes each `Tag_Name` to whether it is a bleaching or nonbleaching tag.

And you could also add other information in if you have it, like taxonomy.

    A_LQuad_Taxa <- categorize(A_LQuad_Bleach, "Tag_Name", values = coral_labelset$full_name, name = "Taxonomic_Name", binary = FALSE, categories = coral_labelset$taxonomic_name)

Option B. Categorizing rows and then combining.

Now after you categorize your rows, perhaps you want to have all the cover values summed at a different level, like at the taxonomy level. To do this, the `summarise()` function from `dplyr` will work great.

    B_LQuad_taxonomy <- A_LQuad_Taxa %>% group_by(Field.Season, Site, Quadrat, Taxonomic_Name) %>% summarise(prop_cover = sum(prop_cover))

    ## `summarise()` has grouped output by 'Field.Season', 'Site', 'Quadrat'. You can
    ## override using the `.groups` argument.

Option C. Wide format summing columns

If you wanted to keep the data in a wide format, and sum columns based on taxonomy, to allow for community level analyses, you could also use the `sum_cols()` function. To do this, we first need a vector of what to change the names too, which can be done with a simple match, unless you have a vector with the new names already in the right order.

    current_names <- colnames(LQuad_cover[,4:131])
    new_names <- coral_labelset[match(current_names, coral_labelset$full_name),]$taxonomic_name

    LQuad_wide_summed <- sum_cols(LQuad_cover, from = current_names, to = new_names)

Whichever of the options you choose, you will be able to customize the data to your analysis needs. After that your data is nearly cleaned. Some other things you may want to add would be environmental data, or more taxonomic data. The `add_data()` function can help with adding multiple columns from a data set at a time.

    B_LQuad_LH_FG <- add_data(B_LQuad_taxonomy, coral_labelset, cols = c("functional_group", "life_history"), data_id = "Taxonomic_Name", add_id = "taxonomic_name", number = 5)

    data("environmental_data")

    B_LQuad_enviro <- add_data(B_LQuad_LH_FG, environmental_data, cols = c("HD_Cat", "HD_Cont", "NPP",  "WE", "Region", "WaveEnergy"), data_id = "Site", add_id = "Site", number = 4)

The final things I will add to this data to get it in shape for analysis is a final categorization of the study years based on the timing of the 2015/2016 El Niño and subset the species to only soft coral.

    B_LQuad_timeblock <- categorize(B_LQuad_enviro, column = "Field.Season", values = unique(B_LQuad_enviro$Field.Season), name = "TimeBlock", binary = FALSE, exact = TRUE, categories = c(rep("Before", times = 4), rep("During", times = 3), rep("After", times = 4)))

    final_cleaned <- keep_rm(B_LQuad_timeblock, values = "Soft_coral", select = "row", colname = "functional_group")

# Exploration

This data has now been sufficiently cleaned and can be used for many different analyses. Often once data has been cleaned, the first step is to start exploring the data. One thing we can look at is the sample sizes, to see how many quadrats I have over the different sites and years.

    sample_size(final_cleaned, dim_1 = "Site", dim_2 = "Field.Season", count = "Quadrat")

| KI2013 | KI2014 | KI2015a | KI2015b | KI2015c | KI2015d | KI2016a | KI2016b | KI2017 | KI2018 | KI2019
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
site1 | 28 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 29 | 0 | 0
site14 | 13 | 21 | 0 | 0 | 30 | 0 | 16 | 0 | 30 | 0 | 0
site15 | 24 | 0 | 29 | 15 | 29 | 0 | 29 | 30 | 30 | 30 | 0
site19 | 21 | 0 | 0 | 0 | 28 | 0 | 0 | 0 | 26 | 0 | 0
site20 | 31 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 19 | 0 | 0
site22 | 19 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
site23 | 25 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 29 | 0 | 0
site24 | 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
site25 | 26 | 25 | 0 | 0 | 31 | 0 | 0 | 0 | 17 | 27 | 0
site26 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 30 | 0 | 0
site27 | 28 | 30 | 30 | 30 | 29 | 30 | 30 | 30 | 29 | 29 | 30
site3 | 28 | 32 | 0 | 0 | 30 | 0 | 0 | 0 | 29 | 28 | 30
site30 | 29 | 25 | 30 | 30 | 30 | 0 | 30 | 30 | 30 | 21 | 30
site32 | 30 | 0 | 0 | 30 | 30 | 34 | 30 | 30 | 30 | 30 | 30
site34 | 28 | 31 | 0 | 30 | 29 | 0 | 30 | 30 | 29 | 27 | 30
site35 | 26 | 31 | 30 | 30 | 29 | 9 | 26 | 30 | 29 | 30 | 30
site6 | 29 | 0 | 0 | 0 | 0 | 0 | 0 | 30 | 30 | 0 | 30
site8 | 29 | 27 | 30 | 30 | 30 | 27 | 30 | 30 | 30 | 30 | 30
site9 | 26 | 0 | 0 | 0 | 0 | 0 | 30 | 0 | 30 | 0 | 0
site38 | 0 | 31 | 0 | 0 | 30 | 0 | 29 | 0 | 0 | 28 | 30
site5 | 0 | 0 | 0 | 29 | 30 | 0 | 30 | 30 | 27 | 29 | 30
site10 | 0 | 0 | 0 | 0 | 30 | 0 | 0 | 0 | 25 | 0 | 0
site12 | 0 | 0 | 0 | 0 | 0 | 0 | 22 | 30 | 31 | 0 | 30
site33 | 0 | 0 | 0 | 0 | 0 | 0 | 30 | 0 | 31 | 0 | 0
site37 | 0 | 0 | 0 | 0 | 0 | 0 | 30 | 30 | 30 | 30 | 30
site40 | 0 | 0 | 0 | 0 | 0 | 0 | 23 | 30 | 31 | 0 | 30
site13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 23 | 0 | 0
site18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 20 | 0 | 0
site31 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 20 | 0 | 30
site36 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 27 | 0 | 0
site7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 25 | 0 | 0

Visualizing the data can be easy with a built in shiny app function. To see an example shiny app you can go here or the following code.

A good combination to examine this shiny with is : - y-axis: prop_cover - x-axis: Field.Season - color: TimeBlock (treat as discrete) - facet: HD_cat - group by: Field.Season, TimeBlock, Site and HD_Cat - view as a box plot

    runGitHub("quadcleanR", username = "DominiqueMaucieri", subdir = "inst/shiny/example", ref = "main")

The data used in this example is this data set we have just cleaned. However this does not help you as this is not your own data. You can use the following code to produce a shiny app of your data, which you can then explore your data.

    visualize_app(data = final_cleaned, xaxis = colnames(final_cleaned[,1:13]), yaxis = "prop_cover")