# Development features of isoreader

#### 2021-10-15

This vignette introduces some of the development features of the isoreader package and is aimed primarily at code contributors interested in expanding its functionality or helping with bug fixes.

# Adding new file format readers

Testing out new file format readers is easiest by registering a new reader function for a specific file extension using `iso_register_dual_inlet_file_reader` and `iso_register_continuous_flow_file_reader`, respectively. Both require an extension (e.g. `".ext"`), name of the new reader function (`"new_reader"`), and optionally a description. Both functions automatically return a data frame with a list of all registered reader. Overwriting of existing readers with a different function requires an explicit `overwrite = TRUE` flag. All reader functions must accept an isoreader data structure object (`ds`) as the first argument, a list of reader specific options as the second argument (`options`), and should return the structure with data filled in for downstream isoreader operations to work smoothly. The following minimal example illustrates how to do this with the `new_reader` function simply printing out the layout of the provided data structure skeleton `ds`.

    new_reader <- function(ds, options = list()) {
      isoreader:::log_message("this is the new reader!")
      str(ds)
      return(ds)
    }

    # register new reader
    readers <- iso_register_dual_inlet_file_reader(".new.did", "new_reader")
    knitr::kable(readers)

type | call | extension | func | cacheable | post_read_check | description | software | env
--- | --- | --- | --- | --- | --- | --- | --- | ---
dual inlet | iso_read_dual_inlet | .caf | iso_read_caf | TRUE | TRUE | Dual Inlet file format (older) | Isodat | isoreader
dual inlet | iso_read_dual_inlet | .did | iso_read_did | TRUE | TRUE | Dual Inlet file format (newer) | Isodat | isoreader
dual inlet | iso_read_dual_inlet | .txt | iso_read_nu | TRUE | TRUE | Dual Inlet file format | Nu | isoreader
continuous flow | iso_read_continuous_flow | .cf | iso_read_cf | TRUE | TRUE | Continuous Flow file format (older) | Isodat | isoreader
continuous flow | iso_read_continuous_flow | .dxf | iso_read_dxf | TRUE | TRUE | Continuous Flow file format (newer) | Isodat | isoreader
continuous flow | iso_read_continuous_flow | .iarc | iso_read_flow_iarc | TRUE | TRUE | Continuous Flow data archive | ionOS | isoreader
scan | iso_read_scan | .scn | iso_read_scn | TRUE | TRUE | Scan file format | Isodat | isoreader
continuous flow | iso_read_continuous_flow | .cf.rds | iso_read_rds | FALSE | FALSE | R Data Storage | isoreader | isoreader
dual inlet | iso_read_dual_inlet | .di.rds | iso_read_rds | FALSE | FALSE | R Data Storage | isoreader | isoreader
scan | iso_read_scan | .scan.rds | iso_read_rds | FALSE | FALSE | R Data Storage | isoreader | isoreader
dual inlet | iso_read_dual_inlet | .new.did | new_reader | TRUE | TRUE | NA | NA | R_GlobalEnv

    # copy an example file from the package with the new extension
    iso_get_reader_example("dual_inlet_example.did") %>% file.copy(to = "example.new.did")
    #> [1] TRUE

    # read the file
    iso_read_dual_inlet("example.new.did", read_cache = FALSE)
    #> Info: preparing to read 1 data files (all will be cached)...
    #> Info: reading file 'example.new.did' with '.new.did' reader...
    #> Info: this is the new reader!
    #> List of 7
    #>  $ version          :Classes 'package_version', 'numeric_version'  hidden list of 1
    #>   ..$ : int [1:3] 1 3 1
    #>  $ read_options     :List of 4
    #>   ..$ file_info        : logi TRUE
    #>   ..$ method_info      : logi TRUE
    #>   ..$ raw_data         : logi TRUE
    #>   ..$ vendor_data_table: logi TRUE
    #>  $ file_info        : tibble [1 × 6] (S3: tbl_df/tbl/data.frame)
    #>   ..$ file_id      : chr "example.new.did"
    #>   ..$ file_root    : chr "."
    #>   ..$ file_path    : chr "example.new.did"
    #>   ..$ file_subpath : chr NA
    #>   ..$ file_datetime: POSIXct[1:1], format: NA
    #>   ..$ file_size    : int 134446
    #>  $ method_info      : list()
    #>  $ raw_data         : tibble [0 × 0] (S3: tbl_df/tbl/data.frame)
    #>  Named list()
    #>  $ vendor_data_table: tibble [0 × 0] (S3: tbl_df/tbl/data.frame)
    #>  Named list()
    #>  $ bgrd_data        : tibble [0 × 0] (S3: tbl_df/tbl/data.frame)
    #>  Named list()
    #>  - attr(*, "class")= chr [1:2] "dual_inlet" "iso_file"
    #>  - attr(*, "problems")= tibble [0 × 3] (S3: tbl_df/tbl/data.frame)
    #>   ..$ type   : chr(0)
    #>   ..$ func   : chr(0)
    #>   ..$ details: chr(0)
    #> Info: finished reading 1 files in 0.09 secs
    #> Dual inlet iso file 'example.new.did': 0 cycles, 0 ions ()
    file.remove("example.new.did")
    #> [1] TRUE

Note that for parallel processing to work during the read process (`parallel = TRUE`), isoreader needs to know where to find the new reader function. It will figure this out automatically as long as the function name is unique but if this fails (or to be on the safe side), please specify e.g. `env = "R_GlobalEnv"` or `env = "newpackage"` during the reader registration. Also note that isoreader will not automatically know where to find all functions called from within the new reader function if they are not part of base R and it is recommended to make all outside calls explicit (e.g. `dplyr::filter(...)`) to preempt this potential problem. For info messages and warnings to work with the progress bar and in parallel reads, make sure to use `isoreader:::log_message(...)` and `isoreader:::log_warning(...)` instead of base R’s `message(...)` and `warning(...)`.

If you have designed and tested a new reader, please consider contributing it to the `isoreader` github repository via pull request.

# Processing hooks

Isoreader defines two processing hooks at the beginning and end of reading an individual file. This is useful for integration into pipelines that require additional output (such as GUIs) but is also sometimes useful for debugging purposes. The expressions are evaluated in the context of the `isoreader:::read_iso_file` function and have access to all parameters passed to this function, such as e.g. `file_n` and `path`. Same as for new readers: for info messages and warnings to work with the progress bar and in parallel reads, make sure to use `isoreader:::log_message(...)` and `isoreader:::log_warning(...)` instead of base R’s `message(...)` and `warning(...)`. The main difference between the two is that `log_message()` will honor the `quiet = TRUE` flag passed to the main `iso_read...()` call whereas `log_warning()` will always show its message no matter the `quiet` setting.

    isoreader:::set_read_file_event_expr({
      isoreader:::log_message(sprintf("starting file #%.d, named '%s'", file_n, basename(path)))
    })
    isoreader:::set_finish_file_event_expr({
      isoreader:::log_message(sprintf("finished file #%.d", file_n))
    })

    c(
      iso_get_reader_example("dual_inlet_example.did"),
      iso_get_reader_example("dual_inlet_example.caf")
    ) %>% iso_read_dual_inlet(read_cache = FALSE)
    #> Info: preparing to read 2 data files (all will be cached)...
    #> Info: reading file 'dual_inlet_example.did' with '.did' reader...
    #> Info: starting file #1, named 'dual_inlet_example.did'
    #> Info: finished file #1
    #> Info: reading file 'dual_inlet_example.caf' with '.caf' reader...
    #> Info: starting file #2, named 'dual_inlet_example.caf'
    #> Info: finished file #2
    #> Info: finished reading 2 files in 2.19 secs
    #> Data from 2 dual inlet iso files:
    #> # A tibble: 2 × 6
    #>   file_id                raw_data file_info method_info vendor_data_tab… file_path
    #>   <chr>                  <glue>   <chr>     <chr>       <chr>            <chr>
    #> 1 dual_inlet_example.did 7 cycle… 16 entri… standards,… 7 rows, 8 colum… dual_inl…
    #> 2 dual_inlet_example.caf 8 cycle… 22 entri… standards,… 8 rows, 9 colum… dual_inl…

    isoreader:::initialize_options() # reset all isoreader options

# Debugging isoreader

The best way to start debugging an isoreader call is to switch the package into debug mode. This is done using the internal `iso_turn_debug_on()` function. This enables debug messages, turns caching off by default so files are always read anew, and makes the package keep more information in the isofile objects. It continues to catch errors inside file readers (keeping track of them in the problems) unless you set `iso_turn_debug_on(catch_errors = FALSE)`, in which case no errors are caught and stop the processing so you get the full traceback and debugging options of your IDE.

## Debugging binary file reads (Isodat)

Errors during the binary file reads usually indicate the approximate position in the file where the error was encountered. The easiest way to get started on figuring out what the file looks like at that position is to use a binary file editor and jump to the position. For a sense of the interpreted structure around that position, one can use the internal function `map_binary_structure` which tries to apply all frequently occurring binary patterns recognized by isoreader. The binary representation of the source file is only available if in debug mode but if debug mode is ON, it can be accessed as follows:

    # turn on debug mode
    isoreader:::iso_turn_debug_on()
    #> Info: debug mode turned on, error catching turned on, caching turned off
    # read example file
    ex <- iso_get_reader_example("dual_inlet_example.did") %>%
      iso_read_dual_inlet(quiet = TRUE)
    # access binary
    bin <- ex$binary
    # use structure mapping
    bin %>%
      isoreader:::move_to_pos(1340) %>%
      isoreader:::map_binary_structure(length = 200)
    #> # Binary data structure:
    #> 0001340: {4d 65 73 73 61 67 65 44 61 74 61}
    #> 0001351:   <etx>
    #> 0001355:   {/}<fef-0><fef-12>{Peak Center foun..}<4x00><1-000><fef-1c>{Peak Center found at [61032]}<fef-09>{CUserInfo}<ffff>{00 00 00 00 1b 80}
    #> 0001501: <etx>
    #> 0001505: {/}<fef-0><fef-12>

This structure representation shows recognized control elements in `<...>` and data elements in `{...}` which are converted to text or numeric representation if the interpretation is unambiguous, or plain hexadecimal characters if the nature of the data cannot be determined with certainty. Because this function tries all possible control elements and data interpretations, it is quite slow and may take a while if run for large stretches of binary code (i.e. if the `length` parameter is very long).

For an overview of all the control elements that are currently consider, use the internal `get_ctrl_blocks_config_df()` function.

    isoreader:::get_ctrl_blocks_config_df()
    #> # A tibble: 26 × 3
    #>    block    regexp                              hexadecimal
    #>    <chr>    <chr>                               <chr>
    #>  1 del-nl   "\x7f\x85"                          7f 85
    #>  2 eop-nl   "܅"                                 dc 85
    #>  3 0b-80    "\v\x80"                            0b 80
    #>  4 e0-81    "\xe0\x81"                          e0 81
    #>  5 ce-80    "\u0380"                            ce 80
    #>  6 ce-8a    "Ί"                                 ce 8a
    #>  7 ee-85    "\xee\x85"                          ee 85
    #>  8 75-84    "u\x84"                             75 84
    #>  9 ff-80    "\\x00\xff\x80\\x00"                5c 78 30 30 ff 80 5c 78 30 30
    #> 10 07-80-id "\x05\x80.\xff(\\x00|\x80|\xff){2}" 05 80 2e ff 28 5c 78 30 30 7c 8…
    #> # … with 16 more rows

Additional information can be gleaned from the so-called control blocks, which are larger structural elements of Isodat binary files and are kept in a data frame within the binary object (again only available in debug mode).

    bin$C_blocks
    #> # A tibble: 107 × 5
    #>    id1   id2   block                 start   end
    #>    <chr> <chr> <chr>                 <int> <dbl>
    #>  1 06    0b    CFileHeader               1    17
    #>  2 01    0b    CTimeObject             182   198
    #>  3 02    04    CStr                    241   250
    #>  4 01    0a    CDataIndex              471   486
    #>  5 01    11    CSeqLineIndexData       513   535
    #>  6 03    05    CData                   588   598
    #>  7 01    13    CDualInletBlockData    1133  1157
    #>  8 01    10    CMeasurmentInfos       1240  1261
    #>  9 01    15    CISLScriptMessageData  1324  1350
    #> 10 01    11    CMeasurmentErrors      1945  1967
    #> # … with 97 more rows

Same as for specific byte positions, one can use the control blocks to navigate the file and `map_binary_structure`.

    bin %>%
      isoreader:::move_to_C_block("CMethod") %>%
      isoreader:::map_binary_structure(length = 200)
    #> # Binary data structure:
    #> 0080269: <etx>
    #> 0080273:   {-}<fef-0><fef-06>{Method}<4x00>
    #> 0080299:   <stx>
    #> 0080303:     <7-000>
    #> 0080307:     <C-01-09 CMolecule>
    #> 0080322:   <etx>
    #> 0080326:   {-}<fef-0d>{Eval@Molecule}<fef-0d>{Eval@Molecule}<4x00><1-000><fef-03>{CO2}
    #> 0080406:   <C-00-0b CPartMirror>
    #> 0080423:   <C-02-13 CMethodPrintoutDesc>
    #> 0080448: <etx>
    #> 0080452: {-}<fef-0><fef-0><4x00>
    #> 0080466: <stx>