| Elektra 0.11.0

Plugin: quickdump

# Benchmarks

## <tt>benchmark_storage</tt>

The following table shows a summary of the results of a `benchmark_storage` run: NOTE: `factor` is always `dump / quickdump` (bigger = `quickdump` is faster), this test was only run with version 1, see below for a comparison of with version 2

operation | dump (µs) | quickdump (µs) | factor
--- | --- | --- | ---
write keyset | 81661 | 4741 | 17.224
read keyset | 94782 | 44673 | 2.122
iterate keyset | 125 | 135 | 0.926
delete keyset | 1823 | 3518 | 0.518
re-read keyset | 92939 | 43749 | 2.124
strcmp key name | 609 | 1826 | 0.334
strcmp key value | 684 | 761 | 0.899

## <tt>benchmark_plugingetset</tt>

A script was used to generate KeySets with `2`, `200`, `2,000`, `200,000` and `2,000,000` Keys. For keynames we used increasing numbers (prefixed with `_` to keep correct order), each key had a randomized value of 15 characters. Additionally we added 4 metakeys to each key. The names and values of these were created similarly.

The parent key used was `dir:/tests/bench`.

We then used `hyperfine` to compare the performance of `dump` and `quickdump`.

hyperfine --min-runs 25 'benchmark_plugingetset <path> <parent> dump' 'benchmark_plugingetset <path> <parent> quickdump'

This means we used the same key sets for each of the runs, but the key values shouldn't really matter. It also means that no shared metadata (`keyCopyMeta`) was involved.

We also used a modified version of `benchmark_plugingetset` that returned after the `get` call, to test the `get` performance by itself. Because `set` doesn't scale linearly because of the checks for `keyCopyMeta`, the largest KeySet was only used for the `get` only version. (The test was aborted after the initial run of `dump` took longer than all the runs for `quickdump` together.)

The complete results can be found in this repository and the files used for these benchmarks are quite big and can be found in another separate repository.

### Raw data sizes

For reference in the file size tests below, here is a list of the raw data sizes of the keysets. These sizes are calculated as the sum of: 1) length of all keynames 2) length of all key values 3) length of all metakeys 4) length of all meta values All lengths include a null terminator, sizes are calculated once with (WP) and once without the parent key (WOP).

no. of keys | raw size WP (B) | raw size WOP (B)
--- | --- | ---
2 | 252 | 220
200 | 25784 | 22584
2000 | 262786 | 229786
200000 | 26977790 | 23777790
20000000 | 273777792 | 241777792

### Version 1

#### File sizes

`factor` is `dump / quickdump` like above (bigger = `quickdump` has smaller files)

no. of keys | dump (B) | quickdump (B) | factor
--- | --- | --- | ---
2 | 430 | 412 | 1.04
200 | 41206 | 40988 | 1.01
2000 | 415807 | 413788 | 1.00
200000 | 42377809 | 42177788 | 1.00
2000000 | 427777810 | 425777788 | 1.00

#### get and set

The values are mean ± standard deviation. `factor` is `dump / quickdump` like above

no. of keys | dump (s) | quickdump (s) | factor
--- | --- | --- | ---
2 | 0.0016 ± 0.0003 | 0.0006 ± 0.0001 | 2.67
200 | 0.0091 ± 0.0009 | 0.0018 ± 0.0000 | 5.06
2000 | 0.0770 ± 0.0030 | 0.0127 ± 0.0008 | 6.06
200000 | 10.7876 ± 4.2564 | 1.2640 ± 0.0061 | 8.53

#### get only

The values are mean ± standard deviation. `factor` is `dump / quickdump` like above

no. of keys | dump (s) | quickdump (s) | factor
--- | --- | --- | ---
2 | 0.0014 ± 0.0000 | 0.0005 ± 0.0000 | 2.8
200 | 0.0032 ± 0.0005 | 0.0013 ± 0.0002 | 2.46
2000 | 0.0179 ± 0.0002 | 0.0086 ± 0.0008 | 2.08
200000 | 1.6830 ± 0.0222 | 0.8515 ± 0.0048 | 1.98
2000000 | 17.1814 ± 0.2201 | 8.8808 ± 0.0344 | 1.93

### Version 2

Running the same `benchmark_plugingetset` tests with version 2 of the plugin yields:

#### File sizes

`factor` is `v1 / v2` like above (bigger is better)

no. of keys | quickdump v1 (B) | quickdump v2 (B) | factor
--- | --- | --- | ---
2 | 412 | 380 | 1.08
200 | 40988 | 23788 | 1.72
2000 | 413788 | 381788 | 1.08
200000 | 42177788 | 38977788 | 1.08
2000000 | 425777788 | 393777788 | 1.08

#### get and set

The values are mean ± standard deviation. `factor` is `v1 / v2`, i.e. bigger is better

no. of keys | quickdump v1 (s) | quickdump v2 (s) | factor
--- | --- | --- | ---
2 | 0.0006 ± 0.0001 | 0.0013 ± 0.0035 | 0.47
200 | 0.0018 ± 0.0000 | 0.0018 ± 0.0002 | 1.02
2000 | 0.0127 ± 0.0008 | 0.0112 ± 0.0021 | 1.13
200000 | 1.2640 ± 0.0061 | 1.2602 ± 0.0372 | 1.00
2000000 | - | 13.2104 ± 0.1185 | -

#### get only

The values are mean ± standard deviation. `factor` is `v1 / v2` like above

no. of keys | quickdump v1 (s) | quickdump v2 (s) | factor
--- | --- | --- | ---
2 | 0.0005 ± 0.0000 | 0.0006 ± 0.0001 | 0.88
200 | 0.0014 ± 0.0002 | 0.0010 ± 0.0001 | 1.33
2000 | 0.0086 ± 0.0008 | 0.0072 ± 0.0005 | 1.20
200000 | 0.8516 ± 0.0048 | 0.6413 ± 0.0082 | 1.33
2000000 | 8.8809 ± 0.0345 | 6.3756 ± 0.0443 | 1.39

### Version 3

Running the same `benchmark_plugingetset` tests with version 3 of the plugin yields:

#### File sizes

`factor` is `v1 / v2` like above (bigger is better)

no. of keys | quickdump v2 (B) | quickdump v3 (B) | factor
--- | --- | --- | ---
2 | 380 | 240 | 1.58
200 | 23788 | 23788 | 1.00
2000 | 381788 | 241788 | 1.58
200000 | 38977788 | 24977788 | 1.56
2000000 | 393777788 | 253777788 | 1.55

#### get and set

The values are mean ± standard deviation. `factor` is `v2 / v3`, i.e. bigger is better

no. of keys | quickdump v2 (s) | quickdump v3 (s) | factor
--- | --- | --- | ---
2 | 0.0013 ± 0.0035 | 0.0012 ± 0.0029 | 1.08
200 | 0.0018 ± 0.0002 | 0.0017 ± 0.0002 | 1.05
2000 | 0.0112 ± 0.0021 | 0.0106 ± 0.0011 | 1.06
200000 | 1.2602 ± 0.0372 | 1.1473 ± 0.0287 | 1.10
2000000 | 13.2104 ± 0.1185 | 11.5082 ± 0.2624 | 1.15

#### get only

The values are mean ± standard deviation. `factor` is `v2 / v3` like above

no. of keys | quickdump v2 (s) | quickdump v3 (s) | factor
--- | --- | --- | ---
2 | 0.0006 ± 0.0001 | 0.0006 ± 0.0001 | 0.99
200 | 0.0010 ± 0.0001 | 0.0012 ± 0.0001 | 0.84
2000 | 0.0072 ± 0.0005 | 0.0070 ± 0.0004 | 1.03
200000 | 0.6413 ± 0.0082 | 0.6288 ± 0.0218 | 1.02
2000000 | 6.3756 ± 0.0443 | 6.2309 ± 0.0462 | 1.02