| Elektra 0.11.0

Plugin: toml

- infos = Information about the toml plugin is in keys below
- infos/author = Jakob Fischer jakob.nosp@m.fisc.nosp@m.her93.nosp@m.@gma.nosp@m.il.co.nosp@m.m
- infos/licence = BSD
- infos/provides = storage/toml
- infos/needs = base64
- infos/recommends = type
- infos/placements = getstorage setstorage
- infos/status = unittest experimental unfinished
- infos/metadata = order comment/# comment/#/start comment/#/space type tomltype origvalue
- infos/description = This storage plugin reads and writes TOML files using Flex and Bison.

# Introduction

This plugin is a storage plugin for reading and writing TOML files. The plugin retains most of the file structure of a read TOML file such as comments, empty lines and TOML tables. It supports all kinds of TOML specific types and tables, including nested inline tables and multiline strings.

# Requirements

The plugin needs Flex (>=2.6.2) and Bison (>=3) for parsing TOML files.

# Types

## Reading

On reading, the plugin will set the `type` metakey for strings, integers, floats and boolean values, if applicable. For decimal integers, the metakey is set to `long_long`. For binary/octal/hexadecimal integers, the metakey is set to `unsigned_long_long`. For floats, the metakey will be set to `double`. These types are chosen to conform with the TOML format as stated on the offical TOML page.

On reading, non-decimal integers will get converted to decimal. The non-decimal representation will be stored in the `origvalue` metakey of the key. When writing a key, the value of that metakey will be written instead, in order to retain the original format. Note that the `origvalue` metakey gets removed if the value of the key changes.

sudo kdb mount test.toml user:/tests/storage/types toml type
# Create a TOML file with 4 keys
mkdir -p $(dirname $(kdb file user:/tests/storage/types))
echo 'plain_decimal = 1000' >> `kdb file user:/tests/storage/types`
echo 'file_permissions = 0o777' >> `kdb file user:/tests/storage/types`
echo 'pi = 3.1415' >> `kdb file user:/tests/storage/types`
echo 'division_gone_wrong = -inf' >> `kdb file user:/tests/storage/types`
# Print the content of the toml file
cat `kdb file user:/tests/storage/types`
# > plain_decimal = 1000
# > file_permissions = 0o777
# > pi = 3.1415
# > division_gone_wrong = -inf
# Print types and values of the keys with `kdb`
kdb meta-get 'user:/tests/storage/types/plain_decimal' 'type'
# > long_long
kdb get 'user:/tests/storage/types/plain_decimal'
# > 1000
kdb meta-get 'user:/tests/storage/types/file_permissions' 'type'
# > unsigned_long_long
# The octal value will be converted to decimal
kdb get 'user:/tests/storage/types/file_permissions'
# > 511
kdb meta-get 'user:/tests/storage/types/pi' 'type'
# > double
kdb get 'user:/tests/storage/types/pi'
# > 3.1415
kdb meta-get 'user:/tests/storage/types/division_gone_wrong' 'type'
# > double
kdb get 'user:/tests/storage/types/division_gone_wrong'
# > -inf
# Cleanup
kdb rm -r user:/tests/storage/types
sudo kdb umount user:/tests/storage/types

## Writing

On writing, for most values, the plugin will infer the appropriate type and will write them accordingly. This means, values, that match a TOML float, integer or date will be written without any quotes around them. If a value does not match any of these types, it will be written as a string.

The plugin uses an existing `type` metakey only to check if it should write a value as a `string` or a `boolean`.

With this functionality, numbers can be written as a string to the file, if wanted.

To write a boolean value, the `type` metakey must be set to `boolean` for the key. Otherwise, no conversion to the TOML boolean values will take place. Per default, Elektra uses 0/1 to represent boolean values.

sudo kdb mount test.toml user:/tests/storage/types toml type
# Create a key, which may be a integer, boolean or string
kdb set 'user:/tests/storage/types/value' '1'
# The plugin infers `long_long` for this value
kdb meta-get 'user:/tests/storage/types/value' 'type'
# > long_long
# The value is written as an integer
cat `kdb file user:/tests/storage/types`
# > value = 1
# Manually set the `type` metakey to boolean
kdb meta-set 'user:/tests/storage/types/value' 'type' 'boolean'
# The value is written as a boolean
cat `kdb file user:/tests/storage/types`
# > value = true
# Manually set the `type` metakey to string
kdb meta-set 'user:/tests/storage/types/value' 'type' 'string'
# The value is written as a string
cat `kdb file user:/tests/storage/types`
# > value = "1"
# Cleanup
kdb rm -r user:/tests/storage/types
sudo kdb umount user:/tests/storage/types

# Numbers

The plugin supports reading and writing of any kind of number supported by the TOML format, such as floating point numbers and binary/octal/decimal/hexadecimal integers. To write a non-decimal integer, add the corresponding prefix to the number (`0b` for binary, `0o` for octal, `0x` for hexadecimal). The value will be written in the given base to the file, but converted to decimal within Elektra (see Reading). Note that the plugin doesn't warn about invalid prefix/digit combinations. If the combination is not valid, it will be written as a string instead.

If the `type` plugin is enabled and you want to change the value of an existing number key which needs conversion (all keys which have a `origvalue`), you have to change the value of `origvalue` instead of the key value. Otherwise the `type` plugin will give an error.

# Mount a new TOML file
sudo kdb mount test.toml user:/tests/storage/numbers toml type
# Write an octal value
kdb set 'user:/tests/storage/numbers/a' '0o777'
# Get the converted decimal value
kdb get 'user:/tests/storage/numbers/a'
# > 511
# Get the original octal value
kdb meta-get 'user:/tests/storage/numbers/a' 'origvalue'
# > 0o777
# Get the type of the number; since it's originally octal, we get `unsigned_long_long`
kdb meta-get 'user:/tests/storage/numbers/a' 'type'
# > unsigned_long_long
# Change the value by changing the `origvalue` metakey
kdb meta-set 'user:/tests/storage/numbers/a' 'origvalue' '0o666'
# Get the new value as decimal
kdb get 'user:/tests/storage/numbers/a'
# > 438
# Change the value to an invalid octal value.
kdb meta-set 'user:/tests/storage/numbers/a' 'origvalue' '0o888'
# The key value is no longer considered a number
kdb meta-get 'user:/tests/storage/numbers/a' 'type'
# > string
# Cleanup
kdb rm -r user:/tests/storage/numbers
sudo kdb umount user:/tests/storage/numbers

# Strings

The plugin can read any kind of TOML string: basic, literal, basic multiline and literal multiline.

Similar to the TOML-specific values (table, array) discussed below, we use the `tomltype` metakey to store the kind of string. The values used are: `string_basic`, `string_literal`, `string_ml_basic` and `string_ml_literal`.

When writing a file, the value of `tomltype` will be respected. The plugin will therefore preserve the kind of string throughout a kdbGet/kdbSet cycle. You can also set `tomltype` manually to transform the string into a different type. If `tomltype` is set to a kind of string that is incompatible with the current value of the key, the plugin will report an error.

If `tomltype` is not set, the plugin defaults to a basic string, or if there are at least two `\n` to a basic multiline string. This way a `kdb set` will always work correctly, as long as the given string is valid UTF-8.

# Mount TOML file
sudo kdb mount test_strings.toml user:/tests/storage toml type
# setting a string containing a newline escape sequence
kdb set 'user:/tests/storage/string' 'I am a basic string\not a literal one.'
kdb get 'user:/tests/storage/string'
# > I am a basic string
# > ot a literal one
# setting the string again, but escape the backslash with another backslash
kdb set 'user:/tests/storage/string' 'I am a basic string\\not a literal one.'
kdb get 'user:/tests/storage/string'
# > I am a basic string\not a literal one
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

The plugin supports all kinds of escape sequences used by TOML in basic and basic multiline strings, like `\n`, `\r`, `\t` and even `\u`/`\U` for Unicode escape sequences. When reading a file, all these escape sequences will be translated into valid UTF-8 byte sequences that will be used as the key values.

When writing a file, the plugin uses the escape sequences, whenever a part of the key value cannot be placed literally into the file. If a key contains a non-UTF-8 string value, the plugin replaces any invalid sections with the Replacement Character U+FFFD and reports a warning.

Finally, the plugin also uses `origvalue` to remember the original formatting of the string. This is particularly useful for escape sequences. Without this mechanism, a `\u` escape sequence would always be written literally (not as an escape sequence), because when reading the file it is translated into UTF-8 bytes. For multiline strings `origvalue` also includes any leading or escaped newlines.

# Binary/NULL values

The plugin handles binary values by using the base64 plugin. As a result, binary values get written as base64 encoded strings, which start with the special prefix `@BASE64`. `NULL` key values are written as special strings of value `@NULL`.

# Mount TOML file
sudo kdb mount test_binary.toml user:/tests/storage toml type
# Creating a key with a NULL value
kdb set 'user:/tests/storage/nullkey'
# > Create a new key user:/test/nullkey with null value
# Print file content
cat `kdb file user:/tests/storage`
# > nullkey = '@NULL'
# Write base64 encoded data to the file
echo "base64 = '@BASE64SSBhbSBiYXNlIDY0IGVuY29kZWQgZm9yIG5vIHJlYXNvbi4='" > `kdb file user:/test`
# Print the value of the key, which is a binary value
kdb get 'user:/test/base64'
#> \x49\x20\x61\x6d\x20\x62\x61\x73\x65\x20\x36\x34\x20\x65\x6e\x63\x6f\x64\x65\x64\x20\x66\x6f\x72\x20\x6e\x6f\x20\x72\x65\x61\x73\x6f\x6e\x2e
# Print the value again, but apply the escape codes
echo -e `kdb get 'user:/test/base64'`
#> I am base 64 encoded for no reason.
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

# TOML specific structures

TOML specific structures are represented by the metakey `tomltype` on a certain key. It will be set when the TOML plugin reads a TOML structure from a file. Additionally, this metakey can be set by the user, if they want a certain TOML structure to be written. No automatic inference of this metakey is done on writing.

## Simple Tables

TOML's simple tables are represented by setting the `tomltype` metakey to `simpletable`.

# Mount TOML file
sudo kdb mount test_table.toml user:/tests/storage toml type
# Create three keys, which are all a subkey of 'common',
# but we have no 'common' simple table key yet
kdb set 'user:/tests/storage/common/a' '0'
kdb set 'user:/tests/storage/common/b' '1'
kdb set 'user:/tests/storage/common/c' '2'
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> common.a = 0
#> common.b = 1
#> common.c = 2
# Create a simple table key
kdb meta-set 'user:/tests/storage/common' 'tomltype' 'simpletable'
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> [common]
#> a = 0
#> b = 1
#> c = 2
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

## Table Arrays

Table arrays are represented by setting the `tomltype` metakey to `tablearray`. It is not required to also set the array metakey, since the plugin will set the metakey, if it is missing.

# Mount TOML file
sudo kdb mount test_table_array.toml user:/tests/storage toml type
# Create a table array containing two entries, each with a key 'a' and 'b'
kdb meta-set 'user:/tests/storage/tablearray' 'tomltype' 'tablearray'
kdb set 'user:/tests/storage/tablearray/#0/a' '1'
kdb set 'user:/tests/storage/tablearray/#0/b' '2'
kdb set 'user:/tests/storage/tablearray/#1/a' '3'
kdb set 'user:/tests/storage/tablearray/#1/b' '4'
# Print the highest index of the table array
kdb meta-get 'user:/tests/storage/tablearray' 'array'
#> #1
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> [[tablearray]]
#> a = 1
#> b = 2
#> [[tablearray]]
#> a = 3
#> b = 4
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

## Inline Tables

Inline tables are represented by setting the `tomltype` metakey to `inlinetable`. The plugin also supports reading/writing nested inline tables.

# Mount TOML file
sudo kdb mount test_inline_table.toml user:/tests/storage toml type
# Create a table array containing two entries, each with a key 'a' and 'b'
kdb meta-set 'user:/tests/storage/inlinetable' 'tomltype' 'inlinetable'
kdb set 'user:/tests/storage/inlinetable/a' '1'
kdb set 'user:/tests/storage/inlinetable/b' '2'
kdb meta-set 'user:/tests/storage/inlinetable/nested' 'tomltype' 'inlinetable'
kdb set 'user:/tests/storage/inlinetable/nested/x' '3'
kdb set 'user:/tests/storage/inlinetable/nested/y' '4'
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> inlinetable = { a = 1, b = 2, nested = { x = 3, y = 4 } }
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

## Arrays

Arrays are recognized by the `array` metakey. On writing, the plugin will detect arrays automatically and set the appropriate metakey if it is missing.

# Mount TOML file
sudo kdb mount test_array.toml user:/tests/storage toml type
# Create array elements
kdb set 'user:/tests/storage/array/#0' '1'
kdb set 'user:/tests/storage/array/#1' '2'
kdb set 'user:/tests/storage/array/#2' '3'
# Print the highest index of the array
kdb meta-get 'user:/tests/storage/array' 'array'
#> #2
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> array = [1, 2, 3]
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

# Comments and Empty Lines

The plugin preserves all comments with only one limitation for arrays. The whitespace in front of a comment is also saved.

Comments can also be created by assigning meta keys to a key. The meta keys must be of the form `comment/#n`, where `n` is a positive number, indicating the position of the comment relative to the key. An index of 0 is always the inline comment of the key. Indices greater than zero are for comments preceding the given key, where 1 is the top-most comment and the highest index comment is right above the key.

Preceding whitespace can be added to a comment by creating a `comment/#n/space` metakey. This metakey can contain any number of space or tab characters, which will be placed before the `#` starting the comment. Any whitespace after the `#` is considered part of the comment and will therefore be part of `comment/#n`.

File ending comments must be assigned to the file root key.

Empty lines in front of a key can be created by adding an empty `comment/#n/start` entry to it. In this case, no `comment/#n` key is needed.

# Mount TOML file
sudo kdb mount test_comments.toml user:/tests/storage toml type
# create a key-value pair, ready for comment decoration
kdb set 'user:/tests/storage/key' '1'
# add an inline comment with 4 leading spaces
kdb meta-set 'user:/tests/storage/key' 'comment/#0' ' This value is very interesting'
kdb meta-set 'user:/tests/storage/key' 'comment/#0/space' ' '
# add some comments preceding the key
kdb meta-set 'user:/tests/storage/key' 'comment/#1' ' I am the top-most comment relative to my key.'
kdb meta-set 'user:/tests/storage/key' 'comment/#2' ' I am in the middle. Just boring.'
kdb meta-set 'user:/tests/storage/key' 'comment/#3' ' I am in the line right above my key.'
# add file ending comments and empty lines
kdb meta-set 'user:/tests/storage' 'comment/#1' ' First file-ending comment'
kdb meta-set 'user:/tests/storage' 'comment/#2/start' ''
kdb meta-set 'user:/tests/storage' 'comment/#3' ' Second file-ending comment. I am the last line of the file.'
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> # I am the top-most comment relative to my key.
#> # I am in the middle. Just boring.
#> # I am in the line right above my key.
#> key = 1 # This value is very interesting
#> # First file-ending comment
#>
#> # Second file-ending comment. I am the last line of the file.
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

## Comments in Arrays

Any amount of comments can be placed between array elements or between the first element and the opening brackets.

However, only one comment - an inline comment - can be placed after the last element and the closing brackets. On reading, the plugin discards any non-inline comments between the last element and the closing brackets.

# Mount TOML file
sudo kdb mount test_array_comments.toml user:/tests/storage toml type
# Create array elements
kdb set 'user:/tests/storage/array/#0' '1'
kdb set 'user:/tests/storage/array/#1' '2'
kdb set 'user:/tests/storage/array/#2' '3'
# Add inline comment after the array
kdb meta-set 'user:/tests/storage/array' 'comment/#0' ' Inline comment after the array'
kdb meta-set 'user:/tests/storage/array' 'comment/#0/start' '#'
kdb meta-set 'user:/tests/storage/array' 'comment/#0/space' ' '
# Add comments for array elements
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#0' ' Inline comment of first element'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#0/start' '#'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#0/space' ' '
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#1' ' Comment preceding the first element'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#1/space' ' '
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#2' ' Another comment preceding the first element'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#2/space' ' '
kdb meta-set 'user:/tests/storage/array/#1' 'comment/#0' ' Inline comment of second element'
kdb meta-set 'user:/tests/storage/array/#1' 'comment/#0/space' ' '
kdb meta-set 'user:/tests/storage/array/#1' 'comment/#1' ' Comment preceding the second element'
kdb meta-set 'user:/tests/storage/array/#1' 'comment/#1/space' ' '
kdb meta-set 'user:/tests/storage/array/#2' 'comment/#0' ' Inline comment of the last element'
kdb meta-set 'user:/tests/storage/array/#2' 'comment/#0/space' ' '
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> array = [ # Comment preceding the first element
#> # Another comment preceding the first element
#> 1, # Inline comment of first element
#> # Comment preceding the second element
#> 2, # Inline comment of second element
#> 3 # Inline comment of the last element
#> ] # Inline comment after the array
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

# Order

The plugin preserves the file order by the usage of the metakey `order`. When reading a file, the order metakey will be set according to the order as read in the file. If new keys are added, eg. via `kdb set`, the order of the set key will be set to the next-to-highest order value present in the existing key set.

However, the order is only relevant between elements with the same TOML-parent. For example keys of a simple table are only sorted with respect to each other, not with any keys outside that table. If that table has it's order changed and moves to another position in the file, so will it's subkeys.

When sorting elements under the same TOML-parent, tables (simple and array) will always be sorted after non-table elements, regardless of their order. With this limitation, we prevent that a newly set key, that is not part of a certain table array/simple table, would be placed after the table declaration, making it a member of that table on a subsequent read.

# Mount TOML file
sudo kdb mount test_order.toml user:/tests/storage toml type
# Create three keys in reverse alphabetical order under the subkey common
# Additionally, create one key not in the common subkey space
kdb set 'user:/tests/storage/common/c' '0'
kdb set 'user:/tests/storage/common/b' '1'
kdb set 'user:/tests/storage/common/a' '2'
kdb set 'user:/tests/storage/d' '3'
# Print the content of the resulting TOML file
# The keys are ordered as they were set
cat `kdb file user:/tests/storage`
#> common.c = 0
#> common.b = 1
#> common.a = 2
#> d = 3
# Create a simple table for the three keys under `common`
kdb meta-set 'user:/tests/storage/common' 'tomltype' 'simpletable'
# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> d = 3
#> [common]
#> c = 0
#> b = 1
#> a = 2
# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage

In this example, `d` and `common` have the same parent, the file root. This means, they need to be sorted with each other. `d` would be placed before `common` by it's order, since it was set before, and thus, has lesser order. However, their order never gets compared, since `common` is a simple table and `d` is not, so `d` will get sorted before the table regardless of order.

# Limitations

While the plugin has good capabilities in handling the TOML file format, it currently lacks some features possible with Elektra:

- Sparse arrays are not preserved on writing, they get a continuous array without index holes.
- Values on non-leaf keys are currently not supported, they get discarded. This applies especially to the parent key of the mountpoint.
- Custom metakeys cannot be written by the plugin, so they get discarded.

Additionally, there are some minor limitations related to the TOML file format, mostly related to the preservation of the original file structure:

- On writing, each string is written as a TOML basic string (single or multiline). See Strings
- Comments and newlines between the last array element and closing brackets are discarded.
- Trailing commas in arrays and inline tables are discarded
- Only spaces in front of comments are preserved.