| Elektra 0.11.0

Transformations

# Problem

Certain plugins, e.g. `rename`, perform transformations on keys and keysets within Elektra. Those transformations include, but are not limited to:

- Changing the names of the keys back and forth
- Changing values back and forth for normalization, e.g. `true` -> `1`, `1` -> `true`

While these features are useful, they do create feature-interaction problems. More specifically, problems have been observed in conjunction with the following (overlapping) types of plugins:

- notification plugins
- plugins that do change tracking

The problem, in general, can be described as: Which representation of the KDB should be used for notifications/change tracking?

We differentiate between:

- persistent name/value/metadata: How it is actually stored, i.e. the state returned by and passed to the `storage` plugins.
- transient name/value/metadata: How it is at runtime, i.e. what is returned by `kdbGet` and passed to `kdbSet`.
- intermediate name/value/metadata: Any state inbetween the two.

## Observed problems with changing key names

Suppose there is a plugin that changes key names. It converts to lowercase for the runtime representation, and to uppercase for the stored representation. The plugin is executed in the post-storage phase for the `get` operation and the pre-storage phase for the `set` operation. This results in the keys being in UPPERCASE in the configuration files, but they are presented in lowercase to other plugins and applications using the Elektra API.

For example, here is a configuration file with a hypothetical format:

/DISPLAY/BRIGHTNESS = 100

As can be seen, the keys are in UPPERCASE within the configuration file. In Elektra keys are case-sensitive. As operations on keysets such as `ksLookup` operate with runtime data after the post-storage phase, `kdb get /DISPLAY/BRIGHTNESS` will fail. For Elektra, the key `/DISPLAY/BRIGHTNESS` does not exist, as the `rename` plugin transformed this into the lowercase `/display/brightness`.

This leads to problems with the notification plugins. As notification plugins are executed after the post-storage phase of the `set` operation, they will receive a keyset with the already transformed keys. In this example, the notification plugins will receive all-UPPERCASE keys, and send out notifications with those all-UPPERCASE keys. An application listening to those notifications will not be able to query Elektra for those keys, as for Elektra those UPPERCASE keys do not exist.

Apart from the problems with notifications, the way key name changing plugins work also breaks change tracking in plugins like `dbus`. This is because key names are read-only when they are contained in a `KeySet`. In order to change the name of a key, such a plugin has to create a new key with the changed name, and delete the key with the old name. The `dbus` plugin implements change tracking by checking the 'key needs sync' flag instead of comparing the values. As new keys by design have the 'key needs sync' flag set, the plugins that implement change tracking via the flag will always erroneously detect transformed keys as changed.

## Observed problems with changing values (normalization)

Suppose we have a plugin that changes the value of a key to the hard coded value `1` during the storage phase of the `set` operation. If a plugin does change tracking, this will lead to false positives.

user:/limits/openfiles = 1

If the user changes the value, e.g. using `kdb set user:/limits/openfiles 23`, plugins will observe the new value `23`. The value-changing plugin, however, will reset that value back to `1`. So in practice, the configuration has not been changed. Plugins relying on change tracking plugins (e.g. notification plugins) will however think that it has.

# Constraints

1. We want to enable some kind of transformations
2. Renaming keys should be possible, at least in storage plugins 3.

# Assumptions

1. False positives for change tracking algorithms are only a minor problem. False positives are that changes are detected, even though nothing changed.
2. There is no reason to modify or delete existing `meta:/` keys.
3. Newly generated `meta:/...` keys can
   - either stay and get permanently stored during `kdbSet`
   - or be written as `meta:/generated/...`. The `meta:/generated/...` metakeys are never stored and are automatically removed during `kdbSet` before `storage` is called.

# Considered Alternatives

## Enable setting and unsetting of the <tt>keyNeedsSync</tt> flag

Require plugins that rename keys to remove the `keyNeedsSync` flag. This would require making the 'key sync' APIs public. Letting user code modify these internal flags may lead to serious bugs. This does not solve the problem of changing values.

## Use change tracking to replace the <tt>keyNeedsSync</tt> flag

Eliminate the need of the `keyNeedsSync` flag by utilizing the planned change tracking API. This does not solve the problem of changing values and false positives.

## Store original names and values in metadata.

For key name transformations require that transformation plugins set a metakey (e.g. `meta:/elektra/runtimename`) with the runtime name before they do any transformations in the `kdbSet` phase. This must not be done if this metakey already exists, i.e. another plugin already tranformed it beforehand. This allows notification and change tracking functionality to work determine and work with the runtime name of the key.

A similiar thing was already attempted for values, i.e. `meta:/origvalue`.

## Create an API for transformations

Plugins must use a special function to transform key names, e.g.:

typedef const char * (*ElektraNameTransform)(const Key *);
int elektraApplyNameTransform (KeySet * ks, ElektraNameTransform transform);

How the `elektraApplyNameTransform` function marks the original name is an internal implementation detail. May be as `meta:/` or something else entireley.

Something similar could be done for the value of a key as well.

## Introduce a new phase for transformations

We could also introduce a new phase between before/after storage exclusively for transformations. Then we can just do a "fake" call to that phase to get back the transient names for change tracking.

## Call value transformation plugin(s) when <tt>keySetValue</tt> / <tt>keySetString</tt> is called

After a value has been set by the user, call the transformation plugin. We could store those callbacks as metakeys, i.e. `meta:/generated/transformation/value/callback/#1`. Alternatively, we could implement a linked list or other data structure for the callbacks.

struct callback {
callback_fn function;
struct callback * next;
}

There will be 3 possibilities where transformations take place:

1. For keys with a registered callback:
   - Within `keySetValue` / `keySetString` the callbacks are called.
2. For new keys, i.e. keys without a callback:
   - Within `kdbSet`, as it is now.
3. For new keys that override keys with an intact callback:
   - Within `ksAppend` and/or `ksAppendKey` the callbacks are copied from the original key and then executed on the new key.

kdbGet (kdb, ks, parent);
// ks contains two keys:
// - user:/limits/openfiles
// - user:/background/color
Key * background = ksLookupByName(ks, "user:/background/color", 0);
keySetString(background, "#ff00ff"); // (1) -> callbacks are called.
ksAppendKey (ks, keyNew ("user:/my/key", KEY_VALUE, "1234", KEY_END); // (2) -> transformation will happen inside kdbSet
ksAppendKey (ks, keyNew ("user:/limits/openfiles", KEY_VALUE, "23", KEY_END); // (3) -> replaces existing key, callbacks of existing key are copied and then executed
kdbSet (kdb, ks, parent);
kdbSet
int kdbSet(KDB *handle, KeySet *ks, Key *parentKey)
Set Keys to the Key database in an atomic and universal way.
Definition kdb.c:2343
kdbGet
int kdbGet(KDB *handle, KeySet *ks, Key *parentKey)
Retrieve Keys from the Key database in an atomic and universal way.
Definition kdb.c:1734
keyNew
Key * keyNew(const char *name,...)
A practical way to fully create a Key object in one step.
Definition key.c:144
ksAppendKey
ssize_t ksAppendKey(KeySet *ks, Key *toAppend)
Appends a Key to the end of ks.
Definition keyset.c:968
ksLookupByName
Key * ksLookupByName(KeySet *ks, const char *name, elektraLookupFlags options)
Convenience method to look for a Key contained in ks with name name.
Definition keyset.c:2778
keySetString
ssize_t keySetString(Key *key, const char *newStringValue)
Set the value for key as newStringValue.
Definition keyvalue.c:381
kdb
This is the main namespace for the C++ binding and libraries.
Definition backend.hpp:31

We also need to provide a simple API to plugins to register such callbacks for a key.

As the transformations for existing keys will be applied before `kdbSet`, this will elimate false positives in changetracking.

A drawback to this solution is that it adds some complexity to `libelektra-core`.

# Decision

# Rationale

# Implications

# Related Decisions

- Hooks
- Change Tracking
- Operation Sequences
- Internal Cache

# Notes

- Issue #404 - dbus and rename plugin do not work together
- Issue #955 - dbus: non-UTF8 key names
- Issue #3626 - origvalue