| Elektra 0.11.0

0.9.12 Release

- guid: F2193578-1773-43A9-85CA-79EA8CE48D7B
- author: Mihael Pranjić
- pubDate: Fri, 03 Mar 2023 08:07:28 +0100
- shortDesc: New Backend Logic, Copy-on-Write, FLOSS Course

We are proud to release Elektra 0.9.12.

# What is Elektra?

Elektra serves as a universal and secure framework to access configuration settings in a global, hierarchical key database. For more information, visit https://libelektra.org.

You can also read the news on our website.

You can try out the latest Elektra release using our docker image elektra/elektra. This is the quickest way to get started with Elektra without compiling and other obstacles, simply run:

docker pull elektra/elektra
docker run -it elektra/elektra

# Highlights

- New Backend
- Copy-on-Write
- FLOSS

## New Backend

The entire logic for backends has been rewritten, to allow for more flexibility und an unlimited number of plugins. Instead of calling plugins directly, `libelektra-kdb` now only calls so-called backend plugins and special hook plugins. There is a contract between `libelektra-kdb` and the backend plugins. All backend plugins must adhere to this contract. To achieve this goal, most backend plugins will call other plugins (like `libelektra-kdb` did previously).

The logic previously implemented in `libelektra-kdb` was moved to the new default backend plugin `backend`. It works like the old system, but now also allows an unlimited number of plugins in positions where that makes sense. For example, you can have unlimited `postgetstorage` plugins, but only a single `getresolver`.

There have also been slight changes to `kdbGet` and `kdbSet`. Please read their API docs to find out, if you rely on any behavior that has been altered. You can also read the new low-level docs to find out all the intricate details.

The structure of `system:/elektra/mountpoints` changed as well. Take a look at the new docs, if you need to know details.

### Updating config

The mountpoint configuration format contains breaking changes and a manual upgrade process is needed. Follow these steps to upgrade the old mountpoint configuration to the new format:

> ‍Warning: BACK UP YOUR CONFIG FILES BEFORE UPDATING! We recommend making a backup of the file printed by `kdb file system:/elektra/mountpoints` before updating your installation. In the unlikely case that the migration script fails, you can still use the information from the backup to manually recreate your mountpoints.

To update your existing `system:/elektra/mountpoints` data you can use the migration script.

> ‍Note: To run the script you must have Elektra, Python (>= 3.7) and the Python binding installed. The script uses the Python binding to manipulate `Key`s and `KeySet`s, but it does not use the `kdb` CLI tool, or the `KDB` API. It is safe to run this script before, or after you update your Elektra installation.

By default, the script loads the file `/etc/kdb/elektra.ecf`. If you changed where `system:/elektra/mountpoints` is stored, you can provide an alternative path:

./migrate-mountpoints.py /path/to/your/mountpoints/config.file

> ‍Note: Because the script does not use the `KDB` API it only works, if the mountpoints config file uses the default `dump` format.

> If your mountpoints config file is not using the `dump` format, you may still be able to use the migration script. However, in that case, you will have to use the script before updating your Elektra installation:

> 1. Run `kdb export system:/elektra/mountpoints dump` to get a copy of your mountpoints config in `dump` format
> 2. Write this data to a file and run the migration script on the file.
> 3. To get the data back in your original format you can use
> ./migrate-mountpoints.py /path/to/file/from/step2 | kdb convert dump your-format > /path/to/converted/file
> 1. Run `kdb file system:/elektra/mountpoints` to find out where your mountpoint config is stored. Make sure to back up this file, before upgrading your installation.
> 2. Now upgrade your Elektra installation.
> 3. Copy the file `/path/to/converted/file` from step 3 to the location you got in step 4.

The script will read the old mountpoint configuration from the given file. It will convert the configuration and print the new version to `stdout`.

You can inspect the output to make sure, everything is in order. When you are ready to commit the changes, you can manually edit the config file, or use:

./migrate-mountpoints.py --output=/etc/kdb/elektra.ecf /etc/kdb/elektra.ecf

### Individual changes

- Implement hooks. _(Maximilian Irlinger @atmaxinger)_
- Removed old global plugins code. _(Maximilian Irlinger @atmaxinger)_
- New backend logic, based on PR #2969 by @vLesk. _(@kodebach)_
- Add script to migrate `system:/elektra/mountpoints` to new format. _(@kodebach)_

## Copy-on-Write

Thanks to _(Maximilian Irlinger @atmaxinger)_ our `Key` and `KeySet` datastructures are now fully copy-on-write! This means noticeably reduced memory usage for cases where keys and keysets are copied and/or duplicated!

We ran some very promising benchmarks, each were performed with 400,000 keys. All benchmarks were executed using `valgrind --tool=massif --time-unit=B --max-snapshots=200 --threshold=0.1`.

Benchmark | Old Implementation | Copy-on-Write | Size Reduction | Remarks
--- | --- | --- | --- | ---
`createkeys.c` | 5.3 MiB | 6.5 MiB | -22 % |
`deepdup.c` | 10.5 MiB | 8.2 MiB | 22 % |
`large.c` | 18.9 MiB | 15.3 MiB | 19 % |
`kdb.c` | 23.5 MiB | 17.8 MiB | 24 % |
`kdbget.c` | 11.0 MiB | 8.8 MiB | 20 % |
`kdbmodify.c` | 11.0 MiB | 8.8 MiB | 20 % | Same results as `kdbget.c`

First, it should be noted that a single key, without counting payload, is about 50% larger with the copy-on-write implementation. This explains why the `createkeys.c` benchmark yields a negative reduction result. This benchmark only allocates keys, so not much improvement can be expected there. Still, as other stuff also uses heap memory, the overall memory consumption only increased by 22%, which is far less than 50%.

All other benchmarks saw meaningful reductions of heap memory used. One interesting observation is that `kdbget.c` and `kdbmodify.c` used exactly the same memory. This can most likely be explained by internal caching within the memory allocator of `glibc`.

We also performed runtime tests on the same benchmarks using `perf stat --repeat 13` to ensure no major performance regressions occur.

Benchmark | Old Implementation | Deviation | Copy-on-Write | Deviation | Runtime Increase
--- | --- | --- | --- | --- | ---
`createkeys.c` | 0.209572 s | 0.36 % | 0.21987 s | 0.77 % | 4.9 %
`deepdup.c` | 0.23025 s | 0.47 % | 0.231804 s | 0.32 % | 0.6 %
`large.c` | 1.14038 s | 0.21 % | 1.14837 s | 0.21 % | 0.7 %
`kdb.c` | 1.9270 s | 2.63 % | 1.93354 s | 0.17 % | 0.3 %
`kdbget.c` | 0.145663 s | 0.17 % | 0.15763 s | 0.70 % | 8.2 %
`kdbmodify.c` | 0.146506 s | 0.19 % | 0.156347 s | 0.15 % | 6.7 %

Overall, the runtime performance hit is less than 10%. The more a program does, the less the additional overhead of the copy-on-write algorithms matter. One interesting detail is that `keyCopy` and `keyDup` have become quite a bit faster. This can be seen by comparing the differences between `createkeys.c` and `deepdup.c`. The differences are 21 ms for the old implementation and 12 ms for the copy-on-write implementation.

## FLOSS

A Free/Libre and Open Source Software (FLOSS) Initiative couldn't survive with many small contributions fixing annoying problems. This release also contains all contributions done via one term of the FLOSS course. The success was tremendous, as shown in the rest of the release notes.

A big thanks to the students for their contributions!

# Plugins

The following text lists news about the plugins we updated in this release.

## yajl

- Fix an issue where trying to set invalid meta-keys won't show an error. _(Juri Schreib @Bujuhu)_

## list

- Removed the outdated `list` plugin. _(Maximilian Irlinger @atmaxinger)_ (Was only needed for global plugins, which are now replaced by hooks.)

## logchange

- Made logchange a notification-send hook plugin. _(Maximilian Irlinger @atmaxinger)_

## toml

- Fix bug, where meta-keys that cannot be inserted don't report an error. _(@Bujuhu)_

## uname

- Add error handling if uname call fails. _(Richard Stöckl @Eiskasten)_

## quickdump

- elektraQuickdumpSet: don't fclose if stdout. _(@hannes99)_

## blockresolver

- Add encoding test for blockresolver read. _(@dtdirect)_
- Refactor and restructure blockresolver. _(@dtdirect)_

## desktop

- Add a unit test. _(Richard Stöckl @Eiskasten)_

## mini

- Fix a bug where writing meta-keys will fail silently. _(Juri Schreib @Bujuhu)_

## mmapstorage

- Remove code duplication in the data block calculation. _(Richard Stöckl @Eiskasten)_

## network

- Add a retry mechanism. _(Richard Stöckl @Eiskasten)_

## xfconf

- Add xfconf storage plugin with the ability to read and write to xfconf channels. _(Richard Stöckl @Eiskasten)_
- Make xfconf valgrind suppressions more flexible to lib updates. _(Mihael Pranjić @mpranj)_

## date

- Fix an issue with validationg RFC 822 date-times. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Improve Code Coverage. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_

## csvstorage

- Fix a bug where writing unkown meta-keys will fail silently. _(Juri Schreib @Bujuhu)_

# Libraries

The text below summarizes updates to the C (and C++)-based libraries of Elektra.

## Compatibility

- Global plugins do not work anymore, use hooks instead.

## Core

- The Key and KeySet datastructures are now fully copy-on-write. _(Maximilian Irlinger @atmaxinger)_
- `keyCopy` now only allocates additional memory if `KEY_CP_META` or `KEY_CP_ALL` is used. _(Maximilian Irlinger @atmaxinger)_
- Check for circular links (overrides). _(@0x6178656c)_

## io

- Check file flags for `elektraIoFdSetFlags`: file flags must be exactly one of: read only, write only or read write. _(Richard Stöckl @Eiskasten)_

## Merge

- Add methods `elektraMergeGetConflictingKeys` and `elektraMergeIsKeyConflicting` to check which keys were causing a merge conflict. _(Maximilian Irlinger @atmaxinger)_

# Bindings

Bindings allow you to utilize Elektra using various programming languages. This section keeps you up-to-date with the multi-language support provided by Elektra.

## intercept/env

- Remove fallback code. _(Markus Raab)_
- Command-line functionality is broken due to new-backend differences.

## intercept/fs

- Use `KDB_MAX_PATH_LENGTH` for better portability. _(Markus Raab)_

## jna

- Documentation: Improved build instructions. _(@Bujuhu)_
- Add validation on get for whitelist plugin. _(@Bujuhu)_
- Upgrade Gradle to 8.0.1. _(Mihael Pranjić @mpranj)_

## rust

- Fix "feature `resolver` is required" error. _(Markus Raab)_

## elixir

- Initial release of the Elixir binding. _(@0x6178656c)_
- Mark tests as `memleak`. _(@0x6178656c)_

# Tools

## kdb

- Removed `global-mount` and `global-umount` commands. _(Maximilian Irlinger @atmaxinger)_
- Fixed SIGSEGV when using `kdb find` without argument. _(Christian Jonak-Moechel @joni1993)_

## elektrad

- Removed leftover package-lock.json file. _(stefnotch)_

# Scripts

- Added automatic spelling corrections for changeset. _(Maximilian Irlinger @atmaxinger)_
- Introduce shebang-conventions. _(@0x6178656c)_
- Apply auto-fixes from shellcheck. _(@0x6178656c)_
- Use prettier 2.8.4. _(Mihael Pranjić @mpranj)_
- Add scripts to enable and disable pre-commit hooks. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Only let http links pass the check if whitelisted. _(Richard Stöckl @Eiskasten)_
- Link Checker: Add documentation for the usage and how it behaves. _(Richard Stöckl @Eiskasten)_
- Sed: Add spelling correction for "key-value storage". _(@Bujuhu)_
- Fix/extends some shell recorder tests. _(@Joni1993)_
- Use clang-format v15. _(Mihael Pranjić @mpranj)_
- Fix warning parsing in shell recorder. _(@Joni1993)_
- Replaced `egrep` by `grep -E`. _(@0x6178656c and @janldeboer)_
- Add audit-dependencies script to check for vulnerabilities for npm depndencies. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_

# Documentation

- Restructured contrib/api. _(Markus Raab)_.
- Improve page on compilation. _(@0x6178656c)_
- Improve page for bindings. _(@0x6178656c)_
- Improve page for getting started. _(@stefnotch)_
- Remove version number from docker README and replace it with latest. _(@Joni1993)_
- Fix grammar for `elektra-granularity.md`. _(@dtdirect)_
- Rephrase sections in `doc/dev/error-\*`. _(@dtdirect)_
- Improve Git.md. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Unify spelling of man pages. _(@stefnotch)_ _(@janldeboer)_
- Extend consistency check `check_doc.sh` to work for contrib, dev and tutorials. _(@Joni1993)_
- Fix internal links. _(@0x6178656c)_
- Update AUR Link from `elektra` to `libelektra` package. _(@Bujuhu)_
- Update example Ansible playbook in VISION.md. _(@Bujuhu)_
- Harmonize spelling of Git. _(@Joni1993)_
- Update packaging instructions for Fedora. _(@0x6178656c)_
- Improve use of gender. _(@0x6178656c)_
- Fix some minor mistakes in CONTRIBUTING.md. _(@Joni1993)_
- Fix various spelling errors. _(@Joni1993)_
- Denoted package names & global variable names in INSTALL.md as `Code`. _(@janldeboer)_
- Improve readability of doc/tutorials/highlevel.md. _(@deoknats861)_
- Improve reference to Podman documentation. _(@0x6178656c)_
- Unify spelling. _(@Joni1993)_
- Fix typo in dev/hooks.md. _(@dtdirect)_
- Remove unused images from doc/images. _(@dtdirect)_
- Fixed Coverage Badge Link. _(@janldeboer)_
- Improve CONTRIBUTING doc. _(Juri Schreib @Bujuhu)_ and _(Nikola Prvulovic @Dynamichost96)_
- Update Doxyfile with Doxygen 1.9.4. _(@0x6178656c)_
- Add project logo to Doxygen in Doxyfile. _(@dtdirect)_
- Add mermaid.js to the project using doxygen-mermaid. _(@dtdirect)_
- Create diagrams in mermaid.js to use in doxygen. _(@dtdirect)_
- Create README for Doxygen and Mermaid JS. _(@dtdirect)_
- Tutorial: Add automatic validation to Docker tutorial _(Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Add mention of audit-dependencies script in doc/todo/RELEASE.md. _(@Bujuhu)_
- Move note in GETSTARTED.md. _(@Joni1993)_
- Use `code` blocks to prevent Markdown from falsy rendering LaTeX. _(@stefnotch)_, _(@janldeboer)_
- Fix broken links in use cases for KDB after files were renamed. _(Florian Lindner @flo91)_
- Replace http links with https. _(Richard Stöckl @Eiskasten)_
- Enhance notifications.md in doc/tutorial. _(@dtdirect)_
- Add tutorial how to suppress memleaks in plugins from dependencies. _(Richard Stöckl @Eiskasten)_
- Write about history to make plans of Elektra's adoption more clear. _(Markus Raab)_

## Use Cases

- Improve use cases Template. _(@kodebach and Markus Raab)_
- Use cases for KDB. _(@kodebach)_
- Use cases for `libelektra-core`. _(@kodebach)_

## Decisions

- Decide and implement decision process. _(Markus Raab)_
- Decided future library split. _(@kodebach)_
- Decided decision process. _(Markus Raab)_
- Draft for man pages. _(Markus Raab)_
- Add decision for change tracking. _(Maximilian Irlinger @atmaxinger)_
- Create decision for allowed and prohibited operation seqences. _(Maximilian Irlinger @atmaxinger)_
- Add decisions about location of headers and use of `#include` in the repo. _(@kodebach)_
- Add decision about metadata semantics. _(@kodebach)_
- Many small fixes to adapt to documentation guidelines and new decision process. _(Markus Raab)_
- Add decision for read-only keynames. _(Maximilian Irlinger @atmaxinger)_
- Revive keyname decision. _(@kodebach)_
- Add decisions for constructor functions and builder functions. _(@kodebach)_
- Add decision for copy-on-write and provide implementation suggestions. _(Maximilian Irlinger @atmaxinger)_
- Update internal cache. _(Markus Raab)_ _(@kodebach)_
- Create transformations. _(Maximilian Irlinger @atmaxinger)_
- Replace TOC-style README.md with folders and generate HTML for website. _(@kodebach)_
- Restructured decisions directories based on new agreed-upon steps. _(Maximilian Irlinger @atmaxinger)_
- Decision for types of `KeySet`s. _(@kodebach)_
- Added Documentation Guidelines. _(Markus Raab)_
- Add links and formatting to documents affected by PR#4492 (Document Guidelines) and rephrase some parts. _(Florian Lindner @flo91)_
- Decisions for changes to `keyIsBelow` and new `keyGetNextPart` functions. _(@kodebach)_
- Apply fix spelling to more files. _(Markus Raab)_

## Tutorials

- Add tutorial for manual installation from the AUR on Arch Linux. _(@Bujuhu)_
- Add Markdown shell recorder validation to install.webui.md. _(@deoknats861)_
- Fix the outdated array tutorial. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Reinstate mounting tutorial. _(@Bujuhu)_
- Make namespaces tutorial verifiable. _(@0x6178656c)_
- Move Podman-related information to a dedicated page. _(@0x6178656c)_

## Man Pages

- Update man page (patch) as suggested by the CI to fix CI error on master. _(Florian Lindner @flo91)_
- Added links to the website & webui after further reading. _(Philipp Nirnberger @nirnberger)_
- Upgrade `ronn-ng` to 0.10.1.pre3. _(Mihael Pranjić @mpranj)_

# Tests

- Fix an Issue where scripts/dev/fix-spelling does not work, if a resolved path contains whitespaces. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Rename scripts/sed to scripts/spelling.sed. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Add `memleak` label to `test_getenv`. _(@0x6178656c)_
- Add test using shellcheck. _(@0x6178656c)_
- Remove `--rerun-failed` from `run_*` scripts. _(@kodebach)_
- Fix paths for icheck test. _(Mihael Pranjić @mpranj)_

## Shell Recorder

- Add check if file exists. _(@0x6178656c)_

# Packaging

- Add missing new `backend` plugin to components of libelektra package. _(Mihael Pranjić @mpranj)_

# Build

## CMake

- Fix warning for CMP0115. _(0x6178656c)_
- Change Doxygen configuration for LaTeX. _(0x6178656c)_
- Fix developer warning for package DISCOUNT. _(Dennis Toth @dtdirect)_
- Pass `--stacktrace` to gradle for the JNA builds. _(Maximilian Irlinger @atmaxinger)_
- Adapt npm build flags to remove reproducability issues. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
- Fix creation of shell recorder tests. _(@0x6178656c)_

## Docker

- Update packagename `libpcrec++-dev` to `libpcrecpp0v5` in Debian Sid. _(Richard Stöckl @Eiskasten)_
- Add shellcheck to Debian containers. _(@0x6178656c)_
- Use `openjdk-17-jdk` in Debian Sid. _(Maximilian Irlinger @atmaxinger)_
- Add Fedora 37 images. _(Mihael Pranjić @mpranj)_
- Update Debian Sid image to use repository Python modules instead of installing with `pip3` due to upstream debian changes. _(Mihael Pranjić @mpranj)_
- Debian Bullsye: use clang 13. _(Mihael Pranjić @mpranj)_
- Update Alpine Linux to 3.17.2. _(Mihael Pranjić @mpranj)_

# Gradle

- Use Gradle 7.5.1. _(Mihael Pranjić @mpranj)_
- Update `java-library.gradle` to use `archiveClassifier` _(Maximilian Irlinger @atmaxinger)_

# Infrastructure

## Jenkins

- Add Fedora 37 builds, drop Fedora 35 builds. _(Mihael Pranjić @mpranj)_
- Run more tests also on Master. _(Markus Raab)_
- Move doc to main build stage. _(Markus Raab)_
- Disable parallel test runs. _(Maximilian Irlinger @atmaxinger)_
- Upgrade Jenkins node container to Debian bullseye. _(@0x6178656c)_
- Undo previous change that added automatic `ctest --rerun-failed` to Jenkins CI. _(@kodebach)_

## Cirrus

- Use Fedora 37. _(Mihael Pranjić @mpranj)_
- Fix `macos_instance` reference, upgrade to macOS Ventura (by default), use Python 3.11 and Ruby 3.x. _(Mihael Pranjić @mpranj)_
- Automatically rerun `testmod_dbus*` tests on macOS. _(@kodebach)_
- Fix dbus not starting on macOS. _(Maximilian Irlinger @atmaxinger)_

## GitHub Actions

- Add auto-cancellation-running action. _(Tomislav Makar @tmakar)_
- Automatically rerun `testmod_dbus*` tests on macOS. _(@kodebach)_
- Fix dbus not starting on macOS. _(Maximilian Irlinger @atmaxinger)_
- Change stale issue/PR checking to GitHub action. _(@0x6178656c)_
- Update configuration of stale issue/PR action. _(@0x6178656c)_
- Upgrade actions to recent versions and remove deprecated ruby-setup action. _(Mihael Pranjić @mpranj)_

# Website

The website is generated from the repository, so all information about plugins, bindings and tools are always up-to-date. Furthermore, we changed:

- Fix broken `/pythongen` link on homepage. _(@stefnotch)_
- Fix redirect logic to not cause loops. _(@stefnotch)_
- Remove duplicated link to `TESTING.md` file. _(@stefnotch)_, _(@janldeboer)_
- Restructure parts of the links on the website. _(@stefnotch)_, _(@janldeboer)_
- Removed broken links to packages for Linux distributions. _(@Dynamichost96)_
- Update npm packages. _(Mihael Pranjić @mpranj)_
- Change URLs to say man-page with a dash. _(@stefnotch)_ _(@janldeboer)_

# Outlook

We are currently working on following topics:

- 1.0 API _(Klemens Böswirth @kodebach)_ and _(Stefan Hanreich)_
- Session recording and better Ansible integration _(Maximilian Irlinger @atmaxinger)_
- Change tracking _(Maximilian Irlinger @atmaxinger)_
- Rewriting tools in C _(@hannes99)_
- Elektrify KDE and GNOME _(Mihael Pranjić @mpranj)_
- Elektrify XFCE _(Richard Stöckl @Eiskasten)_
- Mounting SQL databases _(Florian Lindner @flo91)_
- Recording Configuration _(Maximilian Irlinger)_
- Ansible-Elektra _(Lukas Hartl)_ and _(Maximilian Irlinger)_
- Configure Olimex Base Images _(Maximilian Irlinger)_
- Improving Build Server Infrastructure _(Lukas Hartl)_ and _(Maximilian Irlinger)_
- Improve Java Development Experience _(Michael Tucek)_

# Statistics

We closed about 150 issues for this release.

About 28 authors changed 960 files with 29400 insertions(+) and 20927 deletions(-) in 1421 commits.

Thanks to all authors for making this release possible!

# Join the Initiative!

We welcome new contributors! Read here about how to get started.

As first step, you could give us feedback about these release notes. Contact us via our issue tracker.

# Get the Release!

You can download the release from

- here or
- GitHub

The hashsums are:

- name: elektra-0.9.12.tar.gz
- size: 9297913
- md5sum: a6de9401709283b69ec211681f2a7757
- sha1: cb4e282d1346fda771de7510663652555f8e6c7d
- sha256: 38238ba4a5318f999dc3045da06467abf529344dc46ad3fdf42bdca0155e149c

The release tarball is also available signed using GnuPG from

- here or
- GitHub

The following GPG Key was used to sign this release: 12CC44541E1B8AD9B66AFAD55262E7353324914A

Already built API documentation can be found

- here or
- GitHub.

# Stay tuned!

Subscribe to the RSS feed to always get the release notifications.

If you also want to participate, or for any questions and comments, please contact us via our issue tracker on GitHub.

Permalink to this NEWS entry

For more information, see https://libelektra.org.

Best regards, Elektra Initiative