| Elektra 0.11.0

Profiling

# Execution Time

One of the primary resources in computing is execution time. To keep usage of this resource type low, it makes sense to profile code and check which code paths in a progamm take the longest time to execute. There exist various tools to handle this kind of profiling. For this tutorial we will use

1. Callgrind and the graphical frontend KCacheGrind/QCacheGrind, and
2. XRay and FlameGraph to visualize the data produced by XRay

## Callgrind

### Choosing the Correct Build Type

Since we want to improve the readability of the Callgrind output we choose a build type that includes debug symbols. The two obvious choices for the build type are:

- `RelWithDebInfo` (optimized build with debug symbols), and
- `Debug` (non-optimized build with debug symbols)

. We use `Debug` here, which should provide the most detailed profiling information.

### Disabling <tt>dlclose</tt> Calls

For this tutorial we decided to profile the YAJL plugin. Since Elektra loads plugin code via `dlopen` and Callgrind does not support the function `dlclose` properly we remove the `dlclose` calls in the file `dl.c` temporarily. At the time of writing one option to do that is deleting

- a single line `dlclose` statement, and
- an `if`-statement that checks the return value of a `dlclose` call

. An unfortunate effect of this code update is that Elektra will now leak memory when it unloads a plugin. On the other hand, Callgrind will be able to add source code information about the YAJL plugin to the profiling output.

### Building Elektra

As we already described before we use the `Debug` build type for the profiling run. To make sure we test the actual performance of the YAJL plugin we disable debug code and the logger. The following commands show one option to translate Elektra using this configuration, if we use Ninja as build tool:

mkdir build
cd build
cmake -GNinja .. \
-DCMAKE_BUILD_TYPE=Debug \
-DENABLE_LOGGER=OFF \
-DENABLE_DEBUG=OFF \
-DPLUGINS=ALL
ninja
cd .. # Change working directory back to the root of repository

### Profiling the Code

We use the tool `benchmark_plugingetset` to profile the execution time of YAJL. The file `keyframes_complex.json` serves as input file for the plugin. Since `benchmark_plugingetset` requires a data file called

test.$plugin.in

, we save a copy of `keyframes_complex.json` as `test.yajl.in` in the folder `benchmarks/data`:

mkdir -p benchmarks/data
cp src/plugins/yajl/yajl/keyframes_complex.json benchmarks/data/test.yajl.in

. After that we call `benchmark_plugingetset` directly to make sure that everything works as expected:

build/bin/benchmark_plugingetset benchmarks/data user yajl get

. If the command above fails with a segmentation fault, then please check

- that the build system included YAJL, and
- that your OS is able to locate the plugin (e.g. append the `lib` directory in the build folder to `LD_LIBRARY_PATH` on Linux)

. If `benchmark_plugingetset` executed successfully, then you can now use Callgrind to profile the command:

valgrind --tool=callgrind --callgrind-out-file=callgrind.out \
build/bin/benchmark_plugingetset benchmarks/data user yajl get

. The command above will create a file called `callgrind.out` in the root of the repository. You can now remove the input data and the folder `benchmarks/data`:

rm benchmarks/data/test.yajl.in
rmdir benchmarks/data

. If you use Docker to translate Elektra, then you might want to fix the paths in the file `callgrind.out` before you continue:

# The tool `sponge` is part of the `moreutils` package: https://joeyh.name/code/moreutils
sed -E 's~/home/jenkins/workspace/(\.\./)*~~g' callgrind.out | sponge callgrind.out

. Now we can analyze the file `callgrind.out` with a graphical tool such as QCacheGrind:

qcachegrind&

. If everything worked as expected QCacheGrind should open the file `callgrind.out` and display a window that look similar to the one below:

. You can now select different parts of the call graph on the left to check which parts of the code take a long time to execute.

## XRay

XRay is an extension for LLVM that adds profiling code to binaries. Profiling can be dynamically enabled and disabled via the environment variable `XRAY_OPTIONS`.

### Choosing the Correct Build Type

Since XRay currently requires LLVM we need to set the compiler appropriately. We use Clang 8 in our example.

export CC=clang-8
export CXX=clang++-8

. We enable the static build (`BUILD_STATIC=ON`) and disable the dynamic build (`BUILD_SHARED=OFF`), since XRay currently does not support dynamic libraries. To enable Xray we use the compiler switch `-fxray-instrument`. To instrument every function we set the instruction threshold to `1` with `-fxray-instruction-threshold=1`.

export REPOSITORY_DIRECTORY="$PWD"
export BUILD_DIRECTORY="$REPOSITORY_DIRECTORY/build"
mkdir -p "$BUILD_DIRECTORY"
cd "$BUILD_DIRECTORY"
cmake -GNinja "$REPOSITORY_DIRECTORY" \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED=OFF \
-DBUILD_STATIC=ON \
-DCOMMON_FLAGS='-fxray-instrument -fxray-instruction-threshold=1' \
-DPLUGINS=ALL

We will analyze the YAJL plugin below. Please make sure that the CMake command above includes the plugin:

…
-- Include plugin yajl
…

. Now we can translate the code with Ninja and change the current directory back to the root of the repository:

ninja
cd "$REPOSITORY_DIRECTORY"

. In the next step we use `benchmark_plugingetset` to execute YAJL for the input file `keyframes_complex.json`. To do that we

1. create the folder `data` in the directory `benchmarks`, and
2. save the file `keyframes_complex.json` as `test.yajl.in`

. The following commands show you how to do that:

mkdir -p benchmarks/data
cp src/plugins/yajl/yajl/keyframes_complex.json benchmarks/data/test.yajl.in

. Now we first check if running [`benchmark_plugingetset`][] works without instrumentation:

"$BUILD_DIRECTORY/bin/benchmark_plugingetset" benchmarks/data user yajl get

. If everything worked correctly, then the command above should finish successfully and not produce any output. To instrument the binary we set the environment variable `XRAY_OPTIONS` to the value `xray_mode=xray-basic verbosity=1`.

export XRAY_OPTIONS='xray_mode=xray-basic patch_premain=true verbosity=1'
"$BUILD_DIRECTORY/bin/benchmark_plugingetset" benchmarks/data user yajl get

. The command above will print the location of the XRay log file to `stdterr`:

…
…XRay: Log file in 'xray-log.benchmark_plugingetset.gpcX3t'
…

. Now we can use the log file to analyze the runtime of the execution paths of the binary. To do that we first save the name of the log file in the variable `LOGFILE`. This way we do not need to repeat the filename every time in the commands below.

LOGFILE=$("$BUILD_DIRECTORY/bin/benchmark_plugingetset" benchmarks/data user yajl get 2>&1 |
sed -nE "s/.*Log file in '(.*)'.*/\1/p")

. To list the 10 functions with the longest runtime we use the command `llvm-xray account`:

llvm-xray account "$LOGFILE" -top=10 -sort=sum -sortorder=dsc -instr_map "$BUILD_DIRECTORY/bin/benchmark_plugingetset"
#> Functions with latencies: 35
#> funcid count [ min, med, 90p, 99p, max] sum function
#> 1 1 [ 0.004791, 0.004791, 0.004791, 0.004791, 0.004791] 0.004791 <invalid>:0:0: main
#> 1333 1 [ 0.002007, 0.002007, 0.002007, 0.002007, 0.002007] 0.002007 <invalid>:0:0: elektraYajlGet
#> …

. We can also use the log file to create a Flame Graph. To do that we use the `llvm-xray stack` to create an input file for the tool `flamegraph.pl`

llvm-xray stack "$LOGFILE" -stack-format=flame -aggregation-type=time -all-stacks \
-instr_map "$BUILD_DIRECTORY/bin/benchmark_plugingetset" > flamegraph.txt

. We then create the Flame Graph with the following command:

# Depending on how you installed Flame Graph the executable
# might also be called `flamegraph.pl` instead of `flamegraph`.
flamegraph flamegraph.txt > flamegraph.svg

. The image below shows one example how the picture could look like:

. Additional information on how to use the data produced by XRay is available here.