BigDFT build tree offers the possibility to have non-regression tests.
When the user (or the devloper) types ``make check`` in the
distribution, a number of tests are passed. Then a the
``fldiff_yaml.py`` script is checking that the logfiles of each run
coincides with the reference results that are provided for each test.
Ideally, each functionality of the code should have its own test, to
ensure that further developments do not break it. It is therefore
responsibility of the developer to provide new non-regression tests for
each of the functionality. The optimal test should be

-  Reasonably fast (roughly less than 1 min on a monocore run)
-  Reasonably exhaustive (if the test passes this means that the
   functionality is indeed preserved - no false positive)

Clearly, these two requirements contradict each other. The **art** of
the developer is to find a good compromise to reassure: 1) the other
developers not to slow down the ``make check`` execution, and 2)
himself, to be sure that new developments still would preserve the
functionality.

Having said that, imagine that we have a set of files that we would like
to use for non regression. There are different steps to include them in
the test farm of BigDFT.

As an example, we show the insertion of the test for the molecular
dynamics in BigDFT.

Prepare the directory
---------------------

The tests should go in the ``tests`` directory (rather unexpectedly...
;)). The procedure described here is valid for four subdirectories:
``cubic``, ``linear``, ``overDFT`` and ``tutorials``. All the other
directories have special Makefiles associated and have a less easy way
to automatize the test procedure. As our examples are related to MD, we
put our tests in the ``overDFT`` directory. We create the directory
``MD``:

| ``cd tests/overDFT``
| ``bzr mkdir MD``

Once this is done, all the test files inside this directory should
correspond to a particular naming scheme. For example, a test named
``foo`` have to be associated to two reference files: named
``MD_foo.out.ref``, containing the standard output (which essentially
only contains one line with the name of the logfile), and
``MD_foo.out.ref.yaml`` which contains the lofgile associated to the
test ``foo``. Clearly, also the input files ``foo.yaml`` and ``foo.xyz``
(or any other supported format have to be present, as if it was a normal
running directory. THe presence of the two reference files will cause
the check procedure to run

``$run_parallel ``\ \ ``/src/bigdft foo -l yes``

where ``run_parallel`` is the environment variable containing the
desired command to run tests. These tests will be ran in the directory
``tests/overDFT/MD-test/`` of the **build** tree, to preserve untouched
the input files and the references. The logfile of the run will be
written in the file ``log-foo.yaml`` of the same directory.

Let us imagine we would like to insert two tests, with codename
``LJ-NVE`` and ``LJ-NVT``. According to the above instruction, we
therefore have to add the following files:

| ``> ls MD/``
| ``> LJ-NVE.xyz  LJ-NVE.yaml LJ-NVT.xyz LJ-NVT.yaml  MD_LJ-NVE.out.ref  MD_LJ-NVE.out.ref.yaml  MD_LJ-NVT.out.ref  MD_LJ-NVT.out.ref.yaml  psppar.LJ``

The directory might contain also extra files, that are needed for the
run, in this case ``psppar.LJ``. We should not froget to also add these
files to the development branch:

| ``> bzr add tests/overDFT/MD``
| ``adding tests/overDFT/MD/LJ-NVE.xyz``
| ``adding tests/overDFT/MD/LJ-NVE.yaml``
| ``adding tests/overDFT/MD/LJ-NVT.xyz``
| ``adding tests/overDFT/MD/LJ-NVT.yaml``
| ``adding tests/overDFT/MD/MD_LJ-NVE.out.ref``
| ``adding tests/overDFT/MD/MD_LJ-NVE.out.ref.yaml``
| ``adding tests/overDFT/MD/MD_LJ-NVT.out.ref``
| ``adding tests/overDFT/MD/MD_LJ-NVT.out.ref.yaml``

Insert the test in the ``make check`` command
---------------------------------------------

Even if the directory is ready, the test is still not active. To
activate it in the ``make check`` procedure, the developer has to tell
the build system that ``MD`` is a valid test directory. To do that, it
is enough to insert the name of the directory (``MD`` in our example) in
the ``Makefile.am`` file of the ``tests/overDFT`` directory:

.. code:: make

    # Give here the list of existing tests
    SHORT_TESTDIRS = FF-LENOSKY \
        FF-LJ \
        H2O-slab \
        H2-freq \
        MD #<<<< add it here...

    [...]

    LONG_TESTDIRS = $(SHORT_TESTDIRS) \
        C6H6-freq \
        GEOPT-all \
        GEOPT-BFGS \
        GEOPT-LBFGS \
            [...] #<<< ...or here

As it can be seen, the ``MD`` directory can be added either in the
``SHORT_TESTDIRS`` or in the ``LONG_TESTDIRS`` variable, but not in the
two. Whether to choose one or another is a matter of the
importance/robustness of the test: the tests in ``SHORT_TESTDIRS`` are
ran by anybody typing ``make check`` on the released version whereas the
others are only run by developers and expert users. Therefore a test in
``SHORT_TESTDIRS`` shoudl be.. short, and also should fail only when
there are major instabilities in the code.

Running the tests and prepare tolerances
----------------------------------------

Even though an output file is prepared, we cannot expect the output of
the test to be bitwise identical to the provided output. It is therefore
responsibility of the 'test forger' to provide a set of tolerances for
any of the fields of the output file (which for this reason is in markup
language). This require some running of the tests to have an idea of the
correct tolerances. To run the test the first time, let us go to the
**build** tree (not in the source tree and execute a check for the
``MD`` test:

| ``> cd ``\ \ ``/tests/overDFT``
| ``> make MD.check``

This last command will, in the order:

-  Create the test directory ``MD-test`` in the build tree.
-  Copy the corresponding input files in such a directory.
-  Run the ``bigdft`` executable in the folder, with the command

``$run_parallel ``\ \ ``/src/bigdft ``\ \ `` -l yes``

where is each of the test runs (``NVE`` and ``NVT`` in the examples
defined above).

-  Once the test is finished, the ``fldiff`` scripts are ran to compare
   the logfiles of the output witrh the given references.

As all the log of the run are written in markup language, the different
fields of the logfile might be compared with different tolerances. Such
information is stored in the ``tests/tols-BigDFT.yaml`` file. Such file
might be modified to adapt the tolerance of each field to the nature of
the run.
