.. _readme-launcher:

================
Chapel Launchers
================

When compiling Chapel programs for multiple locales, a launcher binary
is typically created that will execute the appropriate command(s) to
get your program started. For example, when compiling for multiple
locales, typically two binaries will be generated by the compiler
(e.g., myprogram and myprogram_real). The first binary contains code to get
your program up and running on multiple locales while the second
contains your actual program code.

The goals of the launcher binary are: 

#. to wrap details of job startup in a portable way so that new users
   can quickly get Chapel programs up and running on an unfamiliar
   platform.

#. to perform command-line parsing and error checking prior to
   waiting in a queue or firing off a parallel job in order to save
   time and resources related to simple errors/typos in the command
   line.

#. to preserve Chapel's global-view programming model by permitting
   the user to run their program using a single binary (corresponding
   to the single logical task that executes main()) without getting
   bogged down in questions of numbers of nodes, numbers of cores per
   node, numbers of program instances to start up, etc.

#. if necessary, to coordinate runtime functional activity, such as
   I/O.

Executing a Chapel program using the verbose (``-v``) flag will typically
print out the command(s) used to launch the program, along with any
environment variables the launcher set on its behalf.  It will also
cause the program itself to print additional information about how it
configured itself, though most of this will be of more interest to
Chapel developers than regular users.

Executing using the help (``-h``/``--help``) flag will typically print out
any launcher-specific options in addition to the normal help message for
the program itself.

You can also execute the Chapel launcher with the ``--dry-run`` flag.
This will not actually run or launch the user program, but instead simply
print the same thing as ``-v``: the command(s) that would have been used
to launch the program, along with any environment variables the launcher
would have set on its behalf.  Note that ``--dry-run`` will also cause
batch and other files created for the system launcher to be left behind
so you can inspect and/or reuse them.  Normally these are removed by the
Chapel launcher when the program finishes.  An example of such a file
would be the ``sbatch`` file created when a Slurm-based launcher is used
and ``CHPL_LAUNCHER_USE_SBATCH`` is set.

Currently Supported Launchers
+++++++++++++++++++++++++++++

Currently supported launchers include:

=======================  ==============================================================
Launcher Name            Description
=======================  ==============================================================
amudprun                 GASNet launcher for the UDP substrate
aprun                    Cray application launcher using aprun
gasnetrun_ibv            GASNet launcher for the Infiniband substrate
gasnetrun_mpi            GASNet launcher for the MPI substrate
mpirun4ofi               provisional launcher for ``CHPL_COMM=ofi`` on non-Cray systems
lsf-gasnetrun_ibv        GASNet launcher for LSF (bsub) and the Infiniband substrate
pals                     Cray application launcher for PALS on HPE Cray EX systems
pbs-aprun                Cray application launcher for PBS (qsub) + aprun
pbs-gasnetrun_ibv        GASNet launcher for PBS (qsub) and the Infiniband substrate
slurm |-| gasnetrun_ibv  GASNet launcher for SLURM and the Infiniband substrate
slurm |-| gasnetrun_mpi  GASNet launcher for SLURM and the MPI substrate
slurm |-| gasnetrun_ofi  GASNet launcher for SLURM and the OFI substrate
slurm |-| gasnetrun_ucx  GASNet launcher for SLURM and the UCX substrate
slurm-srun               native SLURM launcher
smp                      GASNet launcher for the shared-memory substrate
none                     do not use a launcher
=======================  ==============================================================

A specific launcher can be explicitly requested by setting the
``CHPL_LAUNCHER`` environment variable. For the specific case of the
``mpirun4ofi`` launcher, please see :ref:`readme-libfabric`.

If ``CHPL_LAUNCHER`` is left unset, a default is picked as follows:


* if ``CHPL_COMM`` is gasnet and ``CHPL_COMM_SUBSTRATE`` is udp
  ``CHPL_LAUNCHER`` is set to amudprun

* otherwise, if ``CHPL_COMM`` is gasnet and ``CHPL_COMM_SUBSTRATE`` is smp
  ``CHPL_LAUNCHER`` is set to smp

* otherwise, if ``CHPL_COMM`` is gasnet:

  =======================  ==============================================
  If                       CHPL_LAUNCHER
  =======================  ==============================================
  CHPL_COMM_SUBSTRATE=ibv  [slurm-]gasnetrun_ibv
  CHPL_COMM_SUBSTRATE=mpi  [slurm-]gasnetrun_mpi
  CHPL_COMM_SUBSTRATE=ucx  [slurm-]gasnetrun_ucx
  CHPL_COMM_SUBSTRATE=ofi  [slurm-]gasnetrun_ofi
  =======================  ==============================================

  If salloc is in the user's path, then the slurm version of these launchers
  will be used. Otherwise, the base ``gasnetrun_*`` launcher is used.

* otherwise, if ``CHPL_TARGET_PLATFORM`` is a HPE system or a Cray system:

  ==================================  ===================================
  If                                  CHPL_LAUNCHER
  ==================================  ===================================
  both aprun and srun in user's path  none
  aprun in user's path                aprun
  srun in user's path                 slurm-srun
  otherwise                           none
  ==================================  ===================================

* otherwise, if ``CHPL_COMM`` is not none and srun is in the user's path, then 
  ``CHPL_LAUNCHER`` is set to slurm-srun

* otherwise ``CHPL_LAUNCHER`` is set to none

If the launcher binary does not work for your system (due to an
installation-specific configuration, e.g.), you can often use the
``--dry-run`` flag to capture the commands that the launcher would have
executed on your behalf and customize them for your needs.

Forwarding Environment Variables
++++++++++++++++++++++++++++++++

Chapel launchers generally arrange for environment variables to be
forwarded to worker processes. However, this strategy is not always
reliable. The remote system may override some environment variables, and
some launchers might not correctly forward all environment variables.

.. _chpl-rt-masterip:

CHPL_RT_MASTERIP
****************

This environment variable is used to specify the IP address which should be used
to connect.  By default, the node creating the connection will pass the result
of ``gethostname()`` on to the nodes that need to connect to it, which will
resolve that to an IP address using ``gethostbyname()``.

When ``CHPL_COMM == gasnet``, this will also be used to set the value of
``GASNET_MASTERIP``, which corresponds to the hostname of the master node (see
https://gasnet.lbl.gov/dist/udp-conduit/README ).

.. _chpl-rt-workerip:

CHPL_RT_WORKERIP
****************

This environment variable is used to specify the IP address which should be used
to communicate between worker nodes.  By default, worker nodes will communicate
among themselves using the same interface used to connect to the master node
(see :ref:`chpl-rt-masterip`, above).

When ``CHPL_COMM == gasnet``, this will also be used to set the value of
``GASNET_WORKERIP`` (see https://gasnet.lbl.gov/dist/udp-conduit/README ).

.. _using-slurm:

Using Slurm
+++++++++++

To use native Slurm, set:

.. code-block:: sh

  export CHPL_LAUNCHER=slurm-srun

On Cray systems, this will happen automatically if srun is found in your
path, but not when both srun and aprun are found in your path. Native
Slurm is the best option where it works, but at the time of this writing,
there are problems with it when combined with ``CHPL_COMM=gasnet`` and the
UDP or InfiniBand conduits. So, for these configurations please see:

  * :ref:`readme-infiniband` for information about using Slurm with
    InfiniBand.
  * :ref:`using-udp-slurm` for information about using Slurm with the UDP
    conduit

Common Slurm Settings
*********************

* Optionally, you can specify a node access mode by setting the environment
  variable ``CHPL_LAUNCHER_NODE_ACCESS``. It will default to ``exclusive``
  access, but can be overridden to:

    * ``shared`` to give shared access to nodes
    * ``unset`` to use the system default and not specify a node access mode
    * ``exclusive`` to give exclusive access to nodes (this is the default)

  For example, to grant shared node access, set:

  .. code-block:: bash

    export CHPL_LAUNCHER_NODE_ACCESS=shared

* Optionally, you can specify a slurm partition using either the environment
  variable ``CHPL_LAUNCHER_PARTITION`` or the ``--partition`` flag. For
  example, to use the 'debug' partition for all runs, set:

  .. code-block:: bash

    export CHPL_LAUNCHER_PARTITION=debug

  Or, to use the 'debug' partition for a single run, use:

  .. code-block:: bash

    ./myprogram --partition=debug

* Optionally, you can specify a slurm nodelist using either the environment
  variable ``CHPL_LAUNCHER_NODELIST`` or the ``--nodelist`` flag. For example, to use node nid00001 for all runs, set:

  .. code-block:: bash

    export CHPL_LAUNCHER_NODELIST=nid00001

  Or, to use node nid00001 for a single run, use:

  .. code-block:: bash

    ./myprogram --nodelist=nid00001

* Optionally, you can specify a slurm constraint by setting the environment
  variable ``CHPL_LAUNCHER_CONSTRAINT``. For example, to use nodes with the
  'cal' feature (as defined in the slurm.conf file), set:

  .. code-block:: bash

    export CHPL_LAUNCHER_CONSTRAINT=cal

* Optionally, you can specify a slurm account by setting the environment
  variable ``CHPL_LAUNCHER_ACCOUNT``. For example, to use the account 'acct',
  set:

  .. code-block:: bash

    export CHPL_LAUNCHER_ACCOUNT=acct

* If you wish wish to use sbatch to launch the job to the queue system, either
  set the environment variable ``CHPL_LAUNCHER_USE_SBATCH`` or pass the
  ``--generate-sbatch-script`` flag to the executable. In this mode, the output will be
  written by default to a file called <executableName>.<jobID>.out. The
  environment variable ``CHPL_LAUNCHER_SLURM_OUTPUT_FILENAME`` can be used
  to specify a different filename for the output.

* Optionally, you can specify the number of GPUs required per node using either the environment variable ``CHPL_LAUNCHER_GPUS_PER_NODE`` or the ``--gpus-per-node`` flag. For example, to request 2 GPUs per node for all runs, set:

  .. code-block:: bash

    export CHPL_LAUNCHER_GPUS_PER_NODE=2

  Or, to request 2 GPUs per node for a single run, use:

  .. code-block:: bash

    ./myprogram --gpus-per-node=2

.. _ssh-launchers-with-slurm:

Using any SSH-based launcher with Slurm
***************************************

It is possible to use any SSH-based launcher with Slurm, with some additionally
effort. This strategy can come in handy if other launchers are not working.
However, launchers such as `slurm-srun` and `slurm-gasnetrun_ibv` offer a
better experience.

First, let's see how to use an SSH-based launcher with an interactive `salloc`
session. Here we will assume the UDP conduit, but any other launcher supporting
SSH can be configured analogously.

.. code-block:: bash

   # Compile a sample program
   chpl -o hello6-taskpar-dist examples/hello6-taskpar-dist.chpl

   # Reserve 2 nodes for an interactive run
   salloc -N 2
   # Then, within the salloc shell

     # Specify that ssh should be used
     export GASNET_SPAWNFN=S
     # Specify the list of nodes to use; SSH_SERVERS can also be used
     export GASNET_SSH_SERVERS=`scontrol show hostnames | xargs echo`
     # Run the program on the 2 reserved nodes.
     ./hello6-taskpar-dist -nl 2

This strategy can also be used within an *sbatch* script. Here is an
example script to save to the file `job.bash`:

.. code-block:: bash

  #!/bin/bash
  #SBATCH -t 0:10:0
  #SBATCH --nodes=2
  #SBATCH --exclusive
  #SBATCH --partition=chapel
  #SBATCH --output=job.output

  export GASNET_SPAWNFN=S
  export GASNET_SSH_SERVERS=`scontrol show hostnames | xargs echo`

  ./hello6-taskpar-dist -nl 2

To run this job, use:

.. code-block:: bash

  sbatch job.bash

and when it completes, the output will be available in `job.output` as
specified in `job.bash`.

Changing the _real binary suffix
++++++++++++++++++++++++++++++++

In order to support profiling tools that produce new binaries for the
launcher to execute, the suffix of the real binary executed by the
launcher may be changed with the ``CHPL_LAUNCHER_SUFFIX`` environment
variable. If this variable is unset, the suffix defaults to "_real",
matching the compiler's output.


Bypassing the launcher
++++++++++++++++++++++

If the Chapel launcher capability fails you completely, set
``CHPL_LAUNCHER`` to none, recompile, and execute the resulting binary
according to the following rules using tools and queueing mechanisms
appropriate for your system:

* on most systems, the number of locales should be equal to the number
  of nodes on which you execute. That in turn should match the number
  of copies of the program that you are running.

* some queueing systems require you to specify the number of cores to
  use per node. For best results, you will typically want to use all
  of them. All intra-node parallelism is typically implemented using
  Chapel's threading layer (e.g., pthreads), so extra copies of the
  binary are not required per core.

* in our experience, this technique does not work for InfiniBand
  configurations.

Additional launchers
++++++++++++++++++++

In addition to the supported launchers listed above there are several others
that are not actively maintained but may still work.

=============  ==========================================================
Launcher Name  Description
=============  ==========================================================
mpirun         launch using mpirun (no mpi comm currently) 
=============  ==========================================================

.. |-| unicode:: U+2011 .. non-breaking hyphen
  :trim:
