Skip to content

Fabric

Jump to bottom
Rupert Nash edited this page Nov 3, 2015 · 6 revisions

Fabric

HemeLB uses Fabric to facilitate deployment to remote systems.

This can checkout the code from source control, launch the wiki:CMake builds, and run unit-tests or jobs.

Installation

Fabric is a python tool, and may be installed with:

sudo easy_install fabric

Canopy users should instead use the Canopy package manager.

Note from Hywel: I couldn't get anything working with version less than 1.3. So do this instead:

sudo easy_install fabric==1.3

Optional: Installation through setup.py in user space

Sometimes, it's impossible to easily install a Python distribution, e.g. on major supercomputers such as Titan.

An alternative way to install Fabric is to download the source, and run setup.py as follows:

python setup.py build
python setup.py install --user --prefix=

This will setup Fabric in ~/.local/lib/python2.6/site-packages.

Getting started

You may also need to use the command fab rather than fabric.

The fabric setup resides in the source:deploy section of the HemeLB repository.

User-specific information such as remote machine user-ids should be added to a [source:deploy/machines_user_example.yml machines_user.yml] file placed in the deploy/ folder, which you should create by copying machines_user_example.yml and modifying it to specify your own username. In this file, you also specify where, locally, you wish to store HemeLB setuptool profiles, config files, and results.

To get started with a fresh HemeLB deploy on a remote for which you've not used fabric before, type

fab <machine> cold

at your command prompt.

To see that this works, submit a unit-testing job to the remote queue via:

fab <machine> test

Once the job is finished, run

fab <machine> fetch_results

To submit a real HemeLB job, try:

fab <machine> hemelb:config=cylinder

Basic usage

Fabric creates a series of "tasks", which can be invoked by typing

fab <taskname>

at any location within the HemeLB repository. (Fabric looks above the current working directory to find a fabric configuration file. The HemeLB file source:fabfile.py resides in the top of the repository.)

Multiple fabric tasks can be specified at the command line, and are executed in turn.

fab <task1> <task2>

The HemeLB fabric setup provides a series of tasks to set up the name of the remote machine, which are typically used as the first task of the sequence. Most tasks which expect a remote machine configuration will fail if not preceded by a machine selection task. Thus, typically, one executes fabric via:

fab <machinename> <tasks>

for example

fab hector cold

Machines defined so far:

  • localhost - ssh into this machine, pretending it is a fabric remote
  • planck - James H's CCS linux box.
  • julian - Rupert's CCS Mac.
  • oppenheimer - The UCL chemistry infiniband cluster.
  • legion - The UCL legion cluster.
  • hector - The UK National supercomputing infrastructure.
  • entropy - The CCS fileserver -- (typically only used for result storage tasks, not compute.)

Use while developing

The easiest way to get all the dependencies built is to run fab <remote> cold, however when actively developing the main application, the master build fabric command does a complete reconfigure and rebuild which is rather slow. It is usually more efficient to use the code_only commands instead, that just affect the main application.

  • configure_code_only wipes the build area and reconfigures (recommended when altering CMake files)
  • build_code_only runs make -j <sensible_number>
  • install_code_only runs make install
  • code_only does all of the above, in order.

On most remotes, the source management commands reference the current state of Github's master branch. (If your remote does not have Github access for some reason, it will instead "sync", see below in available tasks.)

However when developing you often want to build and run either a branch commit or your current working directory without a commit (e.g. when you've got debugging statements in). To transfer this, it is usually easiest to use fab <remote> sync to explicitly transfer your working directory.

So when doing the usual edit, compile loop, I often use fab <remote> sync install_code_only.

Available tasks

Managing source code on the remote

NB: Most remotes have access to Github but some do not (e.g. no git, no outgoing connections permitted). If the remote cannot access Github, it will fall back to a sync (see below) from your local checkout. If it can access Github then the remote will work on the master branch only.

  • clone - Delete any existing source on the remote and checkout the repository afresh from Github or sync
  • clone_regression_tests - clone or pull (if they exist) the regression tests from Github, or sync.
  • update - Update the remote from Github (git pull) or sync
  • patch - Update the remote repository with the result of git diff. NB: the diff is against your local HEAD not the remote's HEAD or other state
  • sync - Update the remote repository with local changes via rsync (respecting .gitignore)
  • require_recopy - Notify the CMake "master build" system that the code for hemelb needs to be rebuilt.
    • If you change the code, and then want to build using "configure", "build", "install" tasks, you need to run this or the build will not notice the code has changed.
    • For this reason, if you already have a working set of dependencies built, and change the source code of HemeLB itself, you are better off using the "code_only" tasks, which do not have this requirement.
  • reset - Revert changes in the remote repository. NB: using this on a remote without Github access is an error. Arguments are considered as repository root relative files to be reverted individually.
  • send_distributions - Transmit dependency tarballs to remote.
    • Use on systems which don't have access to the web, see below.
  • fetch_distributions Fetch dependency tarballs tfrom remote.

Build HemeLB executables

Master tasks:

  • deploy_cold - Checkout, build, and install hemelb, from new. (Alias 'cold')
  • code_only - Configure, build, and install for the /Code C++ c...
  • update_build - Update and do incremental build.

Configure:

  • configure - CMake configure step for HemeLB and dependencies.
  • configure_code_only - Wipe the build directory and run the CMake configure step for HemeLB code only.

Build:

  • build - CMake build step for HemeLB and dependencies, via "master build".
  • build_code_only - build step for HemeLB code only (i.e. make)
  • build_python_tools - Build and install python scripts. (Alias: tools)

Install:

  • install - CMake install step for HemeLB and dependencies - places binaries and libraries in configured locations.
  • install_code_only - Install step for HemeLB code only - places binaries and libraries in configured locations - i.e. make install

Clean:

  • clean - Clean remote build area. (Via make-clean)
  • clear_build - Wipe out existing built and installed HemeLB.

Select remote machine

  • archer - Select parallel nodes of UK national supercomputing service as remote.
  • julian - Select CCS Mac Pro as the remote.
  • legion - Select the UCL Legion service as the remote.
  • localhost - ssh into the local machine, pretending it is a fabric remote.
  • entropy - Select UCL fileserver as current remote
  • oppenheimer - Select the UCL Chemistry infiniband cluster as the remote.
  • planck - Select James H's CCS linux box as the remote.

==== Manage result and configuration files

Configuration files, setuptool profile files, and job results, are all managed as directories, which bundle together the various files collected for a configuration or result. The following tasks are used to transfer these to and from remotes.

  • fetch_configs - Fetch config files from the remote.
  • fetch_profiles - Fetch results of remote jobs to local result storage.
  • fetch_results - Fetch results of remote jobs to local result storage.
  • put_profiles - Transfer setuptool profile files to a remote.
  • put_results - Transfer result files to a remote.
  • put_configs - Transfer config files to the remote.

These tasks use rsync so that if files have not changed, they are not transferred.

Invoked with no arguments, they act on all directories, but a named argument can be supplied thus:

fab <machine> fetch_results:<job_name>

where <job_name> is the name defined during job submission, see below.

There are tasks which assist with using setuptool to create configs.

  • create_config - Create a config file

Usage:

fab create_config:<profile_name>[,VoxelSize=<VoxelSize>][,Cycles=<Cycles>][,Steps=<Steps>]
  • create_configs - Create many config files, by looping over multiple values of VoxelSize, Cycles, or Steps

Used as create_config, except VoxelSize, Cycles and Steps may be [start:limit:step] tuples, e.g.

fab create_config:cylinder,VoxelSize=[0.001:0.011:0.001]

The end of the range is not included.

The configs so generated, are given names based on a default template:

${profile}_${VoxelSize}_${Steps}_${Cycles}

But this can be modified by giving an optional argument:

fab create_config:cylinder,config_template="I_like_to_ride_my_bi_${Cycles}"

Job submission

  • regression_test - Submit a regression-testing job to the remote queue (Alias: regress)
  • unit_test - Submit a unit-testing job to the remote queue (Alias: test)
  • hemelb - Submit a HemeLB job to the remote queue.

The last of these takes the configuration to use as a compulsory keyword argument, naming the configuration directory in the config storage defined in machines_user.yml.

fab <machine> hemelb:config=<configuration_name>[,<additional_arguments>]

All these tasks take a number of additional keyword arguments, invoked as

fab <machine> <task>[:<argname>=<argval>,...]

These arguments and defaults for the hemelb task are:

  • name - A template for the name to use for the job.
    • default for hemelb: '${config}${build_number}${machine_name}_$cores'
  • cores=4
  • images=10
  • snapshots=10
  • steering=1111 (ID for HemeLB steering session)
  • wall_time='0:15:0'
  • memory='2G'

An additional task, capable of submitting lots of jobs, is:

  • hemelb_profile - Submit HemeLB job(s) to the remote queue based on a setuptools profile

This is risky, as it can overload the remote queue. Typical usage:

fab oppenheimer hemelb_profile:cylinder,VoxelSize=[0.0001:0.0011:0.0001],cores=[1:13:1]

Queue monitoring

  • stat - Check the remote job queue status
  • monitor - Report on the remote status every 30 seconds. ctrl-C to interrupt

Creating new machine configurations

A [source:deploy/machines.yml] machine configuration file specifies machine-specific configurations. This is checked into source control. The [source:deploy/machines_user_example.yml user-specific configuration] overrides this if present.

The configuration files are YAML files.

Jobs are submitted via templates, which reside at source:deploy/templates. There's usually one for each batch-system header and one for each job-type.

Clone this wiki locally