Skip to content

BUDA job submission

Jump to bottom
David Anderson edited this page Dec 5, 2024 · 7 revisions

This document describes how to run Docker jobs in a BOINC project that supports BUDA. It assumes that you have job-submission access on the project.

BUDA job submission is done entirely through the web interface of the project's web site.

BUDA science apps and variants

BUDA applications are called 'science apps'. Each has a name, like 'worker' or 'autodock', and can have 'variants' that use different types of computer hardware. The name of a variant is cpu if it uses a single CPU. Otherwise it's the name of a plan class. There might be variants for 1 CPU, for N CPUs, and for various GPU types.

User file sandbox

The BUDA tools use the User file sandbox for uploading files to the BOINC server. To access it, go to Computing / File sandbox in the menu bar.

Managing science apps and variants.

Select Computing / Job Submission in the menu bar. Click BUDA This shows a list of existing science apps and their variants. You can

  • add or delete a science app;
  • add or delete a variant;
  • submit jobs to a variant.

Adding a variant

Click Add variant. This leads to a form where you select:

  • A plan class name (leave it blank for 1-CPU variants)
  • a Dockerfile (from your file sandbox).
  • A set of 'app files'. This includes:
    • a main program to run in the container
    • other app files if needed
  • a list of the input files names
  • a list of the output files names

The Dockerfile should use a work directory app/. For example:

FROM debian
WORKDIR /app
CMD ./main_2.sh

This specifies an image based on the latest Debian (from Docker Hub) and a main program (in this case a shell script) main_2.sh. This file (and the executables it presumably runs) are included in the set of app files.

Note: when the container runs, the app files and input files will be in the working directory. The Dockerfile should not use COPY to put them there.

If you need to pass command-line arguments into the container, the Dockerfile should contain e.g.

ENV ARGS ""
CMD ./main2.sh ${ARGS}

The arguments are available in the container in the environment variable ARGS. You can access them from a C program using getenv(), or from a shell script as $ARGS.

Batch files

You submit BUDA jobs in 'batches': a single job, or hundreds, or thousands. The jobs in a batch have different input files, different command-line arguments, or both.

A batch of jobs is described by a 'batch file': a .zip file containing one directory per job. Each directory contains the input files for that job, and an optional file cmdline containing command-line argments to be passed to the main program.

If there are input files shared among all jobs, these can be put in a directory shared_input_files. This can save disk space in some cases.

For example, suppose that the app takes input files file1, file2, and file3, and that all jobs use the same file1 but different file2 and file3. In this case the batch file could contain

shared_input_files/
    file1
jobname1/
    [cmdline]
    file2
    file3
    ...
jobname2/
    [cmdline]
    file2
    file3
    ...
...

The file names in the shared directory and in each job directory must match the variant's list of input file names. Each job must have all the input files.

Submitting jobs

Currently you submit BUDA jobs to a particular variant. (In the future we may change this so that you submit jobs to a BUDA app, and they can run on any variant).

You can submit jobs to a variant as follows:

  • Prepare a batch file and upload it to your file sandbox.

  • In the Computing menu on the project web site, select Job submission.

  • In the Job submission page, click BUDA.

  • Click Submit next to the variant you want.

  • Select the batch file from the file list.

  • You can specify command-line arguments to be passed to all jobs in the batch. You can also have per-job command-line arguments in the batch file; in this case the batch-level arguments are passed before the job-level arguments.

  • You can enable debugging output. This will include all Docker commands and their output in the stderr output of each job. This is useful for debugging problems with your Dockerfile.

Monitoring a batch

After submitting a batch of jobs, you're taken to a web page for the batch. This shows, among other things, how many of the jobs have completed. Reload it to update this information.

You can click on a job to see its status (and if it failed, the stderr output). You can view or download its input files.

On the batch page, you can click to download a zip file of the output files of all completed jobs. These filenames have the form

batch_<batchid>__job_<jobname>__file_<filename>

where

  • batchid is the (integer) ID of the batch;
  • jobname is the job name (the directory name from the batch file);
  • filename is the name of the output file as written by the app.

When you're done with the batch, you can 'retire' it. This removes its intput and output files from the server.

Home

Clone this wiki locally