Skip to content

Latest commit

 

History

History
171 lines (140 loc) · 6.47 KB

eos-storage.md

File metadata and controls

171 lines (140 loc) · 6.47 KB
Raw
layout title subtitle minutes
page
First steps in LHCb
Storing large files on EOS
10

Learning Objectives {.objectives}

  • Run a ganga job which puts output onto EOS
  • Open and view the files on EOS

During a real analysis the output of your jobs will quickly grow beyond what fits onto your AFS space. CERN provides you with 2TB of space on a set of hard drives called the EOS service.

ganga needs configuring in order to know which files to store on EOS, as well as where on EOS to store them. Open ~/.gangarc in your favourite editor and look for a line that starts with MassStorageFile. In ganga a MassStorageFile represents a file stored on something like EOS or CASTOR. Change it to look like the following:

MassStorageFile = {'fileExtensions': [''],
                   'uploadOptions': {'path': '/eos/lhcb/user/a/another',
                                     'cp_cmd': '/afs/cern.ch/project/eos/installation/lhcb/bin/eos.select cp',
                                     'ls_cmd': '/afs/cern.ch/project/eos/installation/lhcb/bin/eos.select ls',
                                     'mkdir_cmd': '/afs/cern.ch/project/eos/installation/lhcb/bin/eos.select mkdir'
                                    },
                   'backendPostprocess': {'LSF': 'WN', 'Dirac': 'client',
                                          'LCG': 'client', 'ARC': 'client', 
                                          'CREAM': 'client', 'Localhost': 'WN',
                                          'Interactive': 'client'}
                  }

The line should look very similar to this already, the only thing that needs changing is the path entry. You should change it to your EOS home directory which is /eos/lhcb/user/a/another if your username is another.

Automatically transfer files {.callout}

You can use the fileExtensions entry to specify a list of file extensions that should be transfered to EOS by default. For the moment leave this set to [''].

Related to this there is an entry for DiracFiles, which represent files stored on the Grid. By default any file ending in .dst will not be downloaded nor stored on EOS. It will be stored on some storage element on the Grid. For files which you do not plan to work with interactively and instead feed into a different grid job it makes sense to leave them on the grid.

Note that EOS is 'Grid storage', specifically that which is located at CERN. Your personal space, at /eos/lhcb/user/<your username's first letter>/, is just another location on EOS, and happens to be different to the default location used by DIRAC for 'grid storage'.

DiracFile = {'fileExtensions': ['*.dst'],
             'uploadOptions': {},
             'backendPostprocess': {'LSF': 'WN', 'Dirac': 'WN',
                                    'LCG': 'WN', 'CREAM': 'WN',
                                    'Localhost': 'WN',
                                    'Interactive': 'WN'}
            }

Now that ganga is configured we will modify the minimal DaVinci job to store the nTuple it produces on EOS.

Make a copy of first-job.py and add the following three lines before the j.submit() line:

f = MassStorageFile('DVntuple.root')
f.outputfilenameformat = '/starterkit/{jid}_{fname}'
j.outputfiles = [f]

ganga uses the string you pass to a MassStorageFile constructor to match which files created by your job this MassStorageFile object represents. In this case only files named DVntuple.root will match.

The outputfilenameformat tells ganga where inside your EOS area to store the file and how to name it. The full path to this particular file will be:

/eos/lhcb/user/a/another/starterkit/{jid}_{fname}

Special outputfileformat patterns {.callout}

Your outputfilenameformat string can contain several special strings which will be replaced on a file by file basis. The special strings are: {jid}, {sjid} and {fname}. They stand for job ID, subjob ID, and the filename of the matched file respectively.

Subjobs and outputfileformat {.callout}

When using subjobs it is important to make sure you include the {sjid} pattern in your outputfilenameformat string, otherwise all the subjobs will overwrite each others output.

The final line tells your ganga job that the outputfiles of this job that need special treatment.

One important thing to note is that ganga has to be running after your job has completed to copy the files to EOS. The job can not copy things to EOS itself. You can leave ganga running in a screen session and it will copy files as they become available.

Once your job has completed and the files have been copied to EOS by ganga you can access them from your terminal by mounting your EOS area.

Important! {.callout}

Important: mounting eos is discouraged by the LHCb data management team, and no support will be provided by them for problems/questions related to it.

On lxplus EOS is mount under /eos like this:

$ ls -al /eos/user
total 0
drwxr-xr-x. 1 daemon root 269935142640435 Oct  6  2014 .
drwxr-xr-x. 9 root   root               0 Oct 24 11:30 ..
drwxr-xr-x. 1 daemon root               0 Dec 15  2014 0
drwxr-xr-x. 1 daemon root  15407422735850 Oct 28 10:13 a
drwxr-xr-x. 1 daemon root   4213907467001 Oct 26 16:10 b
drwxr-xr-x. 1 daemon root   4592926796516 Oct 27 21:49 c
drwxr-xr-x. 1 daemon root   3951569840852 Oct 26 08:20 d
drwxr-xr-x. 1 daemon root   3104338965290 Oct 27 14:27 e
drwxr-xr-x. 1 daemon root   2243039535809 Oct 27 13:48 f
drwxr-xr-x. 1 daemon root   5916063359888 Oct 28 10:05 g
drwxr-xr-x. 1 daemon root   2778840054706 Oct 30 19:56 h
drwxr-xr-x. 1 daemon root    442304842811 Oct 26 10:35 i
drwxr-xr-x. 1 daemon root   8360665587286 Oct 30 13:46 j

If you list the contents of the /eos directory you should see

lhcb ship user

If you used /eos/lhcb/user/a/another/starterkit/{jid}_{fname} when configuring your MassStorageFiles there should now be files visible here:

$ ls /eos/lhcb/user/a/another/starterkit

Once you have found your file you can open it in ROOT like this:

$ root /eos/lhcb/user/a/another/starterkit/myfavouritefile.root

Direct access in ROOT {.callout}

You can also open ROOT files on EOS directly from your ROOT script with:

TFile::Open('root://eoslhcb.cern.ch//eos/lhcb/user/a/another/starterkit/myfavouritefile.root')