From 5323037f25a5e60371794fabe77354712c8d4dfb Mon Sep 17 00:00:00 2001 From: Ray Osborn Date: Tue, 24 Oct 2023 10:58:16 -0500 Subject: [PATCH] Updated documentation --- docs/source/experiment.rst | 80 ++++++++++++++++++++++++++++++++------ 1 file changed, 68 insertions(+), 12 deletions(-) diff --git a/docs/source/experiment.rst b/docs/source/experiment.rst index f47d8f8..9f4cb72 100644 --- a/docs/source/experiment.rst +++ b/docs/source/experiment.rst @@ -3,11 +3,11 @@ Experiment Setup Data that is processed by the NXRefine package are stored as HDF5 files stored according to the `NeXus format `_, which is an international standard for the storage of x-ray and neutron -scattering data. NXRefine uses a hierarchical folder structure to +scattering data. NXRefine uses a hierarchical directory structure to store both the experimental data, ingested from the raw data generated on an x-ray beamline, and processed data generated by the data reduction workflow. Scripts and plugins to the NeXpy GUI are used to facilitate -the creation of both the folder hierarchy and the NeXus files +the creation of both the directory hierarchy and the NeXus files themselves. In the next section, we will describe the workflow used to both ingest @@ -17,16 +17,30 @@ framework, within which these processes function. Experiment Files ---------------- -NXRefine assumes that measurements on a set of samples can be grouped -together as a single experiment, typically scheduled within the same -time period. For example, measurements at synchrotron x-ray facilities -are often grouped by the proposal number used to acquire beam time, and -are therefore labelled by a proposal number and/or a run cycle. For -example, on beamline 6-ID-D at the Advanced Photon Source, measurements +An 'experiment' in NXRefine comprises measurements on a set of samples +that are logically grouped together, often because they are performed +within a specific period and/or share calibration files. At synchrotron +x-ray facilities, it is common to schedule all the measurements +associated with a particular proposal together, so these would be +labelled by the proposal number and/or run cycle. For example, on +beamline 6-ID-D at the Advanced Photon Source, measurements resulting from Proposal No. GUP-75969 may be scheduled in a specific -run cycle, say 23-1, and stored in ``/data/GUP-75969-23-1``. +run cycle, say 23-1, and stored in, *e.g.*, ``/data/GUP-75969-23-1``. -Here is the structure of an experiment directory:: +In NXRefine, it is assumed that all the files associated with a +particular experiment are stored in a single directory, *i.e.*, in the +APS example above, ``GUP-75969-23-1``. This directory should contain +files and directories that conform to a particular layout, that includes +calibration files (stored in the directory ``calibrations``), NeXus +files containing measurement templates (stored in ``configurations``), +settings and log files (stored in ``tasks``), and a set of directories +containing the data from one or more samples measured at one or more +values of a parametric variable (usually temperature). + +Here is the structure of a possible experiment directory. Most of the +names in this example are chosen to be generic; they will be different +for every experiment. The only exceptions are the files in the ``tasks`` +directory, which are set automatically by NXRefine.:: experiment └── tasks @@ -34,9 +48,9 @@ Here is the structure of an experiment directory:: ├── nxlogger.log ├── settings.ini └── calibrations - ├── CeO2_87keV_600mm.tiff + └── powder_calibration.tiff (*or .cbf or .poni*) └── configurations - ├── exp_87keV_600mm.nxs + ├── configuration.nxs └── sample1 └── label1 ├── sample1_100K.nxs @@ -54,3 +68,45 @@ Here is the structure of an experiment directory:: └── sample1_300K.nxs ├── sample2 ├── sample3 + +The aim of this directory structure is for all the data and metadata +required to analyze the results to be stored in an easily accessible +location, although not all files are required. For example, the powder +calibrations can be imported from any location. + +The ``sample`` directories are typically named after a common abbreviation +or chemical formula of the measured sample (*e.g.*, ``TiSe2``). Within +each sample directory are one or more directories usually corresponding +to different crystals, specified by unique labels often provided by the +crystal grower. It is common in these experiments to screen a number of +crystals before selecting one for further measurements, so many of +these directories only contain a single scan. + +Within each ``label`` directory, there are one or more directories that +are named after the parametric variable being modified between each set +of rotation scans, *e.g.*, ``100K``. These ``scan`` directories contain +the raw data in HDF5 files, typically with extension ``.h5``. Each one +of these ``.h5`` files contain the raw data from a single rotation scan +stacked into a single HDF5 array. It is common to perform three +sample rotations, which are then stored in ``f1.h5``, ``f2.h5``, and +``f3.h5``, but any number is possible. The ``scan`` directories also +contain other files produced during the data reduction procedure, such +as data transformed into reciprocal space coordinates. + +For each of these ``scan`` directories, there is a corresponding NeXus +file that is named as, *e.g.*, ``sample_scan.nxs``, where ``sample`` +must be the name of the ``sample`` directory and ``scan`` should be the +name of the directory containing the raw data. These NeXus files +contain external links to the much larger files stored in the ``scan`` +directories. By opening them, the user has access to all the data and +metadata associated with a particular scan, since external links, if +they are available, will appear to be part of the file. + +.. note:: External links are defined by the file name and internal path + to the required HDF5 field. If the file and/or field are not + available, the NeXus file can still be opened, but the + corresponding data cannot be viewed. The file name is stored + as a relative file path, so the NeXus file and a subset of + the files in the ``scan`` directory can be moved to another + location if, for example, access to the raw data is no + longer necessary.