Updated documentation

docs/source/experiment.rst

@@ -3,11 +3,11 @@ Experiment Setup
 Data that is processed by the NXRefine package are stored as HDF5 files
 stored according to the `NeXus format <http://www.nexusformat.org/>`_,
 which is an international standard for the storage of x-ray and neutron
-scattering data. NXRefine uses a hierarchical folder structure to
+scattering data. NXRefine uses a hierarchical directory structure to
 store both the experimental data, ingested from the raw data generated
 on an x-ray beamline, and processed data generated by the data reduction
 workflow. Scripts and plugins to the NeXpy GUI are used to facilitate
-the creation of both the folder hierarchy and the NeXus files
+the creation of both the directory hierarchy and the NeXus files
 themselves.
 
 In the next section, we will describe the workflow used to both ingest
@@ -17,26 +17,40 @@ framework, within which these processes function.
 
 Experiment Files
 ----------------
-NXRefine assumes that measurements on a set of samples can be grouped
-together as a single experiment, typically scheduled within the same
-time period. For example, measurements at synchrotron x-ray facilities
-are often grouped by the proposal number used to acquire beam time, and
-are therefore labelled by a proposal number and/or a run cycle. For
-example, on beamline 6-ID-D at the Advanced Photon Source, measurements
+An 'experiment' in NXRefine comprises measurements on a set of samples
+that are logically grouped together, often because they are performed
+within a specific period and/or share calibration files. At synchrotron
+x-ray facilities, it is common to schedule all the measurements
+associated with a particular proposal together, so these would be
+labelled by the proposal number and/or run cycle. For example, on
+beamline 6-ID-D at the Advanced Photon Source, measurements
 resulting from Proposal No. GUP-75969 may be scheduled in a specific
-run cycle, say 23-1, and stored in ``/data/GUP-75969-23-1``.
+run cycle, say 23-1, and stored in, *e.g.*, ``/data/GUP-75969-23-1``.
 
-Here is the structure of an experiment directory::
+In NXRefine, it is assumed that all the files associated with a
+particular experiment are stored in a single directory, *i.e.*, in the
+APS example above, ``GUP-75969-23-1``. This directory should contain
+files and directories that conform to a particular layout, that includes
+calibration files (stored in the directory ``calibrations``), NeXus
+files containing measurement templates (stored in ``configurations``),
+settings and log files (stored in ``tasks``), and a set of directories
+containing the data from one or more samples measured at one or more
+values of a parametric variable (usually temperature).
+
+Here is the structure of a possible experiment directory. Most of the
+names in this example are chosen to be generic; they will be different
+for every experiment. The only exceptions are the files in the ``tasks``
+directory, which are set automatically by NXRefine.::
 
     experiment
     └── tasks
         ├── nxdatabase.db
         ├── nxlogger.log
         ├── settings.ini
     └── calibrations
-        ├── CeO2_87keV_600mm.tiff
+        └── powder_calibration.tiff (*or .cbf or .poni*)
     └── configurations
-        ├── exp_87keV_600mm.nxs
+        ├── configuration.nxs
     └── sample1
         └── label1
             ├── sample1_100K.nxs
@@ -54,3 +68,45 @@ Here is the structure of an experiment directory::
             └── sample1_300K.nxs
     ├── sample2
     ├── sample3
+
+The aim of this directory structure is for all the data and metadata
+required to analyze the results to be stored in an easily accessible
+location, although not all files are required. For example, the powder
+calibrations can be imported from any location.
+
+The ``sample`` directories are typically named after a common abbreviation
+or chemical formula of the measured sample (*e.g.*, ``TiSe2``). Within
+each sample directory are one or more directories usually corresponding
+to different crystals, specified by unique labels often provided by the
+crystal grower. It is common in these experiments to screen a number of
+crystals before selecting one for further measurements, so many of
+these directories only contain a single scan.
+
+Within each ``label`` directory, there are one or more directories that
+are named after the parametric variable being modified between each set
+of rotation scans, *e.g.*, ``100K``. These ``scan`` directories contain
+the raw data in HDF5 files, typically with extension ``.h5``. Each one
+of these ``.h5`` files contain the raw data from a single rotation scan
+stacked into a single HDF5 array. It is common to perform three
+sample rotations, which are then stored in ``f1.h5``, ``f2.h5``, and
+``f3.h5``, but any number is possible. The ``scan`` directories also
+contain other files produced during the data reduction procedure, such
+as data transformed into reciprocal space coordinates.
+
+For each of these ``scan`` directories, there is a corresponding NeXus
+file that is named as, *e.g.*, ``sample_scan.nxs``, where ``sample``
+must be the name of the ``sample`` directory and ``scan`` should be the
+name of the directory containing the raw data. These NeXus files
+contain external links to the much larger files stored in the ``scan``
+directories. By opening them, the user has access to all the data and
+metadata associated with a particular scan, since external links, if
+they are available, will appear to be part of the file.
+
+.. note:: External links are defined by the file name and internal path
+          to the required HDF5 field. If the file and/or field are not
+          available, the NeXus file can still be opened, but the
+          corresponding data cannot be viewed. The file name is stored
+          as a relative file path, so the NeXus file and a subset of
+          the files in the ``scan`` directory can be moved to another
+          location if, for example, access to the raw data is no
+          longer necessary.