diff --git a/docs/source/experiment.rst b/docs/source/experiment.rst index 8e58e2c..da624eb 100644 --- a/docs/source/experiment.rst +++ b/docs/source/experiment.rst @@ -1,24 +1,24 @@ Experiments -=========== -Data that is processed by the NXRefine package are stored as HDF5 files -stored according to the `NeXus format `_, -which is an international standard for the storage of x-ray and neutron -scattering data. NXRefine uses a hierarchical directory structure to -store both the experimental data, ingested from the raw data generated -on an x-ray beamline, and processed data generated by the data reduction -workflow. Scripts and plugins to the NeXpy GUI are used to facilitate -the creation of both the directory hierarchy and the NeXus files -themselves. +*********** +Data that are processed by the *NXRefine* package are stored as HDF5 +files stored according to the `NeXus format +`_, which is an international standard for +the storage of x-ray and neutron scattering data. *NXRefine* uses a +hierarchical directory structure to store both the experimental data, +ingested from the raw data generated on an x-ray beamline, and processed +data generated by the data reduction workflow. Scripts and plugins to +the *NeXpy* GUI are used to facilitate the creation of both the +directory hierarchy and the NeXus files themselves. In the next section, we will describe the workflow used to both ingest the raw data and transform it into reciprocal space coordinates and pair-distribution-functions. In this section, we will describe the directory framework, within which these processes function, and how -NeXpy can be used to initialize it. +*NeXpy* can be used to initialize it. Experiment Layout ------------------ -An 'experiment' in NXRefine comprises measurements on a set of samples +================= +An 'experiment' in *NXRefine* comprises measurements on a set of samples that are logically grouped together, often because they are performed within a specific time period and/or share calibration files. At synchrotron x-ray facilities, it is common to schedule all the @@ -28,20 +28,21 @@ on beamline 6-ID-D at the Advanced Photon Source, measurements resulting from Proposal No. GUP-75969 may be scheduled in a specific run cycle, say 23-1, and stored in, *e.g.*, ``/data/6-id-d/GUP-75969-23-1``. -In NXRefine, it is assumed that all the files associated with a +In *NXRefine*, it is assumed that all the files associated with a particular experiment are stored in a single directory, *i.e.*, in the APS example above, ``GUP-75969-23-1``. This directory should contain -sub-directories that conform to a particular layout, which store files -such as calibration files (stored in the directory ``calibrations``), -NeXus files containing measurement templates (stored in -``configurations``), settings and log files (stored in ``tasks``), and a -set of directories containing the data from one or more samples measured -at one or more values of a parametric variable (usually temperature). +sub-directories that conform to a particular layout, with calibration +files stored in the directory ``calibrations``, NeXus files containing +measurement templates stored in ``configurations``, settings and log +files stored in ``tasks``, and experimental data from each sample stored +in separate directories with a descriptive name. These sample +directories then contain sub-directories for each crystal, measured +during the experiment, containing the measured data. Here is the structure of a possible experiment directory. Most of the names in this example are chosen to be generic, *i.e.*, they will be different for every experiment. The only exceptions are the files in the -``tasks`` directory, which are set automatically by NXRefine.:: +``tasks`` directory, which are set automatically by *NXRefine*.:: experiment └── tasks @@ -72,15 +73,16 @@ different for every experiment. The only exceptions are the files in the The aim of this directory structure is for all the data and metadata required to analyze the results to be stored in an easily accessible -location, although not all files are required by NXRefine. For example, -the powder calibration files can be imported from any location. +location, although not all files are required by *NXRefine* to be +present. For example, the powder calibration files can be imported from +any location. -If an instrument does not exclusively use NXRefine for its data +If an instrument does not exclusively use *NXRefine* for its data reduction, it is possible that there already exists a directory containing the files that comprise an experiment as described above. To avoid any interference with other instrument files, it is possible to create the above directory structure within a sub-directory, usually -called ``nxrefine``, of the instrument's ``experiment`` directory:: +called ``nxrefine``, of the instrument's ``experiment`` directory.:: experiment └── nxrefine @@ -94,26 +96,28 @@ called ``nxrefine``, of the instrument's ``experiment`` directory:: . . -The name of this sub-directory is defined in the server settings, which -are described in the Server section. +.. note:: The name of this sub-directory is defined in the server + settings, which are described below. Although it can have any + name, it is strongly recommended that it be called + ``nxrefine`` to simplify parsing of the directory tree. Experiment Sub-Directories --------------------------- +========================== **tasks** The ``tasks`` sub-directory contains a number of files used by - NXRefine to store default settings, workflow logs, and a MySQL + *NXRefine* to store default settings, workflow logs, and a MySQL database for recording the status of each workflow component. The - files in this directory are created automatically by NXRefine and - should not be touched. NeXpy GUIs are used to inspect their + files in this directory are created automatically by *NXRefine* and + should not be touched. *NeXpy* GUIs are used to inspect their contents. **calibrations** The ``calibrations`` sub-directory is designed to contain either the TIFF or CBF files generated by measurements of a calibrant powder, or a file, usually with extension ``.poni``, containing the - instrument parameters calibrated using the PyFAI module. The - workflow includes a GUI for performing PyFAI calibrations directly + instrument parameters calibrated using the *PyFAI* module. The + workflow includes a GUI for performing *PyFAI* calibrations directly on powder calibration image files, with the results stored in the NeXus files (described below). The files don't have to be stored in this sub-directory, but if they are in another location, it is @@ -125,23 +129,23 @@ Experiment Sub-Directories The ``configurations`` sub-directory contains NeXus files that act as templates when creating the files used to store the scan results. These files contain scan parameters, such as the goniometer angles, - for one or more sample rotations, and are initialized by a NeXpy GUI - dialog. + for one or more sample rotations, and are initialized by a *NeXpy* + GUI dialog. **scripts** - The ``scripts`` sub-directory is not used directly by NXRefine, but - is created by the ``New Experiment`` dialog described below. It is - designed to store macros for use during an experiment. + The ``scripts`` sub-directory is not used directly by *NXRefine*, + but is created by the ``New Experiment`` dialog described below. It + is designed to store macros for use during an experiment. **sample** The ``sample`` sub-directories are typically named after a common abbreviation or chemical formula of the measured sample (*e.g.*, ``TiSe2``). Within each sample directory are one or more directories usually corresponding to different crystals, specified by unique - labels often provided by the crystal grower. It is common in these - experiments to screen a number of crystals before selecting one for - further measurements, so many of these directories only contain a - single scan. + labels typically provided by the crystal grower. It is common in + these experiments to screen a number of crystals before selecting + one for further measurements, in which case many of these + directories would only contain a single scan. Within each ``label`` directory, there are one or more directories that are named after the parametric variable being modified between @@ -177,27 +181,28 @@ Experiment Sub-Directories .. figure:: /images/instrument-settings.png :align: right :width: 90% - :figwidth: 40% + :figwidth: 50% Instrument Setup ----------------- +================ The experiment directory layout can be created automatically using GUI -dialogs in the NeXpy "Experiment" menu. Before using them, it is +dialogs in the *NeXpy* "Experiment" menu. Before using them, it is important to have initialized the default instrument parameters using -the "Edit Settings" dialog of the NeXpy "Server" menu, or at the command -line using ``nxsettings -i``. +the "Edit Settings" dialog of the *NeXpy* "Server" menu, or at the +command line using ``nxsettings -i``. The instrument settings provide information on the directories, in which -both the raw data and the NXRefine directory tree are located. It is +both the raw data and the *NXRefine* directory tree are located. It is quite common for the raw data to be collected as a set of image files, typically TIFF or CBF files. These are usually not stored in the experiment directories described in the previous section, and may be in -read-only directories. To allow for the input of such files, NXRefine -defines two sets of paths; one to the 'raw' data and one to the NXRefine -(or 'analysis') directories. It is assumed that the experiment names, -*e.g.*, ``GUP-75969-23-1``, are the same in both locations, although -alternative methods of linking the 'raw' and 'analysis' paths could be -defined in the customized beamline classes described later. +read-only directories. To allow for the input of such files, *NXRefine* +defines two sets of paths; one to the 'raw' data and one to the +*NXRefine* (or 'analysis') directories. It is assumed that the +experiment names, *e.g.*, ``GUP-75969-23-1``, are the same in both +locations, although alternative methods of linking the 'raw' and +'analysis' paths could be defined in the customized beamline classes +described later. For example, at CHESS, the 'raw' and 'analysis' paths are defined in parallel directory trees as follows (with generic experiment names):: @@ -241,7 +246,7 @@ Here is a list of instrument parameters. ``/nfs/chess/id4baux/2023-3``. :analysis_path: This is the path within the experiment directory to the - NXRefine sub-directories. In the above example, this + *NXRefine* sub-directories. In the above example, this would be ``nxrefine``. On Sector 6 at the APS, the images are automatically stacked as HDF5 @@ -250,36 +255,97 @@ files and saved in the analysis directories as ``f1.h5``, ``f2.h5``, blank. The 'analysis_path' field is also blank, since the sample directories are at the top level of the experiment directories. -If someone wants to use NXRefine to analyze data collected as image +If someone wants to use *NXRefine* to analyze data collected as image files, which are not stored in a directory tree compatible with the -above description, there are two options. Firstly, the NXBeamLine class, -which is described later, is designed to allow beamline-specific methods -of importing the data and metadata. These can be implemented in separate -packages that are imported into NXRefine as plugins. Secondly, the image -files can be loaded into HDF5 files using the `nexusformat -`_ command-line script, 'nxstack' and -saved to the scan directories described above. Type ``nxstack -h`` at -the terminal command line to see possible options. - -Experiment Setup ----------------- -Once the instrument parameters have been defined in the server settings, -it is possible to initialize the experiment directories using the -"New Experiment" dialog within the NeXpy "Experiment" menu. When the -dialog is launched, click on "Choose Experiment Directory" to launch a -a file browser in order to select a directory. There are two scenarios. - -1. If ``raw_home`` is not blank, the file browser will default to the - ``raw_home`` directory, because this normally means that an - experiment directory already exists, containing the raw image files. - By selecting an existing directory, the dialog automatically fills in - the equivalent path to the analysis directory, in which the NXRefine - files are to be stored. +above description, there are two options. Firstly, the ``NXBeamLine`` +class, which is described later, is designed to allow beamline-specific +methods of importing the data and metadata. These can be implemented in +separate packages that are imported into *NXRefine* as plugins. +Secondly, the image files can be loaded into HDF5 files using the +`nexusformat `_ command-line script, +'nxstack' and saved to the scan directories described above. Type +``nxstack -h`` at the terminal command line to see possible options. + +Experiment Menu +=============== +The *NXRefine* plugin to *NeXpy* installs a top-level menu labelled +"Experiment". The sub-menus run operations to initialize the experiment +layout, create experimental data templates, calibrate powder data, and +initialize new data files. + +New Experiment +-------------- +This dialog initializes a new experiment directory layout using the +server settings to initialize default locations. When the dialog is +launched, click on "Choose Experiment Directory" to launch the system +file browser in order to select or create the new experiment directory. .. figure:: /images/new-experiment-CHESS.png :align: center - :width: 40% + :width: 80% + +There are two scenarios. + +1. If ``raw_home`` is not blank in the server settings, the file browser + will default to the ``raw_home`` directory, in which an experiment + directory, containing the raw image files, should already exist. This + experiment directory is then selected, after which the dialog above + is created, with the experiment name (*i.e.*, the base name of the + experiment directory path) already filled in, along with the path to + analysis home directory (``analysis_home`` in the server settings) + and the name of the analysis sub-directory if required. When the + "Save" button is pressed, the new experiment directory is created + within the analysis home directory if it does not already exist, and + the experiment directory tree is initialized with the + ``calibrations``, ``configurations``, ``scripts`` and ``tasks`` + sub-directories. 2. If ``raw_home`` is blank, the file browser will default to the - ``analysis_home`` directory, from where a new directory can be - created with the experiment name. + ``analysis_home`` directory, but another location can be selected if + required. The file browser can be used either to select an existing + experiment directory or to create a new one. The above dialog is then + created with the experiment name given by the base name of the + selected experiment directory path, and the analysis home directory + defined by its parent. When the "Save" button is pressed, the + experiment directory tree is initialized with the ``calibrations``, + ``configurations``, ``scripts`` and ``tasks`` sub-directories. + +A new ``settings.ini`` file is created in the ``tasks`` sub-directory, +with values copied from the equivalent file in the server directory, +excluding the "Server" section. This allows the refinement parameters to +be customized for each experiment. + +New configuration +----------------- +This dialog creates NeXus files that are used as templates for the +experimental files that are used to store all the data and metadata +associated with a particular set of rotation scans. The initial metadata +is defined by parameters in the settings file in the ``tasks`` +sub-directory, which can be modified by the "Edit Settings" sub-menu +described below. However, some of the metadata will be refined using a +powder calibration, whose results are then stored in this file. + +After selecting the experiment directory, the following dialog is created. + +.. figure:: /images/new-configuration-CHESS.png + :align: center + :width: 80% + +This allows the settings used in subsequent analysis to be initialized, +the parameters defining the rotation scans (range, step size, frame +rate) to be set, the detector configuration to be defined, and the +angles and/or detector positions to be used in one or more rotation +scans. These are all saved to the NeXus template. The wavelength and +detector distance can be nominal values at this stage, since they are +updated by performing a powder calibration. Similarly, the instrument +angles, :math:`\theta`, :math:`\omega`, and :math:`\chi` are set to the +angles set by the motors, but will usually be refined when the sample +orientation is determined. + +It is possible to create more than one configuration template, if, for +example, different angles and/or detector positions are used in +different phases of an experiment. *NXRefine* allows the appropriate +template to be selected when setting up the scan. A separate template +should be created for each configuration that requires a change in the +instrument calibration (wavelength, detector distance, detector +translation) or scan angles. diff --git a/docs/source/images/new-configuration-CHESS.png b/docs/source/images/new-configuration-CHESS.png new file mode 100644 index 0000000..0ae06eb Binary files /dev/null and b/docs/source/images/new-configuration-CHESS.png differ diff --git a/docs/source/index.rst b/docs/source/index.rst index e8e7ed7..1e4d80a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -35,7 +35,9 @@ temperature, are complete. introduction installation + server experiment + refine Indices and tables ================== diff --git a/docs/source/installation.rst b/docs/source/installation.rst index a8ca5f0..67384cf 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -1,6 +1,6 @@ Installation ============ -Currently, NXRefine must be installed from source by cloning the +Currently, *NXRefine* must be installed from source by cloning the `NXRefine Git repository `_:: $ git clone https://github.com/nexpy/nxrefine.git @@ -11,8 +11,8 @@ from within the source directory:: $ python -m build # build a distribution $ python -m pip install . # install the package -In the near future, NXRefine will be uploaded to the PyPI server so that -it can be installed without downloading the source code. +In the near future, *NXRefine* will be uploaded to the PyPI server so +that it can be installed without downloading the source code. Required Libraries ------------------ @@ -31,12 +31,12 @@ persist-queue https://github.com/peter-wangxu/persist-queue NeXpy ----- -Although much of the NXRefine workflow can be performed using +Although much of the *NXRefine* workflow can be performed using command-line scripts, it is recommended that they are used in -conjunction with the Python-based GUI, -`NeXpy `_. Once NXRefine has been -installed, NeXpy will automatically import a set of plugins that add -three menu items to the NeXpy menu bar. +conjunction with the Python-based GUI, `NeXpy +`_. Once NXRefine has been installed, +*NeXpy* will automatically import a set of plugins that add three menu +items to the *NeXpy* menu bar. .. figure:: /images/NeXpy-GUI.png :align: center @@ -61,13 +61,18 @@ three menu items to the NeXpy menu bar. Dialogs to manage workflow operations on existing data, view server logs, and edit default settings for future experiments. +.. note:: If the *NXRefine* menu items do not appear in the menu bar, + check the NeXpy log file ("Show Log File" in the Window menu) + for any error messages preventing the plugin from being + loaded. + The menus will be described in more detail in subsequent sections. CCTW ---- `CCTW `_ (Crystal Coordination Transformation Workflow) is a C++ package written by Guy Jennings. It -is launched as a separate process by NXRefine, which uses the +is launched as a separate process by *NXRefine*, which uses the experimental metadata to define the settings file used to define the input and output grids. It has to be separately installed. @@ -78,181 +83,3 @@ If you are interested in using this package, please contact Ray Osborn `Github issue `_, with relevant tracebacks. -Initial Setup -------------- -In order to allow NXRefine to be used on machines with multiple users, -a directory is defined to store log files, task queues, and settings. -This directory can be initialized on the command line by the 'nxserver' -command: - - $ nxserver -d /path/to/parent - -This will create a directory at ``/path/to/parent/nxserver`` containing -the files that are required by NXRefine server. - -.. note:: If the supplied path already ends in ``nxserver``, it will not - be appended. - -All the files in the ``nxserver`` directory will have group read/write -permissions to allow them to be updated by multiple users in that group. - -This also adds a hidden file to the home directory, in -``~/.nxserver/settings.ini`` containing the path to the server -directory, so that the server path can be read in future login sessions. -Each user should then issue the same command to store the server -directory in their own home directory. If the server directory already -exists, it is not touched. In principle, this only needs to be run once, -although it could also be added to a login script. - -NXRefine uses file-based locking to prevent corruption of data files. -This system is provided by the -`nexusformat package `_, which defines -the directory to contain the lock files using the ``NX_LOCKDIRECTORY`` -environment variable. It is recommended that this directory be placed -within the server directory. - -.. note:: The NeXpy GUI has a settings file that can be used to define - the lock directory, but it is overridden by the environment - variable if it is defined. This allows system administrators - to set up a unique lock file directory for all their users. - -Server Directory -^^^^^^^^^^^^^^^^ -Here is the structure of the ``nxserver`` directory:: - - nxserver - ├── nxserver.log - ├── cpu1.log - ├── cpu2.log - ├── cpu3.log - ├── settings.ini - ├── nxsetup.sh - ├── nxcommand.sh - └── task_list - ├── info - └── q00000 - └── locks - ├── ... - └── ... - -**nxserver.log** - - This is a log file that records jobs submitted to the server queue. - -**cpu1.log**, **cpu2.log**, ... - - These are log files that contain the output of jobs running on the - server. The number depends on the number of simultaneous jobs that - are allowed on the server, which is defined by the settings file. - -**settings.ini** - - A file containing default settings used by the NXRefine package, - including server parameters, instrumental parameters, and parameters - used in the data reduction workflow. When a new experiment is set up, - a copy of these parameters is stored in the experiment directory (to - be described later), so that they can be customized if necessary. - These settings are described below. - -**nxsetup.sh** - - A shell script that could be used to initialize paths to the server - directory or environment variables used by NeXpy. This could be run - within a user's ``~/.bashrc`` file, or by other shell scripts used to - launch NXRefine workflow jobs (see below). Here is an example of what - this file could contain.:: - - export NX_LOCKDIRECTORY=/path/to/parent/nxserver/locks - export NX_LOCK=10 - nxserver -d /path/to/parent/nxserver - - Other commands, *e.g.*, to initialize a particular conda environment, - could be also be added to this file. - -**nxcommand.sh** - - A shell script that is used if jobs need to be wrapped before - submission to the job queue, *e.g.*, using ``qsub``. Here is an - example, in which ``nxsetup.sh`` is run in order to initialize - NXRefine.:: - - echo `date` "USER ${USER} JOB_ID ${JOB_ID}" - source /path/to/parent/nxserver/nxsetup.sh - - -**task_list** - - A directory that contains files that implement a file-based FIFO - queuing system for server jobs. - -**locks** - - A directory that contains files that implement the - `nexusformat `_ file-locking system. - Locked files can be viewed, and removed if they are stale, using the - "Show File Locks" dialog in the NeXpy "File" menu. - -.. note:: The log files can be viewed using the "Manage Server" dialog - and the settings file can be modified using the "Edit - Settings" dialog, both of which are located in the "Server" - menu in NeXpy. - -.. figure:: /images/server_settings.png - :align: right - :width: 90% - :figwidth: 30% - -Default Settings -^^^^^^^^^^^^^^^^ -The file, ``settings.ini`` in the server directory contains the default -settings for the server, the beamline, and the workflow. These values -can be changed, either by opening the "Edit Settings" dialog in the -NeXpy "Server" menu or at the command line using ``nxsettings -i``. -Hitting the [Return] key keeps the current value. - -The right-hand figure shows an example of the first two sections of the -``settings.ini``. The parameters in the first section are described -here. The other sections contain information concerning the location -of the data and default values of the data reduction parameters. They -will be described later. - -Server Settings -^^^^^^^^^^^^^^^ -The server settings are used by the workflow server, which is described -in a later section. They define the server configuration, such as the -number of simultaneous jobs that may be run, the command required to -add them to the system's standard job queues, and whether they need to -be wrapped in a shell script. - -:type: The server type can either be ``multicore`` or ``multinode``. The - only difference is that multinode servers have a list of defined - nodes, to which jobs may be submitted, so their names will also - be stored in the settings file. If jobs are submitted to a job - server, without needing to specify the node, or if all the jobs - are performed on the local machine, then the server type should - be ``multicore``. - -:cores: This sets the number of jobs that can be run simultaneously by - the server. Once reaching the limit, new jobs will only start as - old ones are finished. - -:concurrent: This determines whether parallelized processes should be - used in the workflow. These speed up the computation, but - can be disabled if they cause issues with the server. Note - that this refers to whether multiple processes can be run - simultaneously, *e.g.*, in peaks searches, not whether - multiple jobs can be submitted to the server. Valid values - are ``True`` or ``False``. - -:run_command: This is a string that is prepended to any jobs that are - submitted to the server. It can contain a set of switches - in addition to the job submission command itself. - -:template: In some systems, it is necessary to wrap the command that is - submitted to the server in a shell script. This is the name - of the script, which should be stored in the ``nxserver`` - directory. It should contain the string ````, - which is replaced by the job command. - -:cctw: This is the path to the CCTW executable used to transform data - from instrumental coordinates to reciprocal space. diff --git a/docs/source/introduction.rst b/docs/source/introduction.rst index 0af9d5b..6867d5b 100644 --- a/docs/source/introduction.rst +++ b/docs/source/introduction.rst @@ -103,16 +103,16 @@ detector covers reciprocal space volumes that can exceed such volumes contain hundreds, if not thousands, of Brillouin Zones. *NXRefine* has a peak-search algorithm for identifying all the peaks above a certain intensity threshold, which are then used to generate an -orientation matrix, :math:`\mathcal{U}`, that is refined on a large -number of Bragg peaks. +orientation matrix, :math:`\mathcal{U}`, which is refined using the +positions of a large number of Bragg peaks. -Each peak is defined by its coordinates on the detector, :math:`x_p` and -:math:`y_p`, and the goniometer angles :math:`\theta`, :math:`\omega`, -:math:`\chi`, and :math:`\phi` of the diffractometer when the image was -collected. Once the orientation matrix has been determined, these -experimental coordinates can be converted into reciprocal space -coordinates, :math:`\mathbf{Q}(h,k,l)`. The conversion is accomplished -through a set of matrix operations: +Each Bragg peak is defined by its coordinates on the detector, +:math:`x_p` and :math:`y_p`, and the goniometer angles :math:`\theta`, +:math:`\omega`, :math:`\chi`, and :math:`\phi` of the diffractometer +when the image was collected. Once the orientation matrix has been +determined, these experimental coordinates can be converted into +reciprocal space coordinates, :math:`\mathbf{Q}(h,k,l)`. The conversion +is accomplished through a set of matrix operations: .. math:: @@ -214,11 +214,11 @@ performed by a highly efficient multithreaded C++ application, *Crystal Coordinate Transformation Workflow* (*CCTW*), written by Guy Jennings (APS). -*CCTW* needs to be built from source, which is available on `SourceForge -`_. *NXRefine* generates the -parameter file used by *CCTW* for each set of Φ-rotations, launches -the application, and links to the results. Once all the rotation scans -are processed, they are merged into a single reciprocal space grid. On -a multi-core system, it is possible to accomplish the complete -transformation process in less time than it takes to collect the data, -even though the raw data can exceed 100 GB in size. \ No newline at end of file +*CCTW* needs to be built from the source code, which is available on +`SourceForge `_. *NXRefine* +generates the parameter file used by *CCTW* for each set of Φ-rotations +launches the application, and links to the results. Once all the +rotation scans are processed, they are merged into a single reciprocal +space grid. On a multi-core system, it is possible to accomplish the +complete transformation process in less time than it takes to collect +the data, even though the raw data can exceed 100 GB in size. \ No newline at end of file diff --git a/docs/source/refine.rst b/docs/source/refine.rst new file mode 100644 index 0000000..8635031 --- /dev/null +++ b/docs/source/refine.rst @@ -0,0 +1,2 @@ +Data reduction +************** \ No newline at end of file diff --git a/docs/source/server.rst b/docs/source/server.rst new file mode 100644 index 0000000..2827c94 --- /dev/null +++ b/docs/source/server.rst @@ -0,0 +1,189 @@ +Server Configuration +==================== +*NXRefine* implements a data reduction workflow, which can be run as a +series of line commands in the terminal. However, since some of the +processes can take a long time to complete (from a few minutes to an +hour, depending on the system being used), it is possible to queue these +operations using the *NXRefine*'s queue manager, to be run locally using multiple cores or distributed to other nodes. The *NXRefine* queue manager can be configured to submit jobs to another job queue manager if one is available. + +Initial Setup +------------- +In order to allow *NXRefine* to be used on machines with multiple users, +a directory is defined to store log files, task queues, and settings, +which define how the queue is configured and log the results. However, +if *NXRefine* is installed for use by a single user, this directory can +be in their local home directory. + +The location of the server directory needs to be initialized on the command line by the 'nxserver' command: + + $ nxserver -d /path/to/parent + +This will create a directory at ``/path/to/parent/nxserver`` containing +the files that are required by *NXRefine* server. + +.. note:: If the supplied path already ends in ``nxserver``, it will not + be appended. + +All the files in the ``nxserver`` directory will have group read/write +permissions to allow them to be updated by multiple users in that group. + +This also adds a hidden file to the home directory, in +``~/.nxserver/settings.ini`` containing the path to the server +directory, so that the server path can be read in future login sessions. +Each user should then issue the same command to store the server +directory in their own home directory. If the server directory already +exists, it is not touched. In principle, this only needs to be run once, +although it could also be added to a login script. + +*NXRefine* uses file-based locking to prevent corruption of data files. +This system is provided by the +`nexusformat package `_, which defines +the directory to contain the lock files using the ``NX_LOCKDIRECTORY`` +environment variable. It is recommended that this directory be placed +within the server directory. + +.. note:: The *NeXpy* GUI has a settings file that can be used to define + the lock directory, but it is overridden by the environment + variable if it is defined. This allows system administrators + to set up a unique lock file directory for all their users. + +Server Directory +^^^^^^^^^^^^^^^^ +Here is the structure of the ``nxserver`` directory:: + + nxserver + ├── nxserver.log + ├── cpu1.log + ├── cpu2.log + ├── cpu3.log + ├── settings.ini + ├── nxsetup.sh + ├── nxcommand.sh + └── task_list + ├── info + └── q00000 + └── locks + ├── ... + └── ... + +**nxserver.log** + + This is a log file that records jobs submitted to the server queue. + +**cpu1.log**, **cpu2.log**, ... + + These are log files that contain the output of jobs running on the + server. The number depends on the number of simultaneous jobs that + are allowed on the server, which is defined by the settings file. + +**settings.ini** + + A file containing default settings used by the *NXRefine* package, + including server parameters, instrumental parameters, and parameters + used in the data reduction workflow. When a new experiment is set up, + a copy of these parameters is stored in the experiment directory (to + be described later), so that they can be customized if necessary. + These settings are described below. + +**nxsetup.sh** + + A shell script that could be used to initialize paths to the server + directory or environment variables used by *NeXpy*. This could be run + within a user's ``~/.bashrc`` file, or by other shell scripts used to + launch *NXRefine* workflow jobs (see below). Here is an example of + what this file could contain.:: + + export NX_LOCKDIRECTORY=/path/to/parent/nxserver/locks + export NX_LOCK=10 + nxserver -d /path/to/parent/nxserver + + Other commands, *e.g.*, to initialize a particular conda environment, + could be also be added to this file. + +**nxcommand.sh** + + A shell script that is used if jobs need to be wrapped before + submission to the job queue, *e.g.*, using ``qsub``. Here is an + example, in which ``nxsetup.sh`` is run in order to initialize + *NXRefine*.:: + + echo `date` "USER ${USER} JOB_ID ${JOB_ID}" + source /path/to/parent/nxserver/nxsetup.sh + + +**task_list** + + A directory that contains files that implement a file-based FIFO + queuing system for server jobs. + +**locks** + + A directory that contains files that implement the + `nexusformat `_ file-locking system. + Locked files can be viewed, and removed if they are stale, using the + "Show File Locks" dialog in the *NeXpy* "File" menu. + +.. note:: The log files can be viewed using the "Manage Server" dialog + and the settings file can be modified using the "Edit + Settings" dialog, both of which are located in the "Server" + menu in *NeXpy*. + +.. figure:: /images/server_settings.png + :align: right + :width: 90% + :figwidth: 50% + +Default Settings +^^^^^^^^^^^^^^^^ +The file, ``settings.ini`` in the server directory contains the default +settings for the server, the beamline, and the workflow. These values +can be changed, either by opening the "Edit Settings" dialog in the +*NeXpy* "Server" menu or at the command line using ``nxsettings -i``. +Hitting the [Return] key keeps the current value. + +The right-hand figure shows an example of the first two sections of the +``settings.ini``. The parameters in the first section are described +here. The other sections contain information concerning the location +of the data and default values of the data reduction parameters. They +will be described later. + +Server Settings +^^^^^^^^^^^^^^^ +The server settings are used by the workflow server, which is described +in a later section. They define the server configuration, such as the +number of simultaneous jobs that may be run, the command required to +add them to the system's standard job queues, and whether they need to +be wrapped in a shell script. + +:type: The server type can either be ``multicore`` or ``multinode``. The + only difference is that multinode servers have a list of defined + nodes, to which jobs may be submitted, so their names will also + be stored in the settings file. If jobs are submitted to a job + server, without needing to specify the node, or if all the jobs + are performed on the local machine, then the server type should + be ``multicore``. + +:cores: This sets the number of jobs that can be run simultaneously by + the server. Once reaching the limit, new jobs will only start as + old ones are finished. + +:concurrent: This determines whether parallelized processes should be + used in the workflow. These speed up the computation, but + can be disabled if they cause issues with the server. Note + that this refers to whether multiple processes can be run + simultaneously, *e.g.*, in peaks searches, not whether + multiple jobs can be submitted to the server. Valid values + are ``True`` or ``False``. + +:run_command: This is a string that is prepended to any jobs that are + submitted to the server. It can contain a set of switches + in addition to the job submission command itself. + +:template: In some systems, it is necessary to wrap the command that is + submitted to the server in a shell script. This is the name + of the script, which should be stored in the ``nxserver`` + directory. It should contain the string ````, + which is replaced by the job command. + +:cctw: This is the path to the CCTW executable used to transform data + from instrumental coordinates to reciprocal space.