From ca2f04565ebfd920a419031e61fef7b3f011b766 Mon Sep 17 00:00:00 2001 From: nstelter-slac Date: Thu, 3 Oct 2024 20:21:24 -0700 Subject: [PATCH] DOC: qol doc updates --- docs/environment.md | 9 ++++----- docs/index.md | 12 ++++++------ docs/learning_git.md | 2 +- docs/setup.md | 29 +++++++++++++++-------------- docs/testing.md | 18 +++++++++--------- docs/workflow.md | 21 ++++++++++++--------- 6 files changed, 47 insertions(+), 44 deletions(-) diff --git a/docs/environment.md b/docs/environment.md index 378269a..49dcfa7 100644 --- a/docs/environment.md +++ b/docs/environment.md @@ -1,22 +1,21 @@ # Setup Environment -When running on S3DF, you will first need to setup your environment for _python_, libraries like _psana_, etc: +After you are ssh'd into s3df and ssh'd into psana computing pool, you will need to setup your environment for _python_, libraries like _psana_, etc: ``` source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh ``` -Then you can execute the following to setup some project-specific things (must be ran from _beamtime-calibration-suite_ folder): +Then you can execute the following to setup some project-specific things (must be ran in each new terminal session and from _beamtime-calibration-suite_ folder): ``` source setup.sh ``` -_(must be ran in each new terminal session)_ This script simplifies setting up the terminal environment and should be all most users will need to do This script does the following: * appends your cloned location of this library to your _PYTHONPATH_, so scripts in _/suite_scripts_ and other locations on your machine can find the library code - * sets the library's output-root to the shared /rix dir used for the current experiment - * sets the config file to the currently used `rixSuiteConfig.py` file + * sets the library's output-root to the shared dir used for the current experiment + * sets the config file to the currently used config. The rest of the page has more detail on configuring the scripts. diff --git a/docs/index.md b/docs/index.md index fa50f1d..8ec555b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,7 +1,7 @@ # Beamtime calibration suite -This repository contains scripts ran during (or in-between) beamtimes to support calibrating LCLS detectors +This repository contains scripts ran during (or in-between) beamtimes to support calibrating LCLS detectors. -Some more general info can be found on the team's confluence page: [https://confluence.slac.stanford.edu/display/LCLSDET/LCLS+Detector+Department+Home](https://confluence.slac.stanford.edu/display/LCLSDET/LCLS+Detector+Department+Home) +Some more general info can be found on the team's confluence [page] (https://confluence.slac.stanford.edu/display/LCLSDET/LCLS+Detector+Department+Home) ### File organization: @@ -10,10 +10,9 @@ The following describes what and where things are: * _/calibrationSuite_: The library code lives here, and the functions are able to be imported into other scripts as such: ``` -from calibrationSuite.basicSuiteScript import * -from calibrationSuite.fitFunctions import * -from calibrationSuite.Stats import * -from calibrationSuite.cluster import * +from calibrationSuite.basicSuiteScript import BasicSuiteScript +from calibrationSuite import ancillaryMethods +import calibrationSuite.fitFunctions as fitFunctions ... ``` _(documentation on the library functionality is still to come, but example usage is seen by scripts in the /suite_scripts folder)_ @@ -29,3 +28,4 @@ _(documentation on the library functionality is still to come, but example usage * `development`: contains the newest 'good' code, usually has new changes and bug fixes. This is also the default branch for the repo * `main`: stable code that's used during beamtimes. The code from each beamtime (including the changes made during) is tagged +* `beamtime___`: branch to checkout and push to during a specific beamtime. The date should be the first day of the beamtime. \ No newline at end of file diff --git a/docs/learning_git.md b/docs/learning_git.md index 586a9fa..759f808 100644 --- a/docs/learning_git.md +++ b/docs/learning_git.md @@ -14,4 +14,4 @@ git commands cheat-sheet can be found [here](https://www.jrebel.com/system/files (if you plan on developing the library) read about GitHub's 'Pull Request' feature [here](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) -for commit messages, we can try to follow the PyDM [guidelines](https://slaclab.github.io/pydm/development/development.html#commit-guidelines) \ No newline at end of file +for commit messages, follow what's done for PyDM [guidelines](https://slaclab.github.io/pydm/development/development.html#commit-guidelines) \ No newline at end of file diff --git a/docs/setup.md b/docs/setup.md index e9b7225..4daaff4 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -33,7 +33,8 @@ The only other prerequisites should be a terminal that can run Unix commands (ma (note: in the 1st command replace `` with your slac unix account-name) ``` ssh -Yt @s3dflogin.slac.stanford.edu -//now enter your info to login to s3df... +# now enter your info to login to s3df... +ssh psana // to access data on s3df source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh ``` * all commands entered in the following steps should be done so in this setup terminal _(the terminal should now have an active ssh session into S3DF)_ @@ -44,8 +45,8 @@ source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh ``` git config --global user.name git config --global user.email -//run one of the following cmds, depending on your preferred editor -//(the editor is used during some git operations) +# run one of the following cmds, depending on your preferred editor +# (the editor is used during some git operations) git config --global core.editor vim git config --global core.editor emacs ``` @@ -60,8 +61,8 @@ git config --global core.editor emacs * To check if this worked, run: ``` ssh -T git@github.com -//if SSH is working, you should get the following output: -//Hi ! You've successfully authenticated, but GitHub does not provide shell access. +# if SSH is working, you should get the following output: +# 'Hi ! You've successfully authenticated, but GitHub does not provide shell access.' ``` ## 7: Download the repo @@ -72,14 +73,14 @@ cd ~ mkdir repos cd repos git clone git@github.com:slaclab/beamtime-calibration-suite.git -//wait for download to finish... +# wait for download to finish... cd beamtime-calibration-suite git switch development ls -//should see a bunch of folders +# should see a bunch of folders cd suite_scripts ls -//now should see a bunch of python scripts +# now should see a bunch of python scripts ``` * if an error occurs with these commands, SSH setup in step **6)** may have had an issue and might need to be debugged @@ -93,19 +94,19 @@ source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh source setup.sh // must be run in each new terminal session (or added to ~/bashric) cd suite_scripts mkdir setup_test_output -//OUTPUT_ROOT=. makes the script to look relative to the current directory for the output-folder -OUTPUT_ROOT=. python EventScanParallelSlice.py -r 102 --maxNevents 500 -p /setup_test_outputi -//let the script run to completion... +# OUTPUT_ROOT=. makes the script to look relative to the current directory for the output-folder +OUTPUT_ROOT=. python EventScanParallelSlice.py -r 102 --maxNevents 500 -p /setup_test_output +# let the script run to completion... ls -lt setup_test_output -//if ran correctly, should see these non-empty files: +# if ran correctly, should see some non-empty files in the output dir: eventNumbers_c0_r102_rixx1005922.npy EventScanParallel_c0_r102__n666.h5 means_c0_r102_rixx1005922.npy ``` -## 9: Run the tests (optional): +## 9: Run the tests (optional, mainly for use during development): * run the following terminal commands: ``` cd ~/repos/beamtime_calibtation_suite -// setup the test dependencies and test_data submodule +# setup the test dependencies and test_data submodule source setup_developers.sh pytest . ``` diff --git a/docs/testing.md b/docs/testing.md index 1054d18..abb697a 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,17 +1,17 @@ # Testing -Tests are only need to be run by those actively developing the suite, and should especially be run after -resolving merge-conflicts, adding in large features, refactoring. +Tests really only need to be run by those actively developing the suite, and should especially be run after +resolving merge-conflicts, adding in large features, refactoring, etc. -Running on _S3DF_ (to use psana library) is required to run the entire suite of the tests, +Running on _S3DF_ and ssh'd into the _psana_ pool (to use psana library) is required to run the entire suite of the tests, but a subset can still be run without (utilitiy-function tests, etc). This non-psana subset runs automatically on [github](https://github.com/slaclab/beamtime-calibration-suite/actions/workflows/run-tests.yml) -In short, running the following will check the code: (do so after commiting current changes): +In short, running the following will check the code: (do so after commiting your current changes, since linting/format\ting will modify files): ``` // cd to project root -source setup_developers.sh // only need to run once -./lint_and_format.sh +source setup_developers.sh # only need to run once +./lint_and_format.sh # note: might modify code, but should not modify functionality pytest . ``` @@ -30,7 +30,7 @@ The code be checked with linting by running: ``` ./lint_and_format.sh ``` -This will [lint](https://en.wikipedia.org/wiki/Lint_(software) and attempt to auto-apply fixes, and then format the code. +This will [lint](https://en.wikipedia.org/wiki/Lint_(software)) and attempt to auto-apply fixes, and then format the code. The script may output linting-errors that need to be manually fixed. _(note: this will modify your local files, so reccomended to commit changes first -> run script -> commit changes from running script)_ _(note: running 'ruff --line-length 120' will lint the code without auto-fixing)_ @@ -58,8 +58,8 @@ These tests verify that the code in _/calibrationSuite_ and _/suite\_scripts_ is _test\_SuiteScripts.py_ runs the files in _/suite\_script_ and diffs their real output against expected output in _/tests/test\_data_ _/tests/test\_data_ stores data from this repo: -The repo is added as a [git submodule](https://gist.github.com/gitaarik/8735255), and should be setup by the _setup_developers.sh_ script. +The repo is added as a [git submodule](https://gist.github.com/gitaarik/8735255), and will get setup by the _setup_developers.sh_ script. Since the test data is large, [LFS](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) is used and needs to be installed. (LFS is installed already on _S3DF_) -Some more info on testing, such as adding now data to the _test\_data_ submodule or add a new test-case to_suite\_scripts_ tests, can be found [in this file.](https://github.com/slaclab/beamtime-calibration-suite/blob/development/tests/test_SuiteScripts.py) +Some more info on testing, such as adding now data to the _test\_data_ submodule or add a new test-case to _test_SuiteScripts.py_, can be found [in this file.](https://github.com/slaclab/beamtime-calibration-suite/blob/development/tests/test_SuiteScripts.py) diff --git a/docs/workflow.md b/docs/workflow.md index 78d6f69..235d871 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -12,7 +12,6 @@ We should try to generally follow the branching and pull-request workflow descri * checkout `development` (or a earlier stable point of `development`) * use a pull request to merge `development` into `main` * to do this follow the instructions in section **F)** of the doc page [Git Commands for Common Tasks](https://slaclab.github.io/beamtime-calibration-suite/commands/) - * `main` is safeguarded and will require a +1 from another developer * make a branch for work during this specific beamtime ``` git checkout -b main beamtime___ @@ -52,13 +51,17 @@ git pull ### After Beamtime: * make sure all the changes that need to be saved are commited to `beamtime___` * use a pull request to merge `beamtime___` into `main` -* add a tag for the beamtime +* add a tag for the beamtime, named with date of beamtime: `v..` +* _(if beamtime is multiple days, use the 1st day's date)_ + ``` -//tag number is arbitrary at this point, just look at last tag and increment one of the values -git tag v -a //an example tag number would be 1.0.3 -//this will open your editor to write a description -//the description should be 'Beamtime ' -//(if beamtime is multiple days, use the 1st day's date) +git tag -a //an example tag number would be 1.0.3 +# this will open your editor to write a description +# the description should be 'Beamtime ' + +# push tag +git push origin tag ``` -* use a pull request to merge _main_ into `development` - * this is kinda sloppy, but seems like easiest way to 'reset' things after beamtime \ No newline at end of file + +* lastly, use a pull request to merge _main_ into `development` + * this is kinda sloppy, but seems like easy enough 'reset' things after beamtime \ No newline at end of file