conf management

_data/sidebars/telemetry_sidebar.yml

@@ -0,0 +1,26 @@
+entries:
+- title: Telemetry Analysis
+  product: cacao - Telemetry Analysis
+  folders:
+
+
+  - title: Overview
+    output: web, pdf
+    folderitems:
+
+    - title: Data synchronization
+      url: /cacao_telemetry_datasyncronization.html
+      output: web
+
+
+
+
+  - title: Mapping Functions
+    output: web, pdf
+    folderitems:
+
+    - title: Linear Mapping
+      url: /cacao_telemetry_linearmapping.html
+      output: web
+
+

_data/sidebars/userguide_sidebar.yml

@@ -69,6 +69,9 @@ entries:
       url: /cacao_compute_units.html
       output: web
 
+    - title: Files and Directories
+      url: /cacao_directories_files.html
+      output: web
 
   - title: Running a loop, steps
     output: web, pdf

_data/topnav.yml

@@ -24,6 +24,10 @@ topnav_dropdowns:
       folderitems:
         - title: Tech Notes
           url: /cacao_technotes_about.html
+    - title: Telemetry Analysis
+      folderitems:
+        - title: Linear mapping
+          url: /cacao_telemetry_linearmapping.html
     - title: Contributing
       folderitems:
         - title: Writing Documentation

pages/cacao/cacao_deploying_from_example.md

@@ -21,21 +21,59 @@ cacao-loop-deploy -c <examplename>
 ```
 <span class="label label-info">Tip</span> Run command cacao-loop-deploy with -h option to see list of examples.
 
+The command will create a `-conf` directory holding configuration files:
+- `cacaovars.bash` defines the AO loop name, number, key parameters and list of processes to deploy
+- `fpssetup.setval.conf` has custom (non-default) settings for FPS entries
+- `tasklist.txt` is the list of deployment tasks to be performed.
 
+The `tasklist.txt` usually doesn't need to be edited.
+
+
+You may also create the directory and populate it with the required files without downloading the example.
+
+
+### 1.2. Editing the `cacaovars.bash` file
+
+Edit file `<examplename>-conf/cacaovars.bash` as needed
 The loop number. loop name, DM index and DM simulation index can be changed from their default values by setting the corresponding environment variables. For example:
 
-```bash
-CACAO_LOOPNUMBER=7 cacao-loop-deploy -c <examplename>
-CACAO_LOOPNUMBER=7 CACAO_DMINDEX="03" cacao-loop-deploy -c <examplename>
-```
+<pre>
+CACAO_LOOPNUMBER=7 cacao-loop-deploy -c examplename
+CACAO_LOOPNUMBER=7 CACAO_DMINDEX="03" cacao-loop-deploy -c examplename
+</pre>
+
+### 1.3. Editing the `fpssetup.setval.conf` file
+
+The file has entries in the form:
+<pre>
+setval mlat-LOOPNUMBER.NBiter 453
+</pre>
+The `LOOPNUMBER` word will automatically be replaced by the loop number upon deployment, so keep it such.
+
+This is where users can also configure deployment of processes on CPU sets and realtime priorities:
+<pre>
+setval mlat-LOOPNUMBER.procinfo.RTprio 40
+setval mlat-LOOPNUMBER.procinfo.cset aolRT0
+</pre>
+
+
+### 1.3. Editing the `tasklist.txt` file
+
+A minimum working `tasklist.txt` file is:
+<pre>
+INITSETUP            Initial setup
+TESTCONFIG           Test configuration
+CACAOSETUP           Run cacao-setup
+</pre>
 
-OPTIONAL: Edit file <examplename>-conf/cacaovars.bash as needed
 
 
 ## 2. Deploy processes and tmux sessions
 
 ### 2.1. Run deployment (starts the conf processes):
 
+Once the files in the `-conf` directory have been edited:
+
 ```bash
 cacao-loop-deploy -r <examplename>
 ```

pages/cacao/cacao_directories_files.md

@@ -0,0 +1,101 @@
+---
+title: cacao - Directories and Files
+last_updated: Aug 16, 2024
+tags: [getting_started, setup, framework]
+summary: "Conventions for directories and files"
+sidebar: userguide_sidebar
+permalink: cacao_directories_files.html
+folder: cacao
+---
+
+
+# 1. Operation
+
+
+All cacao commands and scrpts are run from the loop root directory, and an error message will be returned if they are not.
+
+
+# 2. Calibrations
+
+## 2.1. Calibration Files
+
+Calibration files are stored in `conf/` subdirectory of the loop root directory.
+
+- The curent working files (input and output of scripts) are stored as FITS files, such as `dmmask.fits`.
+- These are the key input and output of most cacao scripts.
+- Configuration files, when uploaded to shared memory, are also copied to `shmim.sname.fits` file, which `sname` the name of the shared memory (without the `aolX_` prefix).
+
+{% include note.html content="
+Uploading a calibration file to shared memory should be done with the command `cacao-calib-streamload`, whith will perform the copy to `conf/shmim.sname.fits` and log the operation.
+" %}
+
+
+
+
+
+## 2.2. Calibration scripts
+
+By default cacao scripts will upload results in shared memory and archieved to the `logdir` directory. This allows for quick operation, not having to run separate commands for compute and upload, and performs automatic archival.
+
+{% include warning.html content="
+the uploading to shared memory can disrupt a running AO loop. Use the `-nl` (no loading to memory) to disable it.
+" %}
+
+## 2.3. Managing multiple calibrations
+
+With the environment variable `CACAO_CONFWDIR` set, the configuration working directory (default `conf`) will be changed to `CACAO_CONFWDIR`.
+
+{% include important.html content="
+Except for script `cacao-aorun-042-loadCM`, cacao scripts will only upload files to memory if the configuration working directory is set to `conf` (default), ensuring that calibration files can be handled without affecting the current loop configuration or status.
+" %}
+
+
+For example:
+```bash
+# All commands executed from LOOPROOTDIR
+# Create new configuration directory
+mkdir -p conftest1
+
+# Copy zonal RM from main configuration
+cp ./conf/zrespM.fits ./conftest1/
+
+
+# Create Zernike-Fourier modes
+CACAO_CONFWDIR="conftest1" cacao-aorun-028-mkZFmodes -c0 0 -c1 32 -c 50 -ea 2.0 -t 1.0 -a 0.3
+# output:
+# conftest1/RMmodesDM/modesZF0.fits (not apodized)
+# conftest1/RMmodesDM/modesZF.fits (apodized)
+
+# Create the corresponding response
+CACAO_CONFWDIR="conftest1" cacao-aorun-034-RMzonal2modal modesZF
+# output:
+# conftest1/RMmodesDM.fits
+# conftest1/RMmodesWFS.fits
+
+# Compute the corresponding control matrix using the by-block algorithm
+CACAO_CONFWDIR="conftest1" cacao-aorun-045-compCM-byblocks 0.06 1
+# output:
+# conftest1/CMmodesDM.fits
+# conftest1/CMmodesDM.fits
+
+# Load the CM to shared memory, copy to main conf and log
+CACAO_CONFWDIR="conftest1" cacao-aorun-042-loadCM
+
+```
+
+
+
+## 2.4. User scripts
+
+User calibration scripts should be stored in `conf/scripts/`
+
+
+## 2.2. Archived Files
+
+Most cacao scripts save their output in archive in directory `logdir` which may be a symlink to outside the LOOPROOTDIR.
+
+Files are archieved with the `cacao-calib-logFITSfile` function (part of `cacaofuncs-log`), which will take as argument the `NAME`, and copy file `conf/NAME.fits` to `logdir/aol_LOOPNAME/NAME/NAME.DATESTRING.fits`.
+
+
+
+

pages/cacao/cacao_linearmapping.md

@@ -0,0 +1,16 @@
+---
+title: Linear Mapping
+keywords: 
+last_updated: Aug 14, 2024
+tags: []
+summary: "Building a linear mapping between telemetry streams"
+sidebar: telemetry_sidebar
+datatable: true
+permalink: cacao_telemetry_linearmapping.html
+folder: cacao
+---
+
+
+
+## 1, Description
+

pages/cacao/cacao_message_logging.md

@@ -147,23 +147,23 @@ cacao-msglogCTRL stop
 Processing of the low-level fpsCTRL log to the daily log is running within a tmux session. To view the output in real time, with colors:
 
 ~~~bash
-tmux a -t fpsCTRLlog-$CACAO_LOOPNAME-out-vispyr2
+tmux a -t fpsCTRLlog-out-$CACAO_LOOPNAME
 ~~~
 
 {% include image.html file="fpsCTRLlog-daily-tmux.png" caption="Content of fpsCTRL processing tmux session" %}
 
 ### 2.3. Adding custom log entries to the daily cacao log
 
-The `cacao-log` command adds custom log entries. To add a single entry in the daily cacao log:
+The `cacao-msglogCLI` command adds custom log entries. To add a single entry in the daily cacao log:
 
 ~~~bash
-cacao-log "custom log entry"
+cacao-msglogCLI "custom log entry"
 ~~~
 
 To start an interactive (persistent) logging terminal with a keyword (=category):
 
 ~~~bash
-cacao-log -k "WEATHER" -i
+cacao-msglogCLI -k "WEATHER" -i
 ~~~
 
 {% include image.html file="cacaolog-custom.png" caption="Example use of custom log entries, interactive mode" %}