Add README files and improve comments

system_config/README.md

@@ -0,0 +1,70 @@
+# System Configuration Example
+
+This example shows how a system comprised of multiple services might utilize a common configuration
+despite each service having different configuration file conventions. The example provides a system
+configuration script ([`sys_config.sh`](./scripts/sys_config.sh)) and a yaml replacement script
+([`replace_params_yaml.sh`](./scripts/replace_params_yaml.sh)). The `sys_config` script is flexible
+and can accept custom script implementations for a service that requires configuration steps beyond
+config file generation or do not use yaml.
+
+## Setup
+
+### Install yq
+
+Currently the only requirement for running the [`sys_config.sh`](./scripts/sys_config.sh) script is
+that `yq` is installed. To install yq run:
+
+```shell
+sudo apt update
+sudo apt install -y snapd
+sudo snap install yq
+```
+
+## Running the System Configuration Script
+
+From the root of the project run:
+
+```shell
+./system_config/scripts/sys_config.sh -c ./system_config/sys_config.yaml -p ./system_config/params.yaml
+```
+
+This will output config files to `./system_config/out_dir` under the respective service sub
+directories.
+
+## Script Inputs
+
+The `sys_config` script takes in a configuration file and a parameters file to construct a set of
+configuration files for defined services with common values in the parameters file.
+
+### System Configuration File
+
+The system configuration file describes the set of services to configure, where to place the
+generated service config files and how to compile the service config files.
+
+See [`System Configuration Template`](./template/README.md#system-configuration-template) for
+details on the separate fields in the template and how to fill the config file out.
+
+An example can be found here: [`sys_config.yaml`](./sys_config.yaml).
+
+### Params File
+
+The params file is a simple set of name value pairs that are common amongst some or all of the
+services being configured.
+
+An example can be found here: [`params.yaml`](./params.yaml).
+
+## FAQ
+
+1. <b>Q</b>: I changed a value in the common `params.yaml` file. What should I do?
+
+    <b>A</b>: If a change has been made to common parameter you should re-run the `sys_config.sh` script
+    to regenerate the service configuration files.
+
+1. <b>Q</b>: How do I add a new service to the system configuration?
+
+    <b>A</b>: There are 3 steps for adding a new service to the system configuration:
+
+    1. Add the service name to the `services` array in `sys_config.yaml`.
+    1. Add a new service metadata object (as defined in
+    [`service metadata`](./template/README.md#service-metadata)).
+    1. Update the `params.yaml` file with any new common parameters.

system_config/scripts/replace_params_yaml.sh

@@ -54,10 +54,20 @@ if [[ -z "${file}" || ${#param_list[@]} -eq 0 ]]; then
     exit 1
 fi
 
-# Uncomment any commented out entries following the pattern
-# '(#\s)(.*:(\s<<value>>)?)$' so the file can be properly processed
+# Ensure all possible params in a template can be processed by the script. This
+# will uncomment any params and nested structures that are commented out in the
+# provided template.
+#
+# Regex pattern used: '(#\s)(.*:(\s<<value>>)?)$'
 sed -i -e 's/\(#[[:space:]]\)\(.*:\([[:space:]]<<value>>\)\?\)$/\2/g' $file
 
+# Replace placeholder values (denoted by '<<value>>') in the provided template
+# with the relevant value from the provided parameters. Parameters are passed
+# into the script in the format of 'param_name=param_value'. 'param_name' is
+# used to locate the params in the template to replace. If a param is in a
+# nested structure the format of the param_name will be 'deeply.nested.param'.
+# If the param_name is not matched in the template, yq adds a null value that
+# is cleaned up in a future step.
 for i in "${param_list[@]}"
 do
     # Split entry into name and value
@@ -68,13 +78,14 @@ do
     yq -i "with((.${name} | select(. == \"<<value>>\")) ; . = \"${value}\" | . style=\"double\")" ${file}
 done
 
-# Remove any unpopulated/null values. Will clean up any parents that have no
-# children
+# Remove any params that did not get filled out along with any nested objects
+# that are now empty. Also remove any null value params created as an artifact
+# of the yq param replacement process.
 yq -i 'del(.. | select(. == "<<value>>")), del(.. | select(. | length == 0))' ${file}
 
 # Strip comments. yq does not handle comments very well and leaving comments in
-# after deletion could cause confusion
+# after deletion could cause confusion.
 yq -i '... comments=""' ${file}
 
-# Add header to indicate the config file was made by a script
+# Add header to indicate the config file was modified/filled out by a script.
 yq -i '. head_comment = "Generated configuration file"' ${file}

system_config/template/README.md

@@ -0,0 +1,212 @@
+# System Configuration Template
+
+This readme describes the [`sys_config.yaml`](sys_config.yaml) template in detail.
+
+## Overview
+
+The [`sys_config.yaml`](sys_config.yaml) is used by the [`sys_config.sh`](../sys_config.sh) script
+to determine what services should be configured and how to configure them. The file defines service
+configuration metadata, telling the `sys_config` script what service specific configuration files
+need to be created and what command to use to create these files with the appropriate argument
+information for the command.
+
+## System Configuration Template Components
+
+The template is comprised of three major parts, the `services` array, `out_dir` field, and
+`services metadata` list:
+
+### Services Array:
+
+```yaml
+services: []
+```
+
+This is a simple array where you will add the names of each service to configure. Each name is
+correlated with the [`Service Metadata`](#service-metadata). An example with two services is:
+
+```yaml
+services: ["service1", "service2"]
+```
+
+### Output Directory:
+
+```yaml
+output_dir: <<value>>
+```
+
+This is a path to the directory where the configuration files should be placed. Each service
+will get their own sub_directory (named the same as the name used in the
+[`Services Array`](#services-array)). The path can be relative to the path where the
+[`sys_config.sh`](../sys_config.sh) script is run or an absolute path. An absolute path is
+generally preferred if the sys_config.sh script may be run from an unknown location. An example of
+an absolute path is:
+
+```yaml
+output_dir: /home/userA/system_config
+```
+
+### Service Metadata:
+
+```yaml
+`service_name`:
+  - template: <<value>>
+    command: <<value>>
+    args:
+      - arg: <<value>>
+        params: []
+        optional: <<true|false>>
+```
+
+For each service defined in the [`Services Array`](#services-array) there is an associated metadata
+object with the same `service_name` that was used in the services array. This metadata object is
+comprised of one or more config objects. The config object is comprised of a `template`, `command`,
+and `args` list:
+
+#### Template Field
+
+```yaml
+- template: <<value>>
+```
+
+The template field is a path to the relevant service config template file. The path can be relative
+to the path where the [`sys_config.sh`](../sys_config.sh) script is run or an absolute path. An
+absolute path is generally preferred if the sys_config.sh script may be run from an unknown
+location.
+
+An example pointing to a config template for `service_1` is:
+
+```yaml
+- template: /home/userA/service_1/config/template/svc_config.yaml
+```
+
+#### Command Field
+
+```yaml
+command: <<value>>
+```
+
+The command field is a path to the relevant service config script. The path can be relative to the
+path where the [`sys_config.sh`](../sys_config.sh) script is run or an absolute path. An absolute
+path is generally preferred if the sys_config.sh script may be run from an unknown location. The
+config script will handle creating the appropriate service configuration with any common parameters
+passed to the system config script via the params file. A simple yaml find and replace params
+script is provided in this repo [here](../scripts/replace_params_yaml.sh) and can work for most
+basic yaml config files.
+
+An example pointing to the `replace_params_yaml.sh` script:
+
+```yaml
+command: /home/userA/chariott-examples-applications/system_config/scripts/replace_params_yaml.sh
+```
+
+>Note: Any custom script can be used here if the `replace_params_yaml.sh` script is not sufficient.
+
+#### Args List Field
+
+```yaml
+args:
+  - arg: <<value>>
+    params: []
+    optional: <<true|false>>
+```
+
+The argument list is a set of argument metadata that tells the [`sys_config.sh`](../sys_config.sh)
+script what arguments to pass to the config script defined in the [`command field`](#command-field)
+.
+
+##### Arg Field
+
+The `arg` field is a string defining an argument. The `arg` field could have just a flag:
+
+```yaml
+- arg: "-i"
+```
+
+Or can include a flag and a value:
+
+```yaml
+- arg: "--param param_1"
+```
+
+Or can even be just a value depending on what the `command` script expects:
+
+```yaml
+- arg: "param_1"
+```
+
+If an argument string contains `{}` with a number (starting from `1`), the `sys_config` script will
+attempt to replace the placeholder with a value defined in in the [`params array`](#params-array).
+
+```yaml
+- arg: "-p param_1={1}"
+```
+
+##### Params Array
+
+The `params` array is a list of parameters to extract from a `params.yaml` file passed to the
+[`sys_config.sh`](../sys_config.sh) script. The script will also check to see if the param name is
+set as a variable in the `sys_config` script. An example of this is the `out_file` variable which
+is a variable generated at script runtime and is not present in the `params.yaml` file. The
+extracted parameter values are inserted into the `arg` field, replacing the placeholders in the
+string.
+
+Given an example `params.yaml` file:
+
+```yaml
+service_1_uri: "service_1://uri"
+service_2_uri: "service_2://uri"
+```
+
+And an `arg` object entry of:
+
+```yaml
+- arg: "-p service_uri={1}"
+  params: ["service_1_uri"]
+  optional: false
+```
+
+The argument snippet would be compiled as:
+
+```shell
+command_script.sh -p service_uri=service_1://uri
+```
+
+##### Optional Field
+
+The `optional` field can either be `true` or `false`. If set to `false` the `sys_config` script
+will fail if one of the parameters in the [`params array`](#params-array) cannot be found. If set
+to `true` then a parameter is not found in the `params.yaml` file the argument is just skipped.
+
+## Examples
+
+An example [`sys_config.yaml`](sys_config.yaml) creating a system with a single service called
+`service1`:
+
+```yaml
+services: ["service1"]
+
+output_dir: /home/userA/system_config
+
+service1:
+  - template: /home/userA/service_1/config/template/svc_config.yaml
+    command: /home/userA/chariott-examples-applications/system_config/scripts/replace_params_yaml.sh
+    args:
+    - arg: "--file {1}"
+        params: ["out_file"]
+        optional: false
+    - arg: "--param service_uri={1}"
+        params: ["service_1_uri"]
+        optional: false
+    - arg: "--param service_version={1}"
+        params: ["service_1_version"]
+        optional: true
+```
+>Note: The `out_file` is a variable created by the `sys_config` script at runtime but can still be
+referenced here.
+
+Associated `params.yaml` file:
+
+```yaml
+service_1_uri: "example://uri"
+# service_1_version: "0.1.0" # This parameter is optional.
+```