Update the variable documentation (#266)

Co-authored-by: Salomon Popp <[email protected]>
bakdata · Aug 15, 2023 · 3ce7008 · 3ce7008
1 parent fa37ae2
commit 3ce7008
Show file tree

Hide file tree

Showing 18 changed files with 864 additions and 65 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -53,6 +53,9 @@ jobs:
       - name: Generate CLI Usage docs (typer-cli)
         run: poetry run pre-commit run gen-docs-cli --all-files --show-diff-on-failure
 
+      - name: Generate Environment variable docs
+        run: poetry run pre-commit run gen-docs-env-vars --all-files --show-diff-on-failure
+
       - name: Generate pipeline definitions
         run: poetry run pre-commit run gen-docs-components --all-files --show-diff-on-failure
 

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -53,6 +53,13 @@ repos:
         types: [python]
         require_serial: true
         exclude: ^tests/.*snapshots/
+      - id: gen-docs-env-vars
+        name: gen-docs-env-vars
+        entry: python hooks/gen_docs/gen_docs_env_vars.py
+        language: system
+        types: [python]
+        require_serial: true
+        exclude: ^tests/.*snapshots/
       - id: gen-docs-components
         name: gen-docs-components
         entry: python hooks/gen_docs/gen_docs_components.py

diff --git a/docs/docs/dev/autogeneration.md → docs/docs/dev/development/autogeneration.md b/docs/docs/dev/autogeneration.md → docs/docs/dev/development/autogeneration.md
@@ -6,6 +6,14 @@ Autogeneration happens mostly in the `pre-commit` hooks in `/kpops/hooks`.
 
 ### Documentation (`gen_docs/*`)
 
+#### Variables (`./kpops/docs/docs/resources/variables/*`)
+
+- `cli_env_vars.env` -- All CLI environment variables in a `dotenv` file.
+- `cli_env_vars.md` -- All CLI environment variables in a table.
+- `config_env_vars.env` -- Almost all pipeline config environment variables in a `dotenv` file. The script checks for each field in `PipelineConfig` from `./kpops/kpops/cli/pipeline_config.py` whether it has an `env` attribute defined. The script is currently unable to visit the classes of fields like `topic_name_config`, hence any environment variables defined there would remain unknown to it.
+- `config_env_vars.env` -- Almost all pipeline config environment variables in a table.
+- `variable_substitution.yaml` -- A copy of `./tests/pipeline/resources/component-type-substitution/pipeline.yaml` used as an example of substition.
+
 #### CLI commands (`./kpops/docs/docs/user/references/cli-commands.md`)
 
 Generated by `typer-cli` from the code in `./kpops/kpops/cli/main.py`. It is called with Python's `subprocess` module.
@@ -29,4 +37,11 @@ Cached information about KPOps components
     - `defaults_pipeline_component_dependencies.yaml` -- Specifies per component which files in the `sections` dir should be used for the `defaults.yaml` generation.
     - `kpops_structure.yaml` -- Specifies the inheritance hierarchy of the components and what sections exist in each component.
 - `pipeline-components/*\.yaml` -- All single-component pipeline definitions and one big (complete) `pipeline.yaml` that contains all of them.
-- `pipeline-defaults/*\.yaml` -- All single-component defaults definitions and one big (complete) `defaults.yaml` that contains all of them.
+- `pipeline-defaults/*\.yaml` -- All single-component defaults definitions and one big (complete) `defaults.yaml` that contains all of them.
+
+### Editor integration (`gen_schema.py`)
+
+#### Schemas (`./kpops/docs/docs/schema/*`)
+
+- `config.json`
+- `pipeline.json`
diff --git a/docs/docs/resources/variables/cli_env_vars.env b/docs/docs/resources/variables/cli_env_vars.env
@@ -0,0 +1,18 @@
+# CLI Environment variables
+#
+# The default setup is shown. These variables are a lower priority
+# alternative to the commands' flags. If a variable is set, the
+# corresponding flag does not have to be specified in commands.
+# Variables marked as required can instead be set as flags.
+#
+# Base directory to the pipelines (default is current working
+# directory)
+KPOPS_PIPELINE_BASE_DIR=.
+# Path to the config.yaml file
+KPOPS_CONFIG_PATH=config.yaml
+# Path to defaults folder
+KPOPS_DEFAULT_PATH # No default value, not required
+# Path to YAML with pipeline definition
+KPOPS_PIPELINE_PATH # No default value, required
+# Comma separated list of steps to apply the command on
+KPOPS_PIPELINE_STEPS # No default value, not required
diff --git a/docs/docs/resources/variables/cli_env_vars.md b/docs/docs/resources/variables/cli_env_vars.md
@@ -0,0 +1,10 @@
+
+These variables are a lower priority alternative to the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags.
+
+|         Name          |Default Value|Required|                             Description                              |
+|-----------------------|-------------|--------|----------------------------------------------------------------------|
+|KPOPS_PIPELINE_BASE_DIR|.            |False   |Base directory to the pipelines (default is current working directory)|
+|KPOPS_CONFIG_PATH      |config.yaml  |False   |Path to the config.yaml file                                          |
+|KPOPS_DEFAULT_PATH     |             |False   |Path to defaults folder                                               |
+|KPOPS_PIPELINE_PATH    |             |True    |Path to YAML with pipeline definition                                 |
+|KPOPS_PIPELINE_STEPS   |             |False   |Comma separated list of steps to apply the command on                 |
diff --git a/docs/docs/resources/variables/config_env_vars.env b/docs/docs/resources/variables/config_env_vars.env
@@ -0,0 +1,31 @@
+# Pipeline config environment variables
+#
+# The default setup is shown. These variables are a lower priority
+# alternative to the settings in `config.yaml`. Variables marked as
+# required can instead be set in the pipeline config.
+#
+# environment
+# The environment you want to generate and deploy the pipeline to.
+# Suffix your environment files with this value (e.g.
+# defaults_development.yaml for environment=development).
+KPOPS_ENVIRONMENT # No default value, required
+# brokers
+# The comma separated Kafka brokers address.
+KPOPS_KAFKA_BROKERS # No default value, required
+# schema_registry_url
+# Address of the Schema Registry.
+KPOPS_SCHEMA_REGISTRY_URL # No default value, not required
+# kafka_rest_host
+# Address of the Kafka REST Proxy.
+KPOPS_REST_PROXY_HOST # No default value, not required
+# kafka_connect_host
+# Address of Kafka Connect.
+KPOPS_CONNECT_HOST # No default value, not required
+# timeout
+# The timeout in seconds that specifies when actions like deletion or
+# deploy timeout.
+KPOPS_TIMEOUT=300
+# retain_clean_jobs
+# Whether to retain clean up jobs in the cluster or uninstall the,
+# after completion.
+KPOPS_RETAIN_CLEAN_JOBS=False
diff --git a/docs/docs/resources/variables/config_env_vars.md b/docs/docs/resources/variables/config_env_vars.md
@@ -0,0 +1,12 @@
+
+These variables are a lower priority alternative to the settings in `config.yaml`. Variables marked as required can instead be set in the pipeline config.
+
+|          Name           |Default Value|Required|                                                                                Description                                                                                 |   Setting name    |
+|-------------------------|-------------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
+|KPOPS_ENVIRONMENT        |             |True    |The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development).|environment        |
+|KPOPS_KAFKA_BROKERS      |             |True    |The comma separated Kafka brokers address.                                                                                                                                  |brokers            |
+|KPOPS_SCHEMA_REGISTRY_URL|             |False   |Address of the Schema Registry.                                                                                                                                             |schema_registry_url|
+|KPOPS_REST_PROXY_HOST    |             |False   |Address of the Kafka REST Proxy.                                                                                                                                            |kafka_rest_host    |
+|KPOPS_CONNECT_HOST       |             |False   |Address of Kafka Connect.                                                                                                                                                   |kafka_connect_host |
+|KPOPS_TIMEOUT            |          300|False   |The timeout in seconds that specifies when actions like deletion or deploy timeout.                                                                                         |timeout            |
+|KPOPS_RETAIN_CLEAN_JOBS  |False        |False   |Whether to retain clean up jobs in the cluster or uninstall the, after completion.                                                                                          |retain_clean_jobs  |
diff --git a/docs/docs/resources/variables/variable_substitution.yaml b/docs/docs/resources/variables/variable_substitution.yaml
@@ -0,0 +1,36 @@
+- type: scheduled-producer
+  app:
+    labels:
+      app_type: "${component_type}"
+      app_name: "${component_name}"
+      app_schedule: "${component_app_schedule}"
+    commandLine:
+      FAKE_ARG: "fake-arg-value"
+    schedule: "30 3/8 * * *"
+- type: converter
+  app:
+    commandLine:
+      CONVERT_XML: true
+    resources:
+      limits:
+        memory: 2G
+      requests:
+        memory: 2G
+- type: filter
+  name: "filter-app"
+  app:
+    labels:
+      app_type: "${component_type}"
+      app_name: "${component_name}"
+      app_resources_requests_memory: "${component_app_resources_requests_memory}"
+      ${component_type}: "${component_app_labels_app_name}-${component_app_labels_app_type}"
+      test_placeholder_in_placeholder: "${component_app_labels_${component_type}}"
+    commandLine:
+      TYPE: "nothing"
+    resources:
+      requests:
+        memory: 3G
+    replicaCount: 4
+    autoscaling:
+      minReplicas: 4
+      maxReplicas: 4
diff --git a/docs/docs/user/references/variables.md b/docs/docs/user/references/variables.md
diff --git a/docs/docs/user/references/variables/environment_variables.md b/docs/docs/user/references/variables/environment_variables.md
@@ -0,0 +1,35 @@
+# Environment variables
+
+Environment variables can be set by using the [export](https://www.unix.com/man-page/linux/1/export/){target=_blank} command in Linux or the [set](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1){target=_blank} command in Windows.
+
+!!! tip "dotenv files"
+
+    Support for `.env` files is on the roadmap, but not implemented in KPOps yet. One of the possible ways to still use one and export the contents manually is with the following command: `#!sh export $(xargs < .env)`. This would work in `bash` suppose there are no spaces inside the values.
+
+## Config
+
+--8<--
+./docs/resources/variables/config_env_vars.md
+--8<--
+
+??? example "config_env_vars.env"
+
+    ```py title="Exhaustive list of all config-related environment variables"
+    --8<--
+    ./docs/resources/variables/config_env_vars.env
+    --8<--
+    ```
+
+## CLI
+
+--8<--
+./docs/resources/variables/cli_env_vars.md
+--8<--
+
+??? example "cli_env_vars.env"
+
+    ```py title="Exhaustive list of all cli-related environment variables"
+    --8<--
+    ./docs/resources/variables/cli_env_vars.env
+    --8<--
+    ```
diff --git a/docs/docs/user/references/variables/substitution.md b/docs/docs/user/references/variables/substitution.md
@@ -0,0 +1,51 @@
+# Substitution
+
+KPOps supports the usage of placeholders and environment variables in [pipeline definition](../components.md) and [defaults](../defaults.md).
+
+## Component-specific variables
+
+These variables can be used in a component's definition to refer to any of its attributes, including ones that the user has defined in the defaults.
+
+All of them are prefixed with `component_` and follow the following form: `component_{attribute_name}`. If the attribute itself contains attributes, they can be referred to like this: `component_{attribute_name}_{subattribute_name}`.
+
+??? Example
+    ```yaml
+    --8<--
+    ./docs/resources/variables/variable_substitution.yaml
+    --8<--
+    ```
+
+## Pipeline-config-specific variables
+
+These variables include all fields in the [config](../config.md) and refer to the pipeline configuration that is independent of the components.
+
+!!! info Aliases
+    `error_topic_name` is an alias for `topic_name_config_default_error_topic_name`  
+    `output_topic_name` is an alias for `topic_name_config_default_output_topic_name`
+
+## Environment variables
+
+Environment variables such as `$PATH` can be used in the pipeline definition and defaults without any transformation following the form `${ENV_VAR_NAME}`. This, of course, includes variables like the ones relevant to the [KPOps cli](../cli-commands.md) that are exported by the user.
+
+[See all KPOps environment variables](environment_variables.md)
+
+## Pipeline name variables
+
+These are special variables that refer to the name and path of a pipeline.
+
+- `${pipeline_name}`  
+    Concatenated path of the parent directory where pipeline.yaml is defined in.  
+    For instance, `./data/pipelines/v1/pipeline.yaml`, here the value for the variable would be `data-pipelines-v1`.
+
+- `${pipeline_name_<level>}`  
+    Similar to the previous variable, each `<level>` contains a part of the path to the `pipeline.yaml` file.  
+    Consider the previous example, `${pipeline_name_0}` would be `data`, `${pipeline_name_1}` would be `pipelines`, and `${pipeline_name_2}` equals to `v1`.
+
+## Advanced use cases
+
+1. **Refer to default component field values**  
+As long as a value is assigned to a component attribute, it is possible to refer to it with a placeholder. To see all component fields, take a look at the [pipeline schema](../../../schema/pipeline.json).
+2. **Chaining variables**  
+It is possible to chain any number of variables, see the [example](#component-specific-variables) above.
+3. **Cross-component substitution**  
+[YAML](https://yaml.org/){target=_blank} is quite an intricate language and with some of its [magic](https://yaml.org/spec/1.2.2/#692-node-anchors){target=_blank} one could write cross-component references.
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -86,4 +86,6 @@ nav:
       - Config: user/references/config.md
       - Defaults: user/references/defaults.md
       - Editor integration: user/references/editor-integration.md
-      - Variables: user/references/variables.md
+      - Variables: 
+        - Environment variables: user/references/variables/environment_variables.md
+        - Substitution: user/references/variables/substitution.md
diff --git a/hooks/gen_docs/__init__.py b/hooks/gen_docs/__init__.py
@@ -1 +1,30 @@
 """Documentation generation."""
+
+from collections.abc import Generator
+from enum import Enum
+from typing import Any, Generic, TypeVar
+
+_T = TypeVar("_T")
+
+
+class SuperEnum(Generic[_T], Enum):
+    """Adds constructors that return all items in a ``Generator``.
+
+    Introduces constructors that return a ``Generator`` object
+    either containing all items, only their names or their values.
+    """
+
+    @classmethod
+    def items(cls) -> Generator[tuple[_T, Any], None, None]:
+        """Return all item names and values in tuples."""
+        return ((e.name, e.value) for e in cls)
+
+    @classmethod
+    def keys(cls) -> Generator[_T, None, None]:
+        """Return all item names."""
+        return (e.name for e in cls)
+
+    @classmethod
+    def values(cls) -> Generator[Any, None, None]:
+        """Return all item values."""
+        return (e.value for e in cls)