Merge branch 'v9' of github.com:bakdata/kpops into v9

bakdata · Dec 18, 2024 · 5f85a5a · 5f85a5a
2 parents 93beda5 + 6f074ce
commit 5f85a5a
Show file tree

Hide file tree

Showing 5 changed files with 197 additions and 0 deletions.
diff --git a/docs/docs/user/core-concepts/operation-mode.md b/docs/docs/user/core-concepts/operation-mode.md
@@ -0,0 +1,116 @@
+# Operation Modes in KPOps
+
+KPOps supports three operation modes—`managed`, `manifest`, and `argo`. These modes determine how resources are managed and allow users to tailor their deployment strategy.
+
+- **Managed Mode** (default): KPOps uses Helm, and communicates with services like Kafka Rest Proxy, and Kafka Connect under the hood to manage the installation/(graceful) deletion of applications, creation/deletion of Kafka topics, creation/deletion of Connectors defined in your `pipeline.yaml`.
+- **Manifest Mode**: Focuses on generating Kubernetes manifests.
+- **Argo Mode**: Extends the functionality to include ArgoCD-specific hooks for certain operations, facilitating GitOps workflows with automated cleanup and reset tasks.
+
+---
+
+## Configuring Operation Modes
+
+It is possible to configure the operation mode in the `config.yaml` file. Please refer to the [Configuration documentation page](https://bakdata.github.io/kpops/9.0/user/core-concepts/config/).
+Alternatively, you can to pass the `--operation-mode <OPERATION>` option in the CLI to override the operation mode of the `config.yaml`. You can refer to the [CLI commands documentation](https://bakdata.github.io/kpops/9.0/user/references/cli-commands/#kpops-deploy) for more details.
+
+---
+
+## Generated Resources by Mode and Operation
+
+### `deploy`
+
+#### **Manifest Mode**
+
+- **streams-bootstrap Applications**:
+  - Depending on your pipeline configuration, Kubernetes `Job`, `Deployment`, `ConfigMap`, and `Service` resources.
+  - Please refer to [streams-bootstrap Helm Charts](https://github.com/bakdata/streams-bootstrap/tree/master/charts).
+- **Topics**:
+  - Strimzi `KafkaTopic` CRDs.
+
+#### **Argo Mode**
+
+- **streams-bootstrap Applications**:
+  - Depending on your pipeline configuration, Kubernetes `Job`, `Deployment`, `ConfigMap`, and `Service` resources.
+  - Additional Argo `sync-wave` annotation to ensure Kafka topics are created first (default `sync-wave=0`) before deploying apps (lower priority `sync-wave>0`). All components of each sync wave are deployed in parallel by Argo.
+  - Please refer to [streams-bootstrap Helm Charts](https://github.com/bakdata/streams-bootstrap/tree/master/charts).
+- **Topics**:
+  - Strimzi `KafkaTopic` CRDs.
+- **Cleanup Jobs**:
+  - Kubernetes `Job` resources configured **with** ArgoCD `PostDelete` hooks, ensuring cleanup tasks are executed after ArgoCD application deletion.
+
+---
+
+### `reset`
+
+#### **Manifest Mode**
+
+- **Topics**:
+  - Strimzi `KafkaTopic` CRDs.
+- **Reset Jobs**:
+  - Kubernetes `Job` resources for resetting Kafka Streams application states.
+
+#### **Argo Mode**
+
+- **Topics**:
+  - Strimzi `KafkaTopic` CRDs.
+- **Reset Jobs**:
+  - Kubernetes `Job` resources **without** ArgoCD `PostDelete` hooks, providing a simpler reset process.
+
+---
+
+### `clean`
+
+#### **Manifest Mode**
+
+- **Clean Jobs**:
+  - Kubernetes `Job` resources for cleaning up temporary resources or artifacts using application container images.
+
+#### **Argo Mode**
+
+- **Not Applicable**:
+  - The `clean` command is not supported in Argo mode. The clean is instead achieved through cleanup job hooks during the `deploy` command.
+
+---
+
+### `destroy`
+
+#### **Manifest Mode**
+
+- **Topics**:
+  - Strimzi `KafkaTopic` CRDs.
+
+#### **Argo Mode**
+
+- **Topics**:
+  - Strimzi `KafkaTopic` CRDs.
+
+---
+
+## Use Cases for Each Mode
+
+### Manifest Mode
+
+- **Flexibility**: Use the generated manifests in manual workflows or integrate with any Kubernetes deployment tool.
+- **Version Control**: Commit generated manifests to a Git repository for tracking changes and rollback.
+
+### Argo Mode
+
+- **GitOps Integration**: Simplifies workflows when using ArgoCD for automated deployments and lifecycle management.
+- **PostDelete Hooks**: Automatically cleans up resources after deletion of ArgoCD applications.
+
+---
+
+## Summary of Resource Generation by Operation and Mode
+
+| Resource Type | `deploy`                         | `reset`             | `clean`             | `destroy`           |
+| ------------- | -------------------------------- | ------------------- | ------------------- | ------------------- |
+| Producer Apps | Manifest: Generated              | N/A                 | N/A                 | N/A                 |
+|               | Argo: Generated                  |                     |                     |                     |
+| Streams Apps  | Manifest: Generated              | N/A                 | N/A                 | N/A                 |
+|               | Argo: Generated                  |                     |                     |                     |
+| Topics        | Manifest: Generated              | Manifest: Generated | N/A                 | Manifest: Generated |
+|               | Argo: Generated                  | Argo: Generated     |                     | Argo: Generated     |
+| Cleanup Jobs  | Manifest: N/A                    | N/A                 | Manifest: Generated | N/A                 |
+|               | Argo: With `PostDelete` hooks    | N/A                 | N/A                 | N/A                 |
+| Reset Jobs    | Manifest: N/A                    | Manifest: Generated | N/A                 | N/A                 |
+|               | Argo: Without `PostDelete` hooks |                     |                     |                     |
diff --git a/kpops/components/streams_bootstrap/producer/producer_app.py b/kpops/components/streams_bootstrap/producer/producer_app.py
@@ -21,6 +21,7 @@
 from kpops.const.file_type import DEFAULTS_YAML, PIPELINE_YAML
 from kpops.manifests.argo import ArgoHook, enrich_annotations
 from kpops.manifests.kubernetes import K8S_CRON_JOB_NAME_MAX_LEN, KubernetesManifest
+from kpops.manifests.strimzi.kafka_topic import StrimziKafkaTopic
 from kpops.utils.docstring import describe_attr
 
 log = logging.getLogger("ProducerApp")
@@ -170,6 +171,14 @@ def manifest_deploy(self) -> tuple[KubernetesManifest, ...]:
 
         return manifests
 
+    @override
+    def manifest_reset(self) -> tuple[KubernetesManifest, ...]:
+        if self.to:
+            return tuple(
+                StrimziKafkaTopic.from_topic(topic) for topic in self.to.kafka_topics
+            )
+        return ()
+
     @override
     def manifest_clean(self) -> tuple[KubernetesManifest, ...]:
         if get_config().operation_mode is OperationMode.MANIFEST:

diff --git a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_argo_mode/manifest.yaml b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_argo_mode/manifest.yaml
@@ -1,3 +1,27 @@
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  labels:
+    strimzi.io/cluster: my-cluster
+  name: my-producer-app-output-topic
+spec:
+  config: {}
+  partitions: 1
+  replicas: 1
+
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  labels:
+    strimzi.io/cluster: my-cluster
+  name: my-labeled-producer-app-topic-output
+spec:
+  config: {}
+  partitions: 1
+  replicas: 1
+
 ---
 apiVersion: batch/v1
 kind: Job

diff --git a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_manifest_mode/manifest.yaml b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_manifest_mode/manifest.yaml
@@ -1,3 +1,27 @@
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  labels:
+    strimzi.io/cluster: my-cluster
+  name: my-producer-app-output-topic
+spec:
+  config: {}
+  partitions: 1
+  replicas: 1
+
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  labels:
+    strimzi.io/cluster: my-cluster
+  name: my-labeled-producer-app-topic-output
+spec:
+  config: {}
+  partitions: 1
+  replicas: 1
+
 ---
 apiVersion: batch/v1
 kind: Job

diff --git a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_python_api/manifest.yaml b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_python_api/manifest.yaml
@@ -1,3 +1,27 @@
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  labels:
+    strimzi.io/cluster: my-cluster
+  name: my-producer-app-output-topic
+spec:
+  config: {}
+  partitions: 1
+  replicas: 1
+
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  labels:
+    strimzi.io/cluster: my-cluster
+  name: my-labeled-producer-app-topic-output
+spec:
+  config: {}
+  partitions: 1
+  replicas: 1
+
 ---
 apiVersion: batch/v1
 kind: Job