diff --git a/docs/docs/user/core-concepts/operation-mode.md b/docs/docs/user/core-concepts/operation-mode.md new file mode 100644 index 000000000..91b8cc6ea --- /dev/null +++ b/docs/docs/user/core-concepts/operation-mode.md @@ -0,0 +1,116 @@ +# Operation Modes in KPOps + +KPOps supports three operation modes—`managed`, `manifest`, and `argo`. These modes determine how resources are managed and allow users to tailor their deployment strategy. + +- **Managed Mode** (default): KPOps uses Helm, and communicates with services like Kafka Rest Proxy, and Kafka Connect under the hood to manage the installation/(graceful) deletion of applications, creation/deletion of Kafka topics, creation/deletion of Connectors defined in your `pipeline.yaml`. +- **Manifest Mode**: Focuses on generating Kubernetes manifests. +- **Argo Mode**: Extends the functionality to include ArgoCD-specific hooks for certain operations, facilitating GitOps workflows with automated cleanup and reset tasks. + +--- + +## Configuring Operation Modes + +It is possible to configure the operation mode in the `config.yaml` file. Please refer to the [Configuration documentation page](https://bakdata.github.io/kpops/9.0/user/core-concepts/config/). +Alternatively, you can to pass the `--operation-mode ` option in the CLI to override the operation mode of the `config.yaml`. You can refer to the [CLI commands documentation](https://bakdata.github.io/kpops/9.0/user/references/cli-commands/#kpops-deploy) for more details. + +--- + +## Generated Resources by Mode and Operation + +### `deploy` + +#### **Manifest Mode** + +- **streams-bootstrap Applications**: + - Depending on your pipeline configuration, Kubernetes `Job`, `Deployment`, `ConfigMap`, and `Service` resources. + - Please refer to [streams-bootstrap Helm Charts](https://github.com/bakdata/streams-bootstrap/tree/master/charts). +- **Topics**: + - Strimzi `KafkaTopic` CRDs. + +#### **Argo Mode** + +- **streams-bootstrap Applications**: + - Depending on your pipeline configuration, Kubernetes `Job`, `Deployment`, `ConfigMap`, and `Service` resources. + - Additional Argo `sync-wave` annotation to ensure Kafka topics are created first (default `sync-wave=0`) before deploying apps (lower priority `sync-wave>0`). All components of each sync wave are deployed in parallel by Argo. + - Please refer to [streams-bootstrap Helm Charts](https://github.com/bakdata/streams-bootstrap/tree/master/charts). +- **Topics**: + - Strimzi `KafkaTopic` CRDs. +- **Cleanup Jobs**: + - Kubernetes `Job` resources configured **with** ArgoCD `PostDelete` hooks, ensuring cleanup tasks are executed after ArgoCD application deletion. + +--- + +### `reset` + +#### **Manifest Mode** + +- **Topics**: + - Strimzi `KafkaTopic` CRDs. +- **Reset Jobs**: + - Kubernetes `Job` resources for resetting Kafka Streams application states. + +#### **Argo Mode** + +- **Topics**: + - Strimzi `KafkaTopic` CRDs. +- **Reset Jobs**: + - Kubernetes `Job` resources **without** ArgoCD `PostDelete` hooks, providing a simpler reset process. + +--- + +### `clean` + +#### **Manifest Mode** + +- **Clean Jobs**: + - Kubernetes `Job` resources for cleaning up temporary resources or artifacts using application container images. + +#### **Argo Mode** + +- **Not Applicable**: + - The `clean` command is not supported in Argo mode. The clean is instead achieved through cleanup job hooks during the `deploy` command. + +--- + +### `destroy` + +#### **Manifest Mode** + +- **Topics**: + - Strimzi `KafkaTopic` CRDs. + +#### **Argo Mode** + +- **Topics**: + - Strimzi `KafkaTopic` CRDs. + +--- + +## Use Cases for Each Mode + +### Manifest Mode + +- **Flexibility**: Use the generated manifests in manual workflows or integrate with any Kubernetes deployment tool. +- **Version Control**: Commit generated manifests to a Git repository for tracking changes and rollback. + +### Argo Mode + +- **GitOps Integration**: Simplifies workflows when using ArgoCD for automated deployments and lifecycle management. +- **PostDelete Hooks**: Automatically cleans up resources after deletion of ArgoCD applications. + +--- + +## Summary of Resource Generation by Operation and Mode + +| Resource Type | `deploy` | `reset` | `clean` | `destroy` | +| ------------- | -------------------------------- | ------------------- | ------------------- | ------------------- | +| Producer Apps | Manifest: Generated | N/A | N/A | N/A | +| | Argo: Generated | | | | +| Streams Apps | Manifest: Generated | N/A | N/A | N/A | +| | Argo: Generated | | | | +| Topics | Manifest: Generated | Manifest: Generated | N/A | Manifest: Generated | +| | Argo: Generated | Argo: Generated | | Argo: Generated | +| Cleanup Jobs | Manifest: N/A | N/A | Manifest: Generated | N/A | +| | Argo: With `PostDelete` hooks | N/A | N/A | N/A | +| Reset Jobs | Manifest: N/A | Manifest: Generated | N/A | N/A | +| | Argo: Without `PostDelete` hooks | | | | diff --git a/kpops/components/streams_bootstrap/producer/producer_app.py b/kpops/components/streams_bootstrap/producer/producer_app.py index fdb5a0833..4c59262a1 100644 --- a/kpops/components/streams_bootstrap/producer/producer_app.py +++ b/kpops/components/streams_bootstrap/producer/producer_app.py @@ -21,6 +21,7 @@ from kpops.const.file_type import DEFAULTS_YAML, PIPELINE_YAML from kpops.manifests.argo import ArgoHook, enrich_annotations from kpops.manifests.kubernetes import K8S_CRON_JOB_NAME_MAX_LEN, KubernetesManifest +from kpops.manifests.strimzi.kafka_topic import StrimziKafkaTopic from kpops.utils.docstring import describe_attr log = logging.getLogger("ProducerApp") @@ -170,6 +171,14 @@ def manifest_deploy(self) -> tuple[KubernetesManifest, ...]: return manifests + @override + def manifest_reset(self) -> tuple[KubernetesManifest, ...]: + if self.to: + return tuple( + StrimziKafkaTopic.from_topic(topic) for topic in self.to.kafka_topics + ) + return () + @override def manifest_clean(self) -> tuple[KubernetesManifest, ...]: if get_config().operation_mode is OperationMode.MANIFEST: diff --git a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_argo_mode/manifest.yaml b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_argo_mode/manifest.yaml index 63dc20d35..fb5a29c61 100644 --- a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_argo_mode/manifest.yaml +++ b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_argo_mode/manifest.yaml @@ -1,3 +1,27 @@ +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: my-cluster + name: my-producer-app-output-topic +spec: + config: {} + partitions: 1 + replicas: 1 + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: my-cluster + name: my-labeled-producer-app-topic-output +spec: + config: {} + partitions: 1 + replicas: 1 + --- apiVersion: batch/v1 kind: Job diff --git a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_manifest_mode/manifest.yaml b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_manifest_mode/manifest.yaml index 63dc20d35..fb5a29c61 100644 --- a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_manifest_mode/manifest.yaml +++ b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_manifest_mode/manifest.yaml @@ -1,3 +1,27 @@ +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: my-cluster + name: my-producer-app-output-topic +spec: + config: {} + partitions: 1 + replicas: 1 + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: my-cluster + name: my-labeled-producer-app-topic-output +spec: + config: {} + partitions: 1 + replicas: 1 + --- apiVersion: batch/v1 kind: Job diff --git a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_python_api/manifest.yaml b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_python_api/manifest.yaml index 63dc20d35..fb5a29c61 100644 --- a/tests/pipeline/snapshots/test_manifest/test_manifest_reset_python_api/manifest.yaml +++ b/tests/pipeline/snapshots/test_manifest/test_manifest_reset_python_api/manifest.yaml @@ -1,3 +1,27 @@ +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: my-cluster + name: my-producer-app-output-topic +spec: + config: {} + partitions: 1 + replicas: 1 + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: my-cluster + name: my-labeled-producer-app-topic-output +spec: + config: {} + partitions: 1 + replicas: 1 + --- apiVersion: batch/v1 kind: Job