Merge remote-tracking branch 'origin/main' into refactor/pydantic-v2

bakdata · Oct 24, 2023 · abe4955 · abe4955
2 parents 7571d75 + b6743bb
commit abe4955
Show file tree

Hide file tree

Showing 7 changed files with 219 additions and 157 deletions.
diff --git a/docs/docs/user/examples/atm-fraud-pipeline.md b/docs/docs/user/examples/atm-fraud-pipeline.md
@@ -72,23 +72,23 @@ kubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:808
 
 ### Deploying the ATM fraud detection pipeline
 
+<!-- dprint-ignore-start -->
+
 1. Export environment variables in your terminal:
 
-   ```shell
-   export DOCKER_REGISTRY=bakdata && \
-   export NAMESPACE=kpops
-   ```
+    ```shell
+    export DOCKER_REGISTRY=bakdata && \
+    export NAMESPACE=kpops
+    ```
 
 2. Deploy the pipeline
 
-   ```shell
-   poetry run kpops deploy ./examples/bakdata/atm-fraud-detection/pipeline.yaml \
-   --pipeline-base-dir ./examples \
-   --config ./examples/bakdata/atm-fraud-detection/config.yaml \
-   --execute
-   ```
-
-<!-- dprint-ignore-start -->
+    ```shell
+    poetry run kpops deploy ./examples/bakdata/atm-fraud-detection/pipeline.yaml \
+    --pipeline-base-dir ./examples \
+    --config ./examples/bakdata/atm-fraud-detection/config.yaml \
+    --execute
+    ```
 
 !!! Note
     You can use the `--dry-run` flag instead of the `--execute` flag and check the logs if your pipeline will be
@@ -120,6 +120,8 @@ You should be able to see pipeline shown in the image below:
     Moreover, Streams Explorer needs a while to scrape the information from Kafka connect.
     Therefore, it might take a bit until you see the whole graph.
 
+<!-- dprint-ignore-end -->
+
 ## Teardown resources
 
 ### PostrgreSQL
@@ -132,6 +134,8 @@ helm --namespace kpops uninstall postgresql
 
 ### ATM fraud pipeline
 
+<!-- dprint-ignore-start -->
+
 1. Export environment variables in your terminal.
 
     ```shell
@@ -149,7 +153,6 @@ helm --namespace kpops uninstall postgresql
     --execute
     ```
 
-<!-- dprint-ignore-start --> 
 !!! Note
     You can use the `--dry-run` flag instead of the `--execute` flag and check the logs if your pipeline will be
     destroyed correctly.

diff --git a/docs/docs/user/getting-started/quick-start.md b/docs/docs/user/getting-started/quick-start.md
@@ -67,37 +67,37 @@ kubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:808
 
 ### Deploying the Word-count pipeline
 
+<!-- dprint-ignore-start -->
+
 1. Copy the [configuration](https://github.com/bakdata/kpops-examples/tree/main/word-count/deployment/kpops){target=_blank} from the [kpops-examples repository](https://github.com/bakdata/kpops-examples/tree/main/word-count){target=_blank} into `kpops>examples>bakdata>word-count` like so:
 
-   ```
-   kpops
-   ├── examples
-   |   ├── bakdata
-   |   |   ├── word-count
-   |   |   |   ├── config.yaml
-   |   |   |   ├── defaults
-   |   |   |   │   └── defaults.yaml
-   |   |   |   └── pipeline.yaml
-   |   |   |
-   ```
+    ```
+    kpops
+    ├── examples
+    |   ├── bakdata
+    |   |   ├── word-count
+    |   |   |   ├── config.yaml
+    |   |   |   ├── defaults
+    |   |   |   │   └── defaults.yaml
+    |   |   |   └── pipeline.yaml
+    |   |   |
+    ```
 
 2. Export environment variables in your terminal:
 
-   ```shell
-   export DOCKER_REGISTRY=bakdata && \
-   export NAMESPACE=kpops
-   ```
+    ```shell
+    export DOCKER_REGISTRY=bakdata && \
+    export NAMESPACE=kpops
+    ```
 
 3. Deploy the pipeline
 
-   ```shell
-   kpops deploy ./examples/bakdata/word-count/pipeline.yaml \
-   --pipeline-base-dir ./examples \
-   --config ./examples/bakdata/word-count/config.yaml \
-   --execute
-   ```
-
-<!-- dprint-ignore-start -->
+    ```shell
+    kpops deploy ./examples/bakdata/word-count/pipeline.yaml \
+    --pipeline-base-dir ./examples \
+    --config ./examples/bakdata/word-count/config.yaml \
+    --execute
+    ```
 
 !!! Note
     You can use the `--dry-run` flag instead of the `--execute` flag and check the logs if your pipeline will be
@@ -144,24 +144,24 @@ helm --namespace kpops uninstall redis
 
 ### Word-count pipeline
 
+<!-- dprint-ignore-start -->
+
 1. Export environment variables in your terminal.
 
-   ```shell
-   export DOCKER_REGISTRY=bakdata && \
-   export NAMESPACE=kpops
-   ```
+    ```shell
+    export DOCKER_REGISTRY=bakdata && \
+    export NAMESPACE=kpops
+    ```
 
 2. Remove the pipeline
 
-   ```shell
-   kpops clean ./examples/bakdata/word-count/pipeline.yaml \
-   --pipeline-base-dir ./examples \
-   --config ./examples/bakdata/word-count/config.yaml \
-   --verbose \
-   --execute
-   ```
-
-<!-- dprint-ignore-start -->
+    ```shell
+    kpops clean ./examples/bakdata/word-count/pipeline.yaml \
+    --pipeline-base-dir ./examples \
+    --config ./examples/bakdata/word-count/config.yaml \
+    --verbose \
+    --execute
+    ```
 
 !!! Note
     You can use the `--dry-run` flag instead of the `--execute` flag and check the logs if your pipeline will be
@@ -174,14 +174,18 @@ helm --namespace kpops uninstall redis
 
 ## Common errors
 
+<!-- dprint-ignore-start -->
+
 - `deploy` fails:
-  1. Read the error message.
-  2. Try to correct the mistakes if there were any. Likely the configuration is not correct or the port-forwarding is not working as intended.
-  3. Run `clean`.
-  4. Run `deploy --dry-run` to avoid having to `clean` again. If an error is dropped, start over from step 1.
-  5. If the dry-run is successful, run `deploy`.
+    1. Read the error message.
+    2. Try to correct the mistakes if there were any. Likely the configuration is not correct or the port-forwarding is not working as intended.
+    3. Run `clean`.
+    4. Run `deploy --dry-run` to avoid having to `clean` again. If an error is dropped, start over from step 1.
+    5. If the dry-run is successful, run `deploy`.
 - `clean` fails:
-  1. Read the error message.
-  2. Try to correct the indicated mistakes if there were any. Likely the configuration is not correct or the port-forwarding is not working as intended.
-  3. Run `clean`.
-  4. If `clean` fails, follow the steps in [teardown](../teardown).
+    1. Read the error message.
+    2. Try to correct the indicated mistakes if there were any. Likely the configuration is not correct or the port-forwarding is not working as intended.
+    3. Run `clean`.
+    4. If `clean` fails, follow the steps in [teardown](../teardown).
+
+<!-- dprint-ignore-end -->
diff --git a/docs/docs/user/getting-started/setup.md b/docs/docs/user/getting-started/setup.md
@@ -14,29 +14,30 @@ In this part, you will set up KPOps. This includes:
 
 ## Setup Kubernetes with k3d
 
+<!-- dprint-ignore-start -->
+
 If you don't have access to an existing Kubernetes cluster, this section will guide you through creating a local cluster. We recommend the lightweight Kubernetes distribution [k3s](https://k3s.io/){target=_blank} for this. [k3d](https://k3d.io/){target=_blank} is a wrapper around k3s in Docker that lets you get started fast.
 
+
 1. You can install k3d with its installation script:
 
-   ```shell
-   wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.4.6/install.sh | bash
-   ```
+    ```shell
+    wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.4.6/install.sh | bash
+    ```
 
-   For other ways of installing k3d, you can have a look at their [installation guide](https://k3d.io/v5.4.6/#installation){target=_blank}.
+    For other ways of installing k3d, you can have a look at their [installation guide](https://k3d.io/v5.4.6/#installation){target=_blank}.
 
 2. The [Kafka deployment](#deploy-kafka) needs a modified Docker image. In that case the image is built and pushed to a Docker registry that holds it. If you do not have access to an existing Docker registry, you can use k3d's Docker registry:
 
-   ```shell
-   k3d registry create kpops-registry.localhost --port 12345
-   ```
+    ```shell
+    k3d registry create kpops-registry.localhost --port 12345
+    ```
 
 3. Now you can create a new cluster called `kpops` that uses the previously created Docker registry:
 
-   ```shell
-   k3d cluster create kpops --k3s-arg "--no-deploy=traefik@server:*" --registry-use k3d-kpops-registry.localhost:12345
-   ```
-
-<!-- dprint-ignore-start -->
+    ```shell
+    k3d cluster create kpops --k3s-arg "--no-deploy=traefik@server:*" --registry-use k3d-kpops-registry.localhost:12345
+    ```
 
 !!! Note
     Creating a new k3d cluster automatically configures `kubectl` to connect to the local cluster by modifying your `~/.kube/config`. In case you manually set the `KUBECONFIG` variable or don't want k3d to modify your config, k3d offers [many other options](https://k3d.io/v5.4.6/usage/kubeconfig/#handling-kubeconfigs){target=_blank}.
@@ -47,48 +48,48 @@ You can check the cluster status with `kubectl get pods -n kube-system`. If all
 
 ## Deploy Kafka
 
+<!-- dprint-ignore-start -->
+
 [Kafka](https://kafka.apache.org/){target=_blank} is an open-source data streaming platform. More information about Kafka can be found in the [documentation](https://kafka.apache.org/documentation/){target=_blank}. To deploy Kafka, this guide uses Confluent's [Helm chart](https://github.com/confluentinc/cp-helm-charts){target=_blank}.
 
 1. To allow connectivity to other systems [Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html#kafka-connect){target=_blank} needs to be extended with drivers. You can install a [JDBC driver](https://docs.confluent.io/kafka-connectors/jdbc/current/jdbc-drivers.html){target=_blank} for Kafka Connect by creating a new Docker image:
 
-   1. Create a `Dockerfile` with the following content:
+    1. Create a `Dockerfile` with the following content:
 
-      ```dockerfile
-      FROM confluentinc/cp-kafka-connect:7.1.3
+        ```dockerfile
+        FROM confluentinc/cp-kafka-connect:7.1.3
 
-      RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0
-      ```
+        RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0
+        ```
 
-   2. Build and push the modified image to your private Docker registry:
+    2. Build and push the modified image to your private Docker registry:
 
-      ```shell
-      docker build . --tag localhost:12345/kafka-connect-jdbc:7.1.3 && \
-      docker push localhost:12345/kafka-connect-jdbc:7.1.3
-      ```
+        ```shell
+        docker build . --tag localhost:12345/kafka-connect-jdbc:7.1.3 && \
+        docker push localhost:12345/kafka-connect-jdbc:7.1.3
+        ```
 
-   Detailed instructions on building, tagging and pushing a docker image can be found in [Docker docs](https://docs.docker.com/){target=_blank}.
+    Detailed instructions on building, tagging and pushing a docker image can be found in [Docker docs](https://docs.docker.com/){target=_blank}.
 
 2. Add Confluent's Helm chart repository and update the index:
 
-   ```shell
-   helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ &&  
-   helm repo update
-   ```
+    ```shell
+    helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ &&  
+    helm repo update
+    ```
 
 3. Install Kafka, Zookeeper, Confluent's Schema Registry, Kafka Rest Proxy, and Kafka Connect. A single Helm chart installs all five components. Below you can find an example for the `--values ./kafka.yaml` file configuring the deployment accordingly. Deploy the services:
 
-   ```shell
-   helm upgrade \
-       --install \
-       --version 0.6.1 \
-       --values ./kafka.yaml \
-       --namespace kpops \
-       --create-namespace \
-       --wait \
-       k8kafka confluentinc/cp-helm-charts
-   ```
-
-<!-- dprint-ignore-start -->
+    ```shell
+    helm upgrade \
+        --install \
+        --version 0.6.1 \
+        --values ./kafka.yaml \
+        --namespace kpops \
+        --create-namespace \
+        --wait \
+        k8kafka confluentinc/cp-helm-charts
+    ```
 
 ??? example "Kafka Helm chart values (`kafka.yaml`)"
     An example value configuration for Confluent's Helm chart. This configuration deploys a single Kafka Broker, a Schema Registry, Zookeeper, Kafka Rest Proxy, and Kafka Connect with minimal resources.

diff --git a/docs/docs/user/getting-started/teardown.md b/docs/docs/user/getting-started/teardown.md
@@ -11,28 +11,32 @@
 The [`kpops` CLI](../references/cli-commands.md) can be used to destroy a pipeline that was previously deployed with KPOps.
 In case that doesn't work, the pipeline can always be taken down manually with `helm` (see section [Infrastructure](#infrastructure)).
 
+<!-- dprint-ignore-start -->
+
 1. Export environment variables.
 
-   ```shell
-   export DOCKER_REGISTRY=bakdata && \
-   export NAMESPACE=kpops
-   ```
+    ```shell
+    export DOCKER_REGISTRY=bakdata && \
+    export NAMESPACE=kpops
+    ```
 
 2. Navigate to the `examples` folder.
-   Replace the `<name-of-the-example-directory>` with the example you want to tear down.
-   For example the `atm-fraud-detection`.
+    Replace the `<name-of-the-example-directory>` with the example you want to tear down.
+    For example the `atm-fraud-detection`.
 
 3. Remove the pipeline
 
-   ```shell
-   # Uncomment 1 line to either destroy, reset or clean.
+    ```shell
+    # Uncomment 1 line to either destroy, reset or clean.
 
-   # poetry run kpops destroy <name-of-the-example-directory>/pipeline.yaml \
-   # poetry run kpops reset <name-of-the-example-directory>/pipeline.yaml \
-   # poetry run kpops clean <name-of-the-example-directory>/pipeline.yaml \
-   --config <name-of-the-example-directory>/config.yaml \
-   --execute
-   ```
+    # poetry run kpops destroy <name-of-the-example-directory>/pipeline.yaml \
+    # poetry run kpops reset <name-of-the-example-directory>/pipeline.yaml \
+    # poetry run kpops clean <name-of-the-example-directory>/pipeline.yaml \
+    --config <name-of-the-example-directory>/config.yaml \
+    --execute
+    ```
+
+<!-- dprint-ignore-end -->
 
 ## Infrastructure
 

diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -32,6 +32,9 @@ theme:
 plugins:
   - macros
   - search
+  - exclude-search:
+      exclude:
+        - resources/*
   - glightbox
 
 extra:
@@ -65,8 +68,8 @@ markdown_extensions:
   - pymdownx.tabbed:
       alternate_style: true
   - pymdownx.emoji:
-      emoji_index: !!python/name:materialx.emoji.twemoji
-      emoji_generator: !!python/name:materialx.emoji.to_svg
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
   - pymdownx.superfences:
       preserve_tabs: true
       custom_fences:
@@ -101,7 +104,7 @@ nav:
             - Environment variables: user/core-concepts/variables/environment_variables.md
             - Substitution: user/core-concepts/variables/substitution.md
     - References:
-      - Migration guide: 
+      - Migration guide:
         - Migrate from v1 to v2: user/migration-guide/v1-v2.md
       - CLI usage: user/references/cli-commands.md
       - Editor integration: user/references/editor-integration.md
@@ -110,4 +113,3 @@ nav:
   - Developer Guide:
       - Auto generation: developer/auto-generation.md
       - Formatting: developer/formatting.md
-