Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Kafka Connector resetter as individual HelmApp #400

Merged
merged 58 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
62bfb03
Refactor workaround for setting Helm app `nameOverride`
disrupted Jan 2, 2024
55089be
Update snapshot assertions
disrupted Jan 2, 2024
8d9bb60
Use consistent naming for app values
disrupted Jan 2, 2024
934e4ed
Rename fixture
disrupted Jan 2, 2024
1c4e313
Refactor
disrupted Jan 2, 2024
2309a7b
Rename
disrupted Jan 2, 2024
7c8ff89
Merge remote-tracking branch 'origin/v3' into refactor/helm-nameoverride
disrupted Jan 2, 2024
f67a23f
Refactor streams-bootstrap cleanup
disrupted Jan 2, 2024
0578cb0
Update tests
disrupted Jan 2, 2024
44195e7
Inherit from common streams-bootstrap app
disrupted Jan 2, 2024
a2a8418
Remove desc
disrupted Jan 2, 2024
3457d1f
Fix failing hooks
disrupted Jan 2, 2024
6ac39da
Merge remote-tracking branch 'origin/v3' into refactor/streams-bootst…
disrupted Jan 2, 2024
a725f03
Cosmetic
disrupted Jan 2, 2024
26ac7f9
Cosmetic
disrupted Jan 2, 2024
b193c03
Rename run to clean
disrupted Jan 2, 2024
9d27adc
Fix returned object type
disrupted Jan 3, 2024
c517a3b
Add pydocs and todo
disrupted Jan 3, 2024
10be5f4
Refactor Kafka Connector resetter into separate HelmApp
disrupted Jan 3, 2024
e5bd0da
Update tests
disrupted Jan 3, 2024
cb135bd
Fix resetter instantiation
disrupted Jan 3, 2024
0bb3e6b
Fix reset & refactor suffix handling
disrupted Jan 3, 2024
8467027
Update test
disrupted Jan 3, 2024
10db07d
Update test
disrupted Jan 3, 2024
6a04113
Refactor test using pytest fixture
disrupted Jan 3, 2024
7ad4f27
Update tests
disrupted Jan 3, 2024
11dcc1e
Create streams-bootstrap base for all components based on its Helm
disrupted Jan 3, 2024
d9786dd
Cleanup KPOps components import
disrupted Jan 3, 2024
a49b551
Fix inheritance order of streams-boostrap apps
disrupted Jan 3, 2024
2e2c7ba
Update components hierarchy diagram
disrupted Jan 3, 2024
02c0ef7
Fix docs
disrupted Jan 4, 2024
ffda04c
Merge branch 'refactor/streams-bootstrap-cleanup' into refactor/kafka…
disrupted Jan 4, 2024
70cf1f3
Merge branch 'v3' into refactor/streams-bootstrap-cleanup
disrupted Jan 8, 2024
83ac398
Update defaults schema
disrupted Jan 8, 2024
45c58d8
Add docs for streams-bootstrap
disrupted Jan 8, 2024
7c92421
Add docs for streams-bootstrap
disrupted Jan 8, 2024
a5b621b
Merge remote-tracking branch 'origin/v3' into refactor/streams-bootst…
disrupted Jan 11, 2024
916c239
Update schema
disrupted Jan 11, 2024
e8fc934
Merge remote-tracking branch 'origin/v3' into refactor/streams-bootst…
disrupted Jan 11, 2024
2015c64
Merge branch 'refactor/streams-bootstrap-cleanup' into refactor/kafka…
disrupted Jan 11, 2024
125c20f
Rename variable in test
disrupted Jan 11, 2024
059632e
Cover breaking change in migration guide
disrupted Jan 11, 2024
a19fc74
Fix parameter name
disrupted Jan 11, 2024
09f9f35
Cover Kafka Connector change in migration guide
disrupted Jan 11, 2024
2169ea7
Merge branch 'refactor/streams-bootstrap-cleanup' into refactor/kafka…
disrupted Jan 11, 2024
ec7cdca
Make resetter_namespace an optional override
disrupted Jan 11, 2024
dd905b1
Update ATM fraud example pipeline
disrupted Jan 11, 2024
3552e27
Update defaults
disrupted Jan 11, 2024
6c16d7a
Merge remote-tracking branch 'origin/v3' into refactor/kafka-connecto…
disrupted Jan 15, 2024
bdd260b
Improve docs for Kubernetes namespace attribute
disrupted Jan 15, 2024
0ea5cec
Fix "enriching component of type `None`" log
disrupted Jan 15, 2024
f9aae5c
Delete unused method
disrupted Jan 16, 2024
5e44074
Extract consts in tests
disrupted Jan 16, 2024
3bd1f0d
Rename kafka-sink-connector defaults in test
disrupted Jan 16, 2024
9ea5459
Add paragraph to migration guide
disrupted Jan 16, 2024
ad6ee5b
Move name to pipeline-component defaults
disrupted Jan 16, 2024
3e5d5b9
Fix test
disrupted Jan 16, 2024
605c375
Refactor test
disrupted Jan 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,7 @@ kafka-connector.yaml:
- prefix.yaml
- from_.yaml
- to.yaml
- namespace.yaml
- app-kafka-connector.yaml
- repo_config-kafka-connector.yaml
- version-kafka-connector.yaml
- resetter_values.yaml
kafka-sink-connector.yaml: []
kafka-source-connector.yaml:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,30 +19,24 @@ kpops_components_fields:
- prefix
- from_
- to
- namespace
- app
- repo_config
- version
- resetter_namespace
- resetter_values
kafka-sink-connector:
- name
- prefix
- from_
- to
- namespace
- app
- repo_config
- version
- resetter_namespace
- resetter_values
kafka-source-connector:
- name
- prefix
- from_
- to
- namespace
- app
- repo_config
- version
- resetter_namespace
- resetter_values
- offset_topic
kubernetes-app:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,28 +15,19 @@ kafka-connector.yaml:
- prefix.yaml
- from_.yaml
- to.yaml
- namespace.yaml
- app-kafka-connector.yaml
- repo_config-kafka-connector.yaml
- version-kafka-connector.yaml
- resetter_values.yaml
kafka-sink-connector.yaml:
- prefix.yaml
- from_.yaml
- to.yaml
- namespace.yaml
- app-kafka-connector.yaml
- repo_config-kafka-connector.yaml
- version-kafka-connector.yaml
- resetter_values.yaml
kafka-source-connector.yaml:
- prefix.yaml
- from_-kafka-source-connector.yaml
- to.yaml
- namespace.yaml
- app-kafka-connector.yaml
- repo_config-kafka-connector.yaml
- version-kafka-connector.yaml
- resetter_values.yaml
- offset_topic-kafka-source-connector.yaml
kubernetes-app.yaml:
Expand Down
11 changes: 0 additions & 11 deletions docs/docs/resources/pipeline-components/kafka-connector.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,22 +42,11 @@
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down
11 changes: 0 additions & 11 deletions docs/docs/resources/pipeline-components/kafka-sink-connector.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,22 +43,11 @@
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,22 +24,11 @@
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down
22 changes: 0 additions & 22 deletions docs/docs/resources/pipeline-components/pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -160,22 +160,11 @@
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down Expand Up @@ -206,22 +195,11 @@
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,22 +45,11 @@ kafka-connector:
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down
11 changes: 0 additions & 11 deletions docs/docs/resources/pipeline-defaults/defaults.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -121,22 +121,11 @@ kafka-connector:
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` contains application-specific settings, hence it does not have a rigid
# structure. The fields below are just an example. Extensive documentation on
# connectors: https://kafka.apache.org/documentation/#connectconfigs
app: # required
tasks.max: 1
# Helm repository configuration for resetter
repo_config:
repository_name: my-repo # required
url: https://bakdata.github.io/kafka-connect-resetter/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "1.0.6" # Helm chart version
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
Expand Down
Loading