Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix substitution #449

Merged
merged 52 commits into from
Feb 8, 2024
Merged
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
f71da2a
test: minimal example
sujuka99 Feb 5, 2024
c59efee
Improve tests
sujuka99 Feb 6, 2024
0d3eb25
add much more minimal example
sujuka99 Feb 6, 2024
634ed0c
fix eof
sujuka99 Feb 6, 2024
4e48966
fix: substitution timing WIP
sujuka99 Feb 6, 2024
35fd009
Test using Python API
disrupted Feb 6, 2024
e7722ed
Run the actual assertion
disrupted Feb 6, 2024
b0e029c
Fix assertions
disrupted Feb 6, 2024
1ff50e0
Implement fix
disrupted Feb 6, 2024
169110e
Cosmetic
disrupted Feb 6, 2024
2737a18
Fix pipeline_{environment}.yaml override
disrupted Feb 6, 2024
1157aec
Fix double schema registry url
disrupted Feb 6, 2024
052a7c2
Try model_dump without `by_alias=True`
disrupted Feb 6, 2024
d0cbbbc
Fix Ruff diagnostics
disrupted Feb 6, 2024
df84620
Fix schema generation
disrupted Feb 6, 2024
986ab45
Fix Ruff
disrupted Feb 6, 2024
d91787d
Fix substitution var notation in tests
disrupted Feb 6, 2024
04ac93e
Update test
disrupted Feb 6, 2024
5c89efe
Refactor
disrupted Feb 6, 2024
48459ee
Improve performance
disrupted Feb 6, 2024
2e5f543
Mark todo
disrupted Feb 6, 2024
f49fcef
Fix KafkaConnector test
disrupted Feb 7, 2024
588d668
Revert 1 commits
disrupted Feb 7, 2024
fde7a54
Refactor & mark todo
disrupted Feb 7, 2024
52678b2
Run substitute twice to fix test
disrupted Feb 7, 2024
c9aefd3
Improve typing
disrupted Feb 7, 2024
29a23bf
Clear KPOps env for component tests
disrupted Feb 7, 2024
a19949a
Revert removal of `by_alias=True`
disrupted Feb 7, 2024
0037dba
Fix duplicate schema_registry_url / schemaRegistryUrl in model dump
disrupted Feb 7, 2024
01a4070
Fix order of pipeline steps for clean/reset
disrupted Feb 7, 2024
3b5f494
Merge branch 'fix/clean-order' into fix/inflate-substitution
disrupted Feb 7, 2024
a68f24a
Include Kafka app cleaner in generate dump
disrupted Feb 7, 2024
536c83f
Fix recursion error
disrupted Feb 7, 2024
57eb78e
Merge branch 'feat/cleaner-computed-property' into fix/inflate-substi…
disrupted Feb 7, 2024
6a1f153
Fix validation too early
disrupted Feb 7, 2024
c39d636
Update snapshots
disrupted Feb 7, 2024
47b27c1
Cleanup arguments
disrupted Feb 7, 2024
d7709f1
Rename tmp self
disrupted Feb 7, 2024
d108fd1
Cleanup
disrupted Feb 7, 2024
d4efc48
Remove from & to section from Kafka app cleaner
disrupted Feb 7, 2024
c6bfbbc
Update snapshot
disrupted Feb 7, 2024
9283043
Remove from & to section from Connector resetter
disrupted Feb 7, 2024
c76f072
Update snapshots
disrupted Feb 7, 2024
6b2a56c
Revert 4 commits
disrupted Feb 7, 2024
c7f1868
Cosmetic
disrupted Feb 7, 2024
cbd1172
remove duplicated substitution function
sujuka99 Feb 8, 2024
cfc7676
Remove redundant test.
sujuka99 Feb 8, 2024
3c9bf3e
Unused import
sujuka99 Feb 8, 2024
f4c6339
Merge remote-tracking branch 'origin/main' into fix/inflate-substitution
sujuka99 Feb 8, 2024
42b6d46
Remove leftover comments.
sujuka99 Feb 8, 2024
2d83e7f
Merge branch 'main' into fix/inflate-substitution
disrupted Feb 8, 2024
094c714
Fix custom component name
sujuka99 Feb 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add much more minimal example
sujuka99 committed Feb 6, 2024
commit 0d3eb253de78441912912bcb9af0e597a970b000
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- type: kafka-sink-connector
name: es-sink-connector
app:
connector.class: io.confluent.connect.elasticsearch.ElasticsearchSinkConnector

18 changes: 18 additions & 0 deletions tests/pipeline/test_generate.py
Original file line number Diff line number Diff line change
@@ -867,3 +867,21 @@ def test_substitution_in_inflated_component(self):
enriched_pipeline[1]["_resetter"]["app"]["label"]
== "inflated-connector-name"
)

def test_substitution_in_resetter(self):
result = runner.invoke(
app,
[
"generate",
str(RESOURCE_PATH / "resetter_values/pipeline_connector_only.yaml"),
"--defaults",
str(RESOURCE_PATH / "resetter_values"),
],
catch_exceptions=False,
)
assert result.exit_code == 0, result.stdout
enriched_pipeline: list = yaml.safe_load(result.stdout)
assert (
enriched_pipeline[0]["_resetter"]["app"]["label"]
== "inflated-connector-name"
)

Unchanged files with check annotations Beta

from tests.pipeline.test_components.components import (

Check failure on line 1 in tests/pipeline/test_components/__init__.py

GitHub Actions / Test (ubuntu-22.04, 3.10)

[*] Import block is un-sorted or un-formatted
Converter,
Filter,
ScheduledProducer,
import abc

Check failure on line 1 in tests/pipeline/test_components/components.py

GitHub Actions / Test (ubuntu-22.04, 3.10)

[*] Import block is un-sorted or un-formatted

Check failure on line 1 in tests/pipeline/test_components/components.py

GitHub Actions / Test (ubuntu-22.04, 3.10)

[*] `abc` imported but unused
from typing import Any

Check failure on line 2 in tests/pipeline/test_components/components.py

GitHub Actions / Test (ubuntu-22.04, 3.10)

[*] `typing.Any` imported but unused
from pydantic import BaseModel

Check failure on line 3 in tests/pipeline/test_components/components.py

GitHub Actions / Test (ubuntu-22.04, 3.10)

[*] `pydantic.BaseModel` imported but unused
from schema_registry.client.schema import AvroSchema
from typing_extensions import override
TopicConfig,
ToSection,
)
from kpops.components.streams_bootstrap.streams.model import StreamsAppValues

Check failure on line 23 in tests/pipeline/test_components/components.py

GitHub Actions / Test (ubuntu-22.04, 3.10)

[*] `kpops.components.streams_bootstrap.streams.model.StreamsAppValues` imported but unused
class ScheduledProducer(ProducerApp):