Auto generation happens mostly with pre-commit hooks. You can find the pre-commit configuration here. These pre-commit hooks call different Python scripts to auto generate code for the documentation.
Auto generation happens mostly with Git hooks. You can find the lefthook configuration here. These pre-commit hooks call different Python scripts to auto generate code for the documentation.
This will fetch the resources under the examples folder.
We advise that you stick to our pre-commit hooks for code linting, formatting, and auto-generation of documentation. After you install them using pre-commit install they're triggered automatically during git commit. Additionally, you can manually invoke them with pre-commit run -a. In order for dprint to work, you have to manually install it locally. It will work in the CI, so it is also possible to manually carry out formatting changes flagged by dprint in the CI and skip installing it locally.
To ensure a consistent Python code style, we use Ruff for both linting and formatting. The official docs contain a guide on editor integration.
-Our configuration can be found in KPOps' top-level pyproject.toml.
To ensure a consistent markdown style, we use dprint's Markdown code formatter. Our configuration can be found here.
-To ensure a consistent CSS style, we use the malva dprint's plugin. Our configuration can be found here.
-To ensure a consistent TOML style, we use dprint's TOML code formatter. Our configuration can be found here.
+We advise that you stick to our Git hooks for code linting, formatting, and auto-generation of documentation. After you install them using lefthook install they're triggered automatically during git operations, such as commit or checkout. Additionally, you can manually invoke them with lefthook run pre-commit --all-files. Please also install the dprint formatter.
clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nClean pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Clean pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def clean_runner(component: PipelineComponent):\n log_action(\"Clean\", component)\n await component.clean(dry_run)\n\n async def async_clean():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(clean_runner, reverse=True)\n await pipeline_tasks\n else:\n for component in reversed(pipeline.components):\n await clean_runner(component)\n\n asyncio.run(async_clean())\ndeploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nDeploy pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def deploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Deploy pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def deploy_runner(component: PipelineComponent):\n log_action(\"Deploy\", component)\n await component.deploy(dry_run)\n\n async def async_deploy():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(deploy_runner)\n await pipeline_tasks\n else:\n for component in pipeline.components:\n await deploy_runner(component)\n\n asyncio.run(async_deploy())\ndestroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nDestroy pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def destroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Destroy pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def destroy_runner(component: PipelineComponent):\n log_action(\"Destroy\", component)\n await component.destroy(dry_run)\n\n async def async_destroy():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(\n destroy_runner, reverse=True\n )\n await pipeline_tasks\n else:\n for component in reversed(pipeline.components):\n await destroy_runner(component)\n\n asyncio.run(async_destroy())\ngenerate(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = False,\n operation_mode: OperationMode = OperationMode.MANAGED,\n) -> Pipeline\nGenerate enriched pipeline representation.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: False
operation_mode How KPOps should operate.
TYPE: OperationMode DEFAULT: MANAGED
Pipeline Generated Pipeline object.
kpops/api/__init__.py def generate(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = False,\n operation_mode: OperationMode = OperationMode.MANAGED,\n) -> Pipeline:\n \"\"\"Generate enriched pipeline representation.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param operation_mode: How KPOps should operate.\n :return: Generated `Pipeline` object.\n \"\"\"\n kpops_config = KpopsConfig.create(\n config, dotenv, environment, verbose, operation_mode\n )\n pipeline = _create_pipeline(pipeline_path, kpops_config, environment)\n log.info(f\"Picked up pipeline '{pipeline_path.parent.name}'\")\n if steps:\n component_names = steps\n log.debug(\n f\"KPOPS_PIPELINE_STEPS is defined with values: {component_names} and filter type of {filter_type.value}\"\n )\n\n predicate = filter_type.create_default_step_names_filter_predicate(\n component_names\n )\n pipeline.filter(predicate)\n log.info(f\"Filtered pipeline:\\n{pipeline.step_names}\")\n return pipeline\ninit(path: Path, config_include_optional: bool = False)\nInitiate a default empty project.
PARAMETER DESCRIPTIONpath Directory in which the project should be initiated.
TYPE: Path
config_include_optional Whether to include non-required settings in the generated config file.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def init(\n path: Path,\n config_include_optional: bool = False,\n):\n \"\"\"Initiate a default empty project.\n\n :param path: Directory in which the project should be initiated.\n :param config_include_optional: Whether to include non-required settings\n in the generated config file.\n \"\"\"\n if not path.exists():\n path.mkdir(parents=False)\n elif next(path.iterdir(), False):\n log.warning(\"Please provide a path to an empty directory.\")\n return\n init_project(path, config_include_optional)\nmanifest_clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_clean()\n yield resource\nmanifest_deploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_deploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_deploy()\n yield resource\nmanifest_destroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_destroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_destroy()\n yield resource\nmanifest_reset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_reset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_reset()\n yield resource\nreset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nReset pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def reset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Reset pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def reset_runner(component: PipelineComponent):\n log_action(\"Reset\", component)\n await component.reset(dry_run)\n\n async def async_reset():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(reset_runner, reverse=True)\n await pipeline_tasks\n else:\n for component in reversed(pipeline.components):\n await reset_runner(component)\n\n asyncio.run(async_reset())\n Bases: BaseModel
Pipeline representation.
Source code inkpops/pipeline/__init__.py class Pipeline(BaseModel):\n \"\"\"Pipeline representation.\"\"\"\n\n _component_index: dict[str, PipelineComponent] = {}\n _graph: nx.DiGraph = nx.DiGraph()\n\n model_config = ConfigDict(arbitrary_types_allowed=True)\n\n @property\n def step_names(self) -> list[str]:\n return [step.name for step in self.components]\n\n @computed_field(title=\"Components\")\n @property\n def components(self) -> list[SerializeAsAny[PipelineComponent]]:\n return list(self._component_index.values())\n\n @property\n def last(self) -> PipelineComponent:\n return self.components[-1]\n\n def add(self, component: PipelineComponent) -> None:\n if self._component_index.get(component.id) is not None:\n msg = (\n f\"Pipeline steps must have unique id, '{component.id}' already exists.\"\n )\n raise ValidationError(msg)\n self._component_index[component.id] = component\n self.__add_to_graph(component)\n\n def remove(self, component_id: str) -> None:\n self._component_index.pop(component_id)\n\n def get(self, component_id: str) -> PipelineComponent | None:\n return self._component_index.get(component_id)\n\n def find(self, predicate: ComponentFilterPredicate) -> Iterator[PipelineComponent]:\n \"\"\"Find pipeline components matching a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n :returns: Iterator of components matching the predicate\n \"\"\"\n for component in self.components:\n if predicate(component):\n yield component\n\n def filter(self, predicate: ComponentFilterPredicate) -> None:\n \"\"\"Filter pipeline components using a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n \"\"\"\n for component in self.components:\n # filter out components not matching the predicate\n if not predicate(component):\n self.remove(component.id)\n\n def validate(self) -> None: # pyright: ignore [reportIncompatibleMethodOverride]\n self.__validate_graph()\n\n def to_yaml(self) -> str:\n return yaml.dump(\n self.model_dump(mode=\"json\", by_alias=True, exclude_none=True)[\n \"components\"\n ],\n Dumper=CustomSafeDumper,\n )\n\n def build_execution_graph(\n self,\n runner: Callable[[PipelineComponent], Coroutine[Any, Any, None]],\n /,\n reverse: bool = False,\n ) -> Awaitable[None]:\n async def run_parallel_tasks(\n coroutines: list[Coroutine[Any, Any, None]],\n ) -> None:\n tasks: list[asyncio.Task[None]] = []\n for coro in coroutines:\n tasks.append(asyncio.create_task(coro))\n await asyncio.gather(*tasks)\n\n async def run_graph_tasks(pending_tasks: list[Awaitable[None]]) -> None:\n for pending_task in pending_tasks:\n await pending_task\n\n graph: nx.DiGraph = self._graph.copy() # pyright: ignore[reportAssignmentType, reportGeneralTypeIssues] imprecise type hint in networkx\n\n # We add an extra node to the graph, connecting all the leaf nodes to it\n # in that way we make this node the root of the graph, avoiding backtracking\n root_node = \"root_node_bfs\"\n graph.add_node(root_node)\n\n for node in graph:\n predecessors = list(graph.predecessors(node))\n if not predecessors:\n graph.add_edge(root_node, node)\n\n layers_graph: list[list[str]] = list(nx.bfs_layers(graph, root_node))\n\n sorted_tasks: list[Awaitable[None]] = []\n for layer in layers_graph[1:]:\n if parallel_tasks := self.__get_parallel_tasks_from(layer, runner):\n sorted_tasks.append(run_parallel_tasks(parallel_tasks))\n\n if reverse:\n sorted_tasks.reverse()\n\n return run_graph_tasks(sorted_tasks)\n\n def __getitem__(self, component_id: str) -> PipelineComponent:\n try:\n return self._component_index[component_id]\n except KeyError as exc:\n msg = f\"Component {component_id} not found\"\n raise ValueError(msg) from exc\n\n def __bool__(self) -> bool:\n return bool(self._component_index)\n\n def __iter__(self) -> Iterator[PipelineComponent]: # pyright: ignore [reportIncompatibleMethodOverride]\n yield from self._component_index.values()\n\n def __len__(self) -> int:\n return len(self.components)\n\n def __add_to_graph(self, component: PipelineComponent):\n self._graph.add_node(component.id)\n\n for input_topic in component.inputs:\n self.__add_input(input_topic.id, component.id)\n\n for output_topic in component.outputs:\n self.__add_output(output_topic.id, component.id)\n\n def __add_output(self, topic_id: str, source: str) -> None:\n self._graph.add_node(topic_id)\n self._graph.add_edge(source, topic_id)\n\n def __add_input(self, topic_id: str, target: str) -> None:\n self._graph.add_node(topic_id)\n self._graph.add_edge(topic_id, target)\n\n def __get_parallel_tasks_from(\n self,\n layer: list[str],\n runner: Callable[[PipelineComponent], Coroutine[Any, Any, None]],\n ) -> list[Coroutine[Any, Any, None]]:\n def gen_parallel_tasks():\n for node_in_layer in layer:\n # check if component, skip topics\n if (component := self._component_index.get(node_in_layer)) is not None:\n yield runner(component)\n\n return list(gen_parallel_tasks())\n\n def __validate_graph(self) -> None:\n if not nx.is_directed_acyclic_graph(self._graph):\n msg = \"Pipeline is not a valid DAG.\"\n raise ValueError(msg)\nproperty", "text": "components: list[SerializeAsAny[PipelineComponent]]\nproperty", "text": "last: PipelineComponent\nproperty", "text": "step_names: list[str]\nadd(component: PipelineComponent) -> None\nkpops/pipeline/__init__.py def add(self, component: PipelineComponent) -> None:\n if self._component_index.get(component.id) is not None:\n msg = (\n f\"Pipeline steps must have unique id, '{component.id}' already exists.\"\n )\n raise ValidationError(msg)\n self._component_index[component.id] = component\n self.__add_to_graph(component)\nbuild_execution_graph(\n runner: Callable[\n [PipelineComponent], Coroutine[Any, Any, None]\n ],\n /,\n reverse: bool = False,\n) -> Awaitable[None]\nkpops/pipeline/__init__.py def build_execution_graph(\n self,\n runner: Callable[[PipelineComponent], Coroutine[Any, Any, None]],\n /,\n reverse: bool = False,\n) -> Awaitable[None]:\n async def run_parallel_tasks(\n coroutines: list[Coroutine[Any, Any, None]],\n ) -> None:\n tasks: list[asyncio.Task[None]] = []\n for coro in coroutines:\n tasks.append(asyncio.create_task(coro))\n await asyncio.gather(*tasks)\n\n async def run_graph_tasks(pending_tasks: list[Awaitable[None]]) -> None:\n for pending_task in pending_tasks:\n await pending_task\n\n graph: nx.DiGraph = self._graph.copy() # pyright: ignore[reportAssignmentType, reportGeneralTypeIssues] imprecise type hint in networkx\n\n # We add an extra node to the graph, connecting all the leaf nodes to it\n # in that way we make this node the root of the graph, avoiding backtracking\n root_node = \"root_node_bfs\"\n graph.add_node(root_node)\n\n for node in graph:\n predecessors = list(graph.predecessors(node))\n if not predecessors:\n graph.add_edge(root_node, node)\n\n layers_graph: list[list[str]] = list(nx.bfs_layers(graph, root_node))\n\n sorted_tasks: list[Awaitable[None]] = []\n for layer in layers_graph[1:]:\n if parallel_tasks := self.__get_parallel_tasks_from(layer, runner):\n sorted_tasks.append(run_parallel_tasks(parallel_tasks))\n\n if reverse:\n sorted_tasks.reverse()\n\n return run_graph_tasks(sorted_tasks)\nfilter(predicate: ComponentFilterPredicate) -> None\nFilter pipeline components using a custom predicate.
PARAMETER DESCRIPTIONpredicate Filter function, returns boolean value whether the component should be kept or removed
TYPE: ComponentFilterPredicate
kpops/pipeline/__init__.py def filter(self, predicate: ComponentFilterPredicate) -> None:\n \"\"\"Filter pipeline components using a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n \"\"\"\n for component in self.components:\n # filter out components not matching the predicate\n if not predicate(component):\n self.remove(component.id)\nfind(\n predicate: ComponentFilterPredicate,\n) -> Iterator[PipelineComponent]\nFind pipeline components matching a custom predicate.
PARAMETER DESCRIPTIONpredicate Filter function, returns boolean value whether the component should be kept or removed
TYPE: ComponentFilterPredicate
Iterator[PipelineComponent] Iterator of components matching the predicate
Source code inkpops/pipeline/__init__.py def find(self, predicate: ComponentFilterPredicate) -> Iterator[PipelineComponent]:\n \"\"\"Find pipeline components matching a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n :returns: Iterator of components matching the predicate\n \"\"\"\n for component in self.components:\n if predicate(component):\n yield component\nget(component_id: str) -> PipelineComponent | None\nkpops/pipeline/__init__.py def get(self, component_id: str) -> PipelineComponent | None:\n return self._component_index.get(component_id)\nremove(component_id: str) -> None\nkpops/pipeline/__init__.py def remove(self, component_id: str) -> None:\n self._component_index.pop(component_id)\nto_yaml() -> str\nkpops/pipeline/__init__.py def to_yaml(self) -> str:\n return yaml.dump(\n self.model_dump(mode=\"json\", by_alias=True, exclude_none=True)[\n \"components\"\n ],\n Dumper=CustomSafeDumper,\n )\nvalidate() -> None\nkpops/pipeline/__init__.py def validate(self) -> None: # pyright: ignore [reportIncompatibleMethodOverride]\n self.__validate_graph()\nAuto generation happens mostly with pre-commit hooks. You can find the pre-commit configuration here. These pre-commit hooks call different Python scripts to auto generate code for the documentation.
cli_env_vars.env -- All CLI environment variables in a dotenv file.cli_env_vars.md -- All CLI environment variables in a table.config_env_vars.env -- Almost all pipeline config environment variables in a dotenv file. The script checks for each field in KpopsConfig whether it has an env attribute defined. The script is currently unable to visit the classes of fields like topic_name_config, hence any environment variables defined there would remain unknown to it.config_env_vars.env -- Almost all pipeline config environment variables in a table.variable_substitution.yaml -- A copy of ./tests/pipeline/resources/component-type-substitution/pipeline.yaml used as an example of substitution.Generated by typer-cli from the code in main.py. It is called with Python's subprocess module.
Generates example pipeline.yaml and defaults.yaml for each individual component, stores them and also concatenates them into 1 big pipeline definition and 1 big pipeline defaults definition.
User input
headers/*\\.yaml -- The top of each example. Includes a description comment, type and name. The headers for pipeline.yaml reside in the pipeline-components dir and the defaults.yaml headers reside in the pipeline-defaults dir. The names of the files must be equal to the respective component type.sections/*\\.yaml -- Each YAML file contains a single section (component attribute) definition. The intention is to keep the minimal set of definitions there from which any component definition can be built. The names of the files must be equal to the respective component type and the attribute name. The sections are used for both defaults.yaml and pipeline.yaml generation and reside in the pipeline-components dir.Generated
pipeline-components/dependencies/* Cached information about KPOps componentspipeline_component_dependencies.yaml -- Specifies per component which files in the sections dir should be used for the pipeline.yaml generation.defaults_pipeline_component_dependencies.yaml -- Specifies per component which files in the sections dir should be used for the defaults.yaml generation.kpops_structure.yaml -- Specifies the inheritance hierarchy of the components and what sections exist in each component.pipeline-components/*\\.yaml -- All single-component pipeline definitions and one big (complete) pipeline.yaml that contains all of them.pipeline-defaults/*\\.yaml -- All single-component defaults definitions and one big (complete) defaults.yaml that contains all of them.Welcome! We are glad to have you visit our contributing guide!
If you find any bugs or have suggestions for improvements, please open an issue and optionally a pull request (PR). In the case of a PR, we would appreciate it if you preface it with an issue outlining your goal and means of achieving it.
"}, {"location": "developer/contributing/#git", "title": "git", "text": "We are using git submodules to import the KPOps examples repository. You need to fetch the repository locally on your machine. To do so use this command:
git submodule init\ngit submodule update --recursive\nThis will fetch the resources under the examples folder.
We advise that you stick to our pre-commit hooks for code linting, formatting, and auto-generation of documentation. After you install them using pre-commit install they're triggered automatically during git commit. Additionally, you can manually invoke them with pre-commit run -a. In order for dprint to work, you have to manually install it locally. It will work in the CI, so it is also possible to manually carry out formatting changes flagged by dprint in the CI and skip installing it locally.
To ensure a consistent Python code style, we use Ruff for both linting and formatting. The official docs contain a guide on editor integration.
Our configuration can be found in KPOps' top-level pyproject.toml.
To ensure a consistent markdown style, we use dprint's Markdown code formatter. Our configuration can be found here.
"}, {"location": "developer/contributing/#css", "title": "CSS", "text": "To ensure a consistent CSS style, we use the malva dprint's plugin. Our configuration can be found here.
"}, {"location": "developer/contributing/#toml", "title": "TOML", "text": "To ensure a consistent TOML style, we use dprint's TOML code formatter. Our configuration can be found here.
"}, {"location": "developer/getting-started/", "title": "Getting started", "text": "Welcome! We are glad to have you visit our developer guide! If you find any bugs or have suggestions for improvements, please open an issue and optionally a pull request (PR). In the case of a PR, we would appreciate it if you preface it with an issue outlining your goal and means of achieving it.
Find more about our code-style or insights into KPOps' code base here in our developer guide.
Work in progress
The developer guide is still under construction. If you have a question left unanswered here, feel free to ask it by opening an issue.
"}, {"location": "user/changelog/", "title": "Changelog", "text": "All notable changes to this project will be documented in this file.
"}, {"location": "user/changelog/#921-2025-01-15", "title": "9.2.1 - 2025-01-15", "text": ""}, {"location": "user/changelog/#whats-changed", "title": "What's changed", "text": "Full Changelog: https://github.com/bakdata/kpops/compare/9.2.0-dev...9.2.1
"}, {"location": "user/changelog/#920-dev-2025-01-14", "title": "9.2.0-dev - 2025-01-14", "text": ""}, {"location": "user/changelog/#whats-changed_1", "title": "What's changed", "text": "Improve Pyright matcher by @disrupted in #579
Migrate from Poetry to uv by @disrupted in #578
Fix circular imports when running individual tests by @disrupted in #583
Configure Pyright to report import cycles by @disrupted in #585
Fix kpops package build by @disrupted in #588
Fail if streams-boostrap v3 model is instantiated with v2 attribute by @disrupted in #587
Bump version 9.1.0 \u2192 9.2.0-dev by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/9.1.0...9.2.0-dev
"}, {"location": "user/changelog/#910-2025-01-07", "title": "9.1.0 - 2025-01-07", "text": ""}, {"location": "user/changelog/#whats-changed_2", "title": "What's changed", "text": "Update CODEOWNERS by @raminqaf in #572
Update test components to streams-bootstrap v3 by @disrupted in #576
Silence deprecation warnings for streams-bootstrap v2 in tests by @disrupted in #577
Represent multiline strings using YAML block style by @disrupted in #574
Indent sequence items to follow style recommendations by @disrupted in #575
Bump version 9.0.1 \u2192 9.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/9.0.1...9.1.0
"}, {"location": "user/changelog/#901-2024-12-20", "title": "9.0.1 - 2024-12-20", "text": ""}, {"location": "user/changelog/#whats-changed_3", "title": "What's changed", "text": "Add operation-mode documentation to mkdocs index by @raminqaf in #573
Bump version 9.0.0 \u2192 9.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/9.0.0...9.0.1
"}, {"location": "user/changelog/#900-2024-12-20", "title": "9.0.0 - 2024-12-20", "text": ""}, {"location": "user/changelog/#whats-changed_4", "title": "What's changed", "text": "Merge main by @raminqaf
Add topic manifestation of ProducerApps for reset command by @raminqaf in #566
Add documentation for operation-mode in KPOps by @raminqaf in #565
Merge branch 'main' into v9 by @raminqaf
Merge branch 'v9' of github.com:bakdata/kpops into v9 by @raminqaf
Set Python target version to 3.11 by @disrupted
Hide operation_mode from KPOps config by @raminqaf in #571
Add migration guide v8-v9 by @raminqaf in #562
KPOps V9 by @raminqaf in #558
Bump version 8.4.0 \u2192 9.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.4.0...9.0.0
"}, {"location": "user/changelog/#840-2024-12-18", "title": "8.4.0 - 2024-12-18", "text": ""}, {"location": "user/changelog/#whats-changed_5", "title": "What's changed", "text": "Create generic SerializeAsOptional type for Pydantic by @disrupted in #564
Bump version 8.3.2 \u2192 8.4.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.3.2...8.4.0
"}, {"location": "user/changelog/#832-2024-12-17", "title": "8.3.2 - 2024-12-17", "text": ""}, {"location": "user/changelog/#whats-changed_6", "title": "What's changed", "text": "Fix allow optional resources requests and limits by @disrupted in #570
Bump version 8.3.1 \u2192 8.3.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.3.1...8.3.2
"}, {"location": "user/changelog/#831-2024-12-17", "title": "8.3.1 - 2024-12-17", "text": ""}, {"location": "user/changelog/#whats-changed_7", "title": "What's changed", "text": "Fix Kubernetes memory not accepting decimal values by @disrupted in #568
Add ephemeral storage to Kubernetes resource requests and limits by @disrupted in #569
Bump version 8.3.0 \u2192 8.3.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.3.0...8.3.1
"}, {"location": "user/changelog/#830-2024-12-17", "title": "8.3.0 - 2024-12-17", "text": ""}, {"location": "user/changelog/#whats-changed_8", "title": "What's changed", "text": "Merge branch 'main' into v9 by @raminqaf
Drop support for Python 3.10 by @disrupted in #561
Manifest Kubernetes resources for reset command by @raminqaf in #563
Add Kubernetes affinity and tolerations to streams-bootstrap v2 values by @disrupted in #567
Bump version 8.2.0 \u2192 8.3.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.2.0...8.3.0
"}, {"location": "user/changelog/#820-2024-12-12", "title": "8.2.0 - 2024-12-12", "text": ""}, {"location": "user/changelog/#whats-changed_9", "title": "What's changed", "text": "merge by @raminqaf
Manifest toSection with Strimzi KafkaTopic by @raminqaf in #545
Manifest Kubernetes resources for destroy command by @raminqaf in #552
Bump streams-bootstrap to 3.1.0 by @disrupted in #557
Merge branch 'main' into v9 by @raminqaf
Manifest Kubernetes resources for clean command by @raminqaf in #559
Update KPOps example snapshots and fix broken link to defaults.yaml by @raminqaf in #560
Merge branch 'main' into v9 by @raminqaf
Add Pydantic models for Kubernetes Affinity by @disrupted in #555
Bump version 8.1.4 \u2192 8.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.4...8.2.0
"}, {"location": "user/changelog/#814-2024-12-09", "title": "8.1.4 - 2024-12-09", "text": ""}, {"location": "user/changelog/#whats-changed_10", "title": "What's changed", "text": "Fix kpops --version by @disrupted in #551
Trim Helm name override for Producer CronJob to 52 characters by @disrupted in #550
Bump version 8.1.3 \u2192 8.1.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.3...8.1.4
"}, {"location": "user/changelog/#813-2024-12-05", "title": "8.1.3 - 2024-12-05", "text": ""}, {"location": "user/changelog/#whats-changed_11", "title": "What's changed", "text": "Merge branch 'main' of github.com:bakdata/kpops into v9 by @raminqaf
Remove repeated defaults from streams-bootstrap values by @disrupted in #547
Bump version 8.1.2 \u2192 8.1.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.2...8.1.3
"}, {"location": "user/changelog/#812-2024-12-04", "title": "8.1.2 - 2024-12-04", "text": ""}, {"location": "user/changelog/#whats-changed_12", "title": "What's changed", "text": "Introduce KPOps operation and manifest resources for deployment by @raminqaf in #541
Define Pydantic model to representing Kubernetes manifest by @raminqaf in #546
Convert all values of Kafka connector and topic config to string by @disrupted in #544
Bump version 8.1.1 \u2192 8.1.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.1...8.1.2
"}, {"location": "user/changelog/#811-2024-12-02", "title": "8.1.1 - 2024-12-02", "text": ""}, {"location": "user/changelog/#whats-changed_13", "title": "What's changed", "text": "Fix files field value type in Streamsboostrap component by @raminqaf in #542
Fix: Use enum values when dumping models by @raminqaf in #543
Bump version 8.1.0 \u2192 8.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.0...8.1.1
"}, {"location": "user/changelog/#810-2024-10-25", "title": "8.1.0 - 2024-10-25", "text": ""}, {"location": "user/changelog/#whats-changed_14", "title": "What's changed", "text": "Upgrade typer to support union types by @raminqaf in #533
Extend StreamsBootstrap model by @raminqaf in #534
Bump version 8.0.1 \u2192 8.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.0.1...8.1.0
"}, {"location": "user/changelog/#801-2024-08-22", "title": "8.0.1 - 2024-08-22", "text": ""}, {"location": "user/changelog/#whats-changed_15", "title": "What's changed", "text": "Fix changelog in docs by @raminqaf in #532
Bump version 8.0.0 \u2192 8.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.0.0...8.0.1
"}, {"location": "user/changelog/#800-2024-08-21", "title": "8.0.0 - 2024-08-21", "text": ""}, {"location": "user/changelog/#whats-changed_16", "title": "What's changed", "text": "Make KafkaApp responsible of deploying/cleaning streams bootstrap components (#522) by @raminqaf
Add support for streams-bootstrap v3 (#519) by @raminqaf
Rename role to label (#525) by @raminqaf
Fix Pyright warning about type override without default value (#524) by @disrupted
Remove v3 and suffix old streams bootstrap with v2 (#526) by @raminqaf
KPOps 8.0.0 by @raminqaf in #531
Bump version 7.1.0 \u2192 8.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/7.1.0...8.0.0
"}, {"location": "user/changelog/#710-2024-08-15", "title": "7.1.0 - 2024-08-15", "text": ""}, {"location": "user/changelog/#whats-changed_17", "title": "What's changed", "text": "Improve incomplete type hints by @disrupted in #515
Fallback to user defined model when the validation of cluster model fails by @raminqaf in #521
Fix incorrect parameter type annotation by @disrupted in #523
Update pytest by @disrupted in #527
Replace kubernetes-asyncio with lightkube by @disrupted in #517
Bump version 7.0.0 \u2192 7.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/7.0.0...7.1.0
"}, {"location": "user/changelog/#700-2024-07-23", "title": "7.0.0 - 2024-07-23", "text": ""}, {"location": "user/changelog/#whats-changed_18", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v7 by @disrupted
Call destroy from inside of reset or clean by @raminqaf in #501
clean/reset streams-bootstrap components with cluster values by @raminqaf in #498
Rename app field by @disrupted in #506
Fix circular dependency when running individual tests by @raminqaf
Add tests for global config & handlers by @disrupted
Update examples by @disrupted
Bump version 6.1.0 \u2192 7.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.1.0...7.0.0
"}, {"location": "user/changelog/#610-2024-07-09", "title": "6.1.0 - 2024-07-09", "text": ""}, {"location": "user/changelog/#whats-changed_19", "title": "What's changed", "text": "Add image tag field to streams-bootstrap app values by @raminqaf in #499
Automatic loading of namespaced custom components by @disrupted in #500
Improve dataclass instance check by @disrupted in #507
Delete ignored keys from diff by @disrupted in #510
Bump version 6.0.2 \u2192 6.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.0.2...6.1.0
"}, {"location": "user/changelog/#602-2024-07-04", "title": "6.0.2 - 2024-07-04", "text": ""}, {"location": "user/changelog/#whats-changed_20", "title": "What's changed", "text": "Update codeowners by @disrupted in #504
Generate developer docs for Python API by @sujuka99 in #503
Bump version 6.0.1 \u2192 6.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.0.1...6.0.2
"}, {"location": "user/changelog/#601-2024-06-12", "title": "6.0.1 - 2024-06-12", "text": ""}, {"location": "user/changelog/#whats-changed_21", "title": "What's changed", "text": "Fix connector resetter offset topic by @disrupted in #497
Bump version 6.0.0 \u2192 6.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.0.0...6.0.1
"}, {"location": "user/changelog/#600-2024-06-06", "title": "6.0.0 - 2024-06-06", "text": ""}, {"location": "user/changelog/#whats-changed_22", "title": "What's changed", "text": "KPOps 6.0.0 by @raminqaf in #496
Bump version 5.1.1 \u2192 6.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.1.1...6.0.0
"}, {"location": "user/changelog/#511-2024-05-22", "title": "5.1.1 - 2024-05-22", "text": ""}, {"location": "user/changelog/#whats-changed_23", "title": "What's changed", "text": "Add YAML separator (---) to stdout by @raminqaf in #491
Bump version 5.1.0 \u2192 5.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.1.0...5.1.1
"}, {"location": "user/changelog/#510-2024-05-22", "title": "5.1.0 - 2024-05-22", "text": ""}, {"location": "user/changelog/#whats-changed_24", "title": "What's changed", "text": "Add computed field for Helm release name and name override by @disrupted in #490
Bump version 5.0.1 \u2192 5.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.0.1...5.1.0
"}, {"location": "user/changelog/#501-2024-05-15", "title": "5.0.1 - 2024-05-15", "text": ""}, {"location": "user/changelog/#whats-changed_25", "title": "What's changed", "text": "Fix missing await on Kubernetes API by @raminqaf in #488
Bump version 5.0.0 \u2192 5.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.0.0...5.0.1
"}, {"location": "user/changelog/#500-2024-05-02", "title": "5.0.0 - 2024-05-02", "text": ""}, {"location": "user/changelog/#whats-changed_26", "title": "What's changed", "text": "Update examples for v4 by @disrupted in #486
Allow custom timeout for external services by @disrupted in #485
Bump version 4.2.1 \u2192 5.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.2.1...5.0.0
"}, {"location": "user/changelog/#421-2024-04-25", "title": "4.2.1 - 2024-04-25", "text": ""}, {"location": "user/changelog/#whats-changed_27", "title": "What's changed", "text": "Add support for cleaning StatefulSets with PVCs by @raminqaf in #482
Bump version 4.2.0 \u2192 4.2.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.2.0...4.2.1
"}, {"location": "user/changelog/#420-2024-04-25", "title": "4.2.0 - 2024-04-25", "text": ""}, {"location": "user/changelog/#whats-changed_28", "title": "What's changed", "text": "Update Ruff by @disrupted in #475
Improve type annotations for parallel pipeline jobs by @disrupted in #476
Set Pyright to warn on unknown types by @disrupted in #480
Quiet faker debug logs in tests by @disrupted in #483
Add pyright matcher by @sujuka99 in #481
Bump version 4.1.2 \u2192 4.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.1.2...4.2.0
"}, {"location": "user/changelog/#412-2024-03-11", "title": "4.1.2 - 2024-03-11", "text": ""}, {"location": "user/changelog/#whats-changed_29", "title": "What's changed", "text": "fix(docs): Correct from.components.<component-name>.type to input by @raminqaf in #473
Bump version 4.1.1 \u2192 4.1.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.1.1...4.1.2
"}, {"location": "user/changelog/#411-2024-03-11", "title": "4.1.1 - 2024-03-11", "text": ""}, {"location": "user/changelog/#whats-changed_30", "title": "What's changed", "text": "Update httpx by @disrupted in #471
Fix import errors by @sujuka99 in #472
Bump version 4.1.0 \u2192 4.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.1.0...4.1.1
"}, {"location": "user/changelog/#410-2024-03-07", "title": "4.1.0 - 2024-03-07", "text": ""}, {"location": "user/changelog/#whats-changed_31", "title": "What's changed", "text": "Document precedence between env vars and config.yaml by @JakobEdding in #465
Create init command by @sujuka99 in #394
Bump version 4.0.2 \u2192 4.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.0.2...4.1.0
"}, {"location": "user/changelog/#402-2024-03-04", "title": "4.0.2 - 2024-03-04", "text": ""}, {"location": "user/changelog/#whats-changed_32", "title": "What's changed", "text": "Add support for Python 3.12 by @disrupted in #467
Update Pyright by @disrupted in #468
Remove package classifiers that are automatically assigned by Poetry by @disrupted in #469
Reference editor plugin for Neovim in docs by @disrupted in #464
Validate autoscaling mandatory fields when enabled by @raminqaf in #470
Bump version 4.0.1 \u2192 4.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.0.1...4.0.2
"}, {"location": "user/changelog/#401-2024-02-29", "title": "4.0.1 - 2024-02-29", "text": ""}, {"location": "user/changelog/#whats-changed_33", "title": "What's changed", "text": "Set supported Python cutoff to 3.11 by @disrupted in #466
Bump version 4.0.0 \u2192 4.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.0.0...4.0.1
"}, {"location": "user/changelog/#400-2024-02-27", "title": "4.0.0 - 2024-02-27", "text": ""}, {"location": "user/changelog/#whats-changed_34", "title": "What's changed", "text": "Distribute defaults across multiple files by @raminqaf in #438
Bump version 3.2.4 \u2192 4.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.4...4.0.0
"}, {"location": "user/changelog/#324-2024-02-26", "title": "3.2.4 - 2024-02-26", "text": ""}, {"location": "user/changelog/#whats-changed_35", "title": "What's changed", "text": "Refactor Kafka topics by @disrupted in #447
Fix docs CI to include the latest changes to a tagged version in the changelog by @sujuka99 in #459
Refactor PipelineGenerator to use component ids by @disrupted in #460
Fix tempfile creation by @sujuka99 in #461
Fix symbolic link to CONTRIBUTING.md and parallel option in action.yaml by @raminqaf in #462
Bump version 3.2.3 \u2192 3.2.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.3...3.2.4
"}, {"location": "user/changelog/#323-2024-02-19", "title": "3.2.3 - 2024-02-19", "text": ""}, {"location": "user/changelog/#whats-changed_36", "title": "What's changed", "text": "Trim and hash Helm name override to 63 characters by @disrupted in #456
Bump version 3.2.2 \u2192 3.2.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.2...3.2.3
"}, {"location": "user/changelog/#322-2024-02-12", "title": "3.2.2 - 2024-02-12", "text": ""}, {"location": "user/changelog/#whats-changed_37", "title": "What's changed", "text": "Fix nested substitution by @sujuka99 in #451
Bump version 3.2.1 \u2192 3.2.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.1...3.2.2
"}, {"location": "user/changelog/#321-2024-02-08", "title": "3.2.1 - 2024-02-08", "text": ""}, {"location": "user/changelog/#whats-changed_38", "title": "What's changed", "text": "Simplify execution graph logic by @disrupted in #446
Fix order of pipeline steps for clean/reset by @disrupted in #450
Fix substitution by @sujuka99 in #449
Fix cleaner inheritance, parent model should be aliased during instantiation by @disrupted in #452
Bump version 3.2.0 \u2192 3.2.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.0...3.2.1
"}, {"location": "user/changelog/#320-2024-02-01", "title": "3.2.0 - 2024-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_39", "title": "What's changed", "text": "Improve Sphinx docs highlighting using RST markup by @disrupted in #443
Refactor enrichment using Pydantic model validator by @disrupted in #444
Refactor pipeline filter and add to public API by @disrupted in #405
Bump version 3.1.0 \u2192 3.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.1.0...3.2.0
"}, {"location": "user/changelog/#310-2024-01-30", "title": "3.1.0 - 2024-01-30", "text": ""}, {"location": "user/changelog/#whats-changed_40", "title": "What's changed", "text": "Simplify loading of defaults by @disrupted in #435
Update poetry publish workflow version to latest by @raminqaf in #430
Add support for pipeline steps parallelization by @irux in #312
Add custom PascalCase to snake_case alias generator by @disrupted in #436
Add parallel flag support to kpops runner by @irux in #439
Bump version 3.0.2 \u2192 3.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.0.2...3.1.0
"}, {"location": "user/changelog/#302-2024-01-23", "title": "3.0.2 - 2024-01-23", "text": ""}, {"location": "user/changelog/#whats-changed_41", "title": "What's changed", "text": "Add step for submodule initialization on the docs by @irux in #431
Add message if examples git submodule is not initialized by @disrupted in #432
Update type annotation for deserialized pipeline by @disrupted in #433
Fix Helm diff output by @disrupted in #434
Bump version 3.0.1 \u2192 3.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.0.1...3.0.2
"}, {"location": "user/changelog/#301-2024-01-19", "title": "3.0.1 - 2024-01-19", "text": ""}, {"location": "user/changelog/#whats-changed_42", "title": "What's changed", "text": "Update pydantic dependency by @sujuka99 in #422
Update docs of word-count example for v3 & new folder structure by @disrupted in #423
Move ATM fraud to examples repo by @disrupted in #425
Fix broken doc link by @raminqaf in #427
Add warning log if SR handler is disabled but URL is set by @raminqaf in #428
Add git submodule instructions to the contributing.md by @raminqaf in #429
Bump version 3.0.0 \u2192 3.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.0.0...3.0.1
"}, {"location": "user/changelog/#300-2024-01-17", "title": "3.0.0 - 2024-01-17", "text": ""}, {"location": "user/changelog/#whats-changed_43", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v3 by @disrupted
Fix test by @disrupted
Add missing HelmApp docs by @disrupted
Replace black with ruff by @sujuka99 in #365
Add toml formatter to dprint by @sujuka99 in #386
Add malva to dprint by @sujuka99 in #385
Merge branch 'main' of github.com:bakdata/kpops into v3 by @raminqaf
Migrate to Pydantic v2 by @sujuka99 in #347
Allow overriding config files by @sujuka99 in #391
Change substitution variables separator to . by @sujuka99 in #388
Refactor pipeline generator & representation by @disrupted in #392
Define custom components module & pipeline base dir globally by @disrupted in #387
Update KPOps runner with the new options by @raminqaf in #395
Add steps for KubernetesApp->HelmApp to migration guide by @disrupted
Fix KPOps action to get package from testPyPI by @raminqaf in #396
Use hash and trim long Helm release names instead of only trimming by @raminqaf in #390
Refactor Helm nameOverride by @disrupted in #397
Mark component type as computed Pydantic field by @disrupted in #399
Fix missing component type in pipeline schema by @disrupted in #401
Refactor generate template for Python API usage by @disrupted in #380
Generate defaults schema by @disrupted in #402
Update docs for substitution variable usage in v3 by @sujuka99 in #409
Namespace substitution vars by @sujuka99 in #408
Support multiple inheritance for doc generation by @sujuka99 in #406
Refactor streams-bootstrap cleanup jobs as individual HelmApp by @disrupted in #398
Update docs for v3 by @sujuka99 in #416
Refactor Kafka Connector resetter as individual HelmApp by @disrupted in #400
Update tests resources by @sujuka99 in #417
Fix enrichment of nested Pydantic BaseModel by @disrupted in #415
Summarize all breaking changes in diffs at the top of the migration guide by @sujuka99 in #419
Fix wrong Helm release name character limit by @disrupted in #418
KPOps 3.0 by @disrupted in #420
Update release workflow template to support custom changelog file path by @disrupted in #421
Bump version 2.0.11 \u2192 3.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.11...3.0.0
"}, {"location": "user/changelog/#2011-2023-10-24", "title": "2.0.11 - 2023-10-24", "text": ""}, {"location": "user/changelog/#whats-changed_44", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v3 by @disrupted
Create HelmApp component by @disrupted in #370
Fix early exit upon Helm exit code 1 (#376) by @sujuka99
Migrate deprecated mkdocs-material-extensions (#378) by @disrupted
Fix docs setup page list indentation (#377) by @sujuka99
Exclude resources from docs search (#371) by @disrupted
Bump version 2.0.10 \u2192 2.0.11 by @bakdata-bot
Fix early exit upon Helm exit code 1 by @sujuka99 in #376
Migrate deprecated mkdocs-material-extensions by @disrupted in #378
Fix docs setup page list indentation by @sujuka99 in #377
Exclude resources from docs search by @disrupted in #371
Bump version 2.0.10 \u2192 2.0.11 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.10...2.0.11
"}, {"location": "user/changelog/#2010-2023-10-12", "title": "2.0.10 - 2023-10-12", "text": ""}, {"location": "user/changelog/#whats-changed_45", "title": "What's changed", "text": "Fix environment variables documentation generation by @sujuka99 in #362
Merge branch 'main' of github.com:bakdata/kpops into v3 by @raminqaf
Make Kafka REST Proxy & Kafka Connect hosts default and improve Schema Registry config by @raminqaf in #354
Introduce ruff by @sujuka99 in #363
Print details on connector name mismatch error by @disrupted in #369
Enable transparent OS environment lookups from internal environment by @disrupted in #368
Bump version 2.0.9 \u2192 2.0.10 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.9...2.0.10
"}, {"location": "user/changelog/#209-2023-09-19", "title": "2.0.9 - 2023-09-19", "text": ""}, {"location": "user/changelog/#whats-changed_46", "title": "What's changed", "text": "Move GitHub action to repository root by @disrupted in #356
Fix link to kpops-examples by @sujuka99 in #357
Fix Kafka connect config name for deletion by @raminqaf in #361
Bump version 2.0.8 \u2192 2.0.9 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.8...2.0.9
"}, {"location": "user/changelog/#208-2023-09-06", "title": "2.0.8 - 2023-09-06", "text": ""}, {"location": "user/changelog/#whats-changed_47", "title": "What's changed", "text": "Refactor component prefix & name by @disrupted in #326
Remove unnecessary condition during inflate by @disrupted in #328
Fix config.yaml overriding environment variables by @sujuka99 in #353
Bump version 2.0.7 \u2192 2.0.8 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.7...2.0.8
"}, {"location": "user/changelog/#207-2023-08-31", "title": "2.0.7 - 2023-08-31", "text": ""}, {"location": "user/changelog/#whats-changed_48", "title": "What's changed", "text": "Print only rendered templates when --template flag is set by @raminqaf in #350
Add migration guide by @raminqaf in #352
Bump version 2.0.6 \u2192 2.0.7 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.6...2.0.7
"}, {"location": "user/changelog/#206-2023-08-30", "title": "2.0.6 - 2023-08-30", "text": ""}, {"location": "user/changelog/#whats-changed_49", "title": "What's changed", "text": "Simplify deployment with local Helm charts by @raminqaf in #349
Bump version 2.0.5 \u2192 2.0.6 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.5...2.0.6
"}, {"location": "user/changelog/#205-2023-08-30", "title": "2.0.5 - 2023-08-30", "text": ""}, {"location": "user/changelog/#whats-changed_50", "title": "What's changed", "text": "Fix versioning of docs when releasing by @raminqaf in #346
Bump version 2.0.4 \u2192 2.0.5 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.4...2.0.5
"}, {"location": "user/changelog/#204-2023-08-29", "title": "2.0.4 - 2023-08-29", "text": ""}, {"location": "user/changelog/#whats-changed_51", "title": "What's changed", "text": "Exclude abstract components from pipeline schema by @disrupted in #332
Add dprint as the markdown formatter by @raminqaf in #337
Publish pre-release docs for PRs & main branch by @raminqaf in #339
Fix GitHub ref variable for pushing docs to main branch by @raminqaf in #343
Align docs colours by @raminqaf in #345
Bump version 2.0.3 \u2192 2.0.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.3...2.0.4
"}, {"location": "user/changelog/#203-2023-08-24", "title": "2.0.3 - 2023-08-24", "text": ""}, {"location": "user/changelog/#whats-changed_52", "title": "What's changed", "text": "Lint GitHub action by @disrupted in #342
Fix GitHub action error in non-Python projects by @disrupted in #340
Bump version 2.0.2 \u2192 2.0.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.2...2.0.3
"}, {"location": "user/changelog/#202-2023-08-23", "title": "2.0.2 - 2023-08-23", "text": ""}, {"location": "user/changelog/#whats-changed_53", "title": "What's changed", "text": "Add version dropdown to the documentation by @raminqaf in #336
Break the documentation down into smaller subsection by @raminqaf in #329
Bump version 2.0.1 \u2192 2.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.1...2.0.2
"}, {"location": "user/changelog/#201-2023-08-22", "title": "2.0.1 - 2023-08-22", "text": ""}, {"location": "user/changelog/#whats-changed_54", "title": "What's changed", "text": "Fix optional flags in GitHub action by @disrupted in #334
Bump version 2.0.0 \u2192 2.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.0...2.0.1
"}, {"location": "user/changelog/#200-2023-08-17", "title": "2.0.0 - 2023-08-17", "text": ""}, {"location": "user/changelog/#whats-changed_55", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v2 by @disrupted
v2 by @disrupted in #321
Bump version 1.7.2 \u2192 2.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.7.2...2.0.0
"}, {"location": "user/changelog/#172-2023-08-16", "title": "1.7.2 - 2023-08-16", "text": ""}, {"location": "user/changelog/#whats-changed_56", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v2 by @disrupted
Refactor input/output types by @sujuka99 in #232
Fix editor integration example in docs by @sujuka99 in #273
Add KPOps Runner GitHub Action to the documentation by @raminqaf in #325
Refactor Kafka Connect handler by @disrupted in #322
Remove :type and :rtype from docstrings by @raminqaf in #324
Merge remote-tracking branch 'origin/main' into v2 by @disrupted
Bump version 1.7.1 \u2192 1.7.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.7.1...1.7.2
"}, {"location": "user/changelog/#171-2023-08-15", "title": "1.7.1 - 2023-08-15", "text": ""}, {"location": "user/changelog/#whats-changed_57", "title": "What's changed", "text": "Modularize and autogenerate examples for the documentation by @sujuka99 in #267
Update the variable documentation by @sujuka99 in #266
Merge remote-tracking branch 'origin/main' into v2 by @disrupted
Update docs generation by @disrupted
Bump version 1.7.0 \u2192 1.7.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.7.0...1.7.1
"}, {"location": "user/changelog/#170-2023-08-15", "title": "1.7.0 - 2023-08-15", "text": ""}, {"location": "user/changelog/#whats-changed_58", "title": "What's changed", "text": "Add flag to exclude pipeline steps by @raminqaf in #300
Bump version 1.6.0 \u2192 1.7.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.6.0...1.7.0
"}, {"location": "user/changelog/#160-2023-08-10", "title": "1.6.0 - 2023-08-10", "text": ""}, {"location": "user/changelog/#whats-changed_59", "title": "What's changed", "text": "Refactor handling of Helm flags by @disrupted in #319
Bump version 1.5.0 \u2192 1.6.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.5.0...1.6.0
"}, {"location": "user/changelog/#150-2023-08-10", "title": "1.5.0 - 2023-08-10", "text": ""}, {"location": "user/changelog/#whats-changed_60", "title": "What's changed", "text": "Remove camel case conversion of internal models by @disrupted in #308
Automatically support schema generation for custom components by @disrupted in #307
Derive component type automatically from class name by @disrupted in #309
Refactor Helm wrapper and add --set-file flag by @disrupted in #311
Set default for ToSection topics by @disrupted in #313
Annotate types for ToSection models mapping by @disrupted in #315
Check Poetry lock file consistency by @disrupted in #316
Bump version 1.4.0 \u2192 1.5.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.4.0...1.5.0
"}, {"location": "user/changelog/#140-2023-08-02", "title": "1.4.0 - 2023-08-02", "text": ""}, {"location": "user/changelog/#whats-changed_61", "title": "What's changed", "text": "Update Black by @disrupted in #294
Fix vulnerability in mkdocs-material by @disrupted in #295
Move breaking changes section upper in the change log config by @raminqaf in #287
Order PipelineComponent fields by @disrupted in #290
Migrate requests to httpx by @irux in #302
Validate unique step names by @disrupted in #292
Refactor CLI using dtyper by @disrupted in #306
Bump version 1.3.2 \u2192 1.4.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.3.2...1.4.0
"}, {"location": "user/changelog/#132-2023-07-13", "title": "1.3.2 - 2023-07-13", "text": ""}, {"location": "user/changelog/#whats-changed_62", "title": "What's changed", "text": "Exclude Helm tests from dry-run diff by @raminqaf in #293
Bump version 1.3.1 \u2192 1.3.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.3.1...1.3.2
"}, {"location": "user/changelog/#131-2023-07-11", "title": "1.3.1 - 2023-07-11", "text": ""}, {"location": "user/changelog/#whats-changed_63", "title": "What's changed", "text": "Update codeowners by @disrupted in #281
Reactivate Windows CI by @irux in #255
Downgrade Poetry version on the Windows CI pipeline by @irux in #286
Remove workaround for pipeline steps by @disrupted in #276
Set ANSI theme for output of kpops generate by @disrupted in #289
Bump version 1.3.0 \u2192 1.3.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.3.0...1.3.1
"}, {"location": "user/changelog/#130-2023-07-07", "title": "1.3.0 - 2023-07-07", "text": ""}, {"location": "user/changelog/#whats-changed_64", "title": "What's changed", "text": "Update KPOps runner readme for dev versions by @raminqaf in #279
Add breaking changes section to change log config by @raminqaf in #280
Plural broker field in pipeline config by @raminqaf in #278
Bump version 1.2.4 \u2192 1.3.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.4...1.3.0
"}, {"location": "user/changelog/#124-2023-06-27", "title": "1.2.4 - 2023-06-27", "text": ""}, {"location": "user/changelog/#whats-changed_65", "title": "What's changed", "text": "Update changelog action to contain miscellaneous PRs by @raminqaf in #269
Bump version 1.2.3 \u2192 1.2.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.3...1.2.4
"}, {"location": "user/changelog/#123-2023-06-22", "title": "1.2.3 - 2023-06-22", "text": ""}, {"location": "user/changelog/#whats-changed_66", "title": "What's changed", "text": "Refactor custom component validation & hide field from kpops output by @disrupted in #265
Bump version 1.2.2 \u2192 1.2.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.2...1.2.3
"}, {"location": "user/changelog/#122-2023-06-21", "title": "1.2.2 - 2023-06-21", "text": ""}, {"location": "user/changelog/#whats-changed_67", "title": "What's changed", "text": "Create workflow to lint CI by @disrupted in #260
Fix update docs when releasing by @irux in #261
Rename change log message for uncategorized issues by @raminqaf in #262
Bump version 1.2.1 \u2192 1.2.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.1...1.2.2
"}, {"location": "user/changelog/#121-2023-06-21", "title": "1.2.1 - 2023-06-21", "text": ""}, {"location": "user/changelog/#whats-changed_68", "title": "What's changed", "text": "Fix update docs in release workflow by @irux in #258
Bump version 1.2.0 \u2192 1.2.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.0...1.2.1
"}, {"location": "user/changelog/#120-2023-06-21", "title": "1.2.0 - 2023-06-21", "text": ""}, {"location": "user/changelog/#whats-changed_69", "title": "What's changed", "text": "Add background to docs home page by @disrupted in #236
Remove enable option from helm diff by @raminqaf in #235
add --namespace option to Helm template command by @raminqaf in #237
Add missing type annotation for Pydantic attributes by @disrupted in #238
Add helm repo update <repo-name> for Helm >3.7 by @raminqaf in #239
Fix helm version check by @sujuka99 in #242
Refactor variable substitution by @sujuka99 in #198
Fix Helm Version Check by @sujuka99 in #244
Update Poetry version in CI by @sujuka99 in #247
Add pip cache in KPOps runner action by @raminqaf in #249
Check types using Pyright by @disrupted in #251
Remove MyPy by @disrupted in #252
Disable broken Windows CI temporarily by @sujuka99 in #253
Update release and publish workflows by @irux in #254
Fix import from external module by @disrupted in #256
Fix release & publish workflows by @irux in #257
Bump version 1.1.5 \u2192 1.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.5...1.2.0
"}, {"location": "user/changelog/#115-2023-06-07", "title": "1.1.5 - 2023-06-07", "text": ""}, {"location": "user/changelog/#whats-changed_70", "title": "What's changed", "text": "Fix links to ATM-fraud defaults by @sujuka99 in #219
Exclude pytest snapshots from pre-commit hook by @sujuka99 in #226
Add Windows support by @irux in #217
Fix missing extra input topics by @disrupted in #230
Bump version 1.1.4 \u2192 1.1.5 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.4...1.1.5
"}, {"location": "user/changelog/#114-2023-05-22", "title": "1.1.4 - 2023-05-22", "text": ""}, {"location": "user/changelog/#whats-changed_71", "title": "What's changed", "text": "Document environment-specific pipeline definitions by @sujuka99 in #210
Set up Helm inside composite action & install latest KPOps by default by @disrupted in #211
Update example pipeline by @sujuka99 in #216
Bump version 1.1.3 \u2192 1.1.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.3...1.1.4
"}, {"location": "user/changelog/#113-2023-05-04", "title": "1.1.3 - 2023-05-04", "text": ""}, {"location": "user/changelog/#whats-changed_72", "title": "What's changed", "text": "Rewrite bash pre-commit hooks in Python by @sujuka99 in #207
Collapse pip install output for GitHub action by @disrupted in #209
Fix misleading error of 'File or directory not found' by @irux in #208
Bump version 1.1.2 \u2192 1.1.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.2...1.1.3
"}, {"location": "user/changelog/#112-2023-04-27", "title": "1.1.2 - 2023-04-27", "text": ""}, {"location": "user/changelog/#whats-changed_73", "title": "What's changed", "text": "Add titles and descriptions to Pydantic model fields by @sujuka99 in #191
Respect object docstring titles by @sujuka99 in #196
Allow manually running the CI by @sujuka99 in #204
Generate schema in CI by @sujuka99 in #197
Add kpops --version command by @disrupted in #206
Bump version 1.1.1 \u2192 1.1.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.1...1.1.2
"}, {"location": "user/changelog/#111-2023-04-17", "title": "1.1.1 - 2023-04-17", "text": ""}, {"location": "user/changelog/#whats-changed_74", "title": "What's changed", "text": "Expose pipeline component by @irux in #192
Bump version 1.1.0 \u2192 1.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.0...1.1.1
"}, {"location": "user/changelog/#110-2023-04-11", "title": "1.1.0 - 2023-04-11", "text": ""}, {"location": "user/changelog/#whats-changed_75", "title": "What's changed", "text": "Error when running generate with --steps by @sujuka99 in #169
Make schema generation a builtin CLI command by @sujuka99 in #166
Add CLI Usage doc generation to CI by @sujuka99 in #174
Add new badges to readme and improve KubernetesApp docs by @raminqaf in #186
Read from component by @disrupted in #193
Bump version 1.0.1 \u2192 1.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.0.1...1.1.0
"}, {"location": "user/changelog/#101-2023-03-23", "title": "1.0.1 - 2023-03-23", "text": ""}, {"location": "user/changelog/#whats-changed_76", "title": "What's changed", "text": "fix(README): documentation leads to user-guide by @sujuka99 in #163
Fix serialization of pathlib.Path type on model export by @disrupted in #168
Bump version 1.0.0 \u2192 1.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.0.0...1.0.1
"}, {"location": "user/changelog/#100-2023-03-20", "title": "1.0.0 - 2023-03-20", "text": ""}, {"location": "user/changelog/#whats-changed_77", "title": "What's changed", "text": "Update \"What is KPOps\" section to be more catchy by @sujuka99 in #148
Fix broken links in README by @raminqaf in #160
Update CLI usage Reference by @sujuka99 in #152
Fix config.yaml defaults_path being overridden by CLI by @sujuka99 in #151
Bump version 0.12.0 \u2192 1.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.12.0...1.0.0
"}, {"location": "user/changelog/#0120-2023-03-15", "title": "0.12.0 - 2023-03-15", "text": ""}, {"location": "user/changelog/#whats-changed_78", "title": "What's changed", "text": "Create documentation for defaults.yaml by @sujuka99 in #146
Rename kafka-connect to kafka-connector by @sujuka99 in #150
Set schema for Kafka Connect config by @disrupted in #132
Fix missing enum keys in Kafka REST proxy response model by @irux in #135
Bump version 0.11.2 \u2192 0.12.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.11.2...0.12.0
"}, {"location": "user/changelog/#0112-2023-03-07", "title": "0.11.2 - 2023-03-07", "text": ""}, {"location": "user/changelog/#whats-changed_79", "title": "What's changed", "text": "Create documentation of KPOps components by @sujuka99 in #112
Helm diff should not render NOTES.txt by @sujuka99 in #130
Improve inflate example & enum comparison in test by @disrupted in #104
Remove duplicate documentation about CLI environment variables by @disrupted in #140
Provide documentation for editor integration by @sujuka99 in #137
Create documentation of config.yaml by @sujuka99 in #138
Refactor loading of component defaults to independent function by @disrupted in #147
Bump version 0.11.1 \u2192 0.11.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.11.1...0.11.2
"}, {"location": "user/changelog/#0111-2023-02-23", "title": "0.11.1 - 2023-02-23", "text": ""}, {"location": "user/changelog/#whats-changed_80", "title": "What's changed", "text": "Skip FromSection for producers by @disrupted in #125
Fix pipeline environment override by @disrupted in #127
Bump version 0.11.0 \u2192 0.11.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.11.0...0.11.1
"}, {"location": "user/changelog/#0110-2023-02-22", "title": "0.11.0 - 2023-02-22", "text": ""}, {"location": "user/changelog/#whats-changed_81", "title": "What's changed", "text": "Full Changelog: https://github.com/bakdata/kpops/compare/0.10.4...0.11.0
"}, {"location": "user/changelog/#0104-2023-02-22", "title": "0.10.4 - 2023-02-22", "text": ""}, {"location": "user/changelog/#whats-changed_82", "title": "What's changed", "text": "Fix enrichment of inflated components by @disrupted in #118
Assign default reviewers through codeowners by @disrupted in #124
Update streams-bootstrap autoscaling config by @disrupted in #122
Bump version 0.10.3 \u2192 0.10.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.3...0.10.4
"}, {"location": "user/changelog/#0103-2023-02-16", "title": "0.10.3 - 2023-02-16", "text": ""}, {"location": "user/changelog/#whats-changed_83", "title": "What's changed", "text": "Update screenshot of word count pipeline by @disrupted in #116
Fix topic name substitution of ${component_name} in ToSection by @disrupted in #117
Bump version 0.10.2 \u2192 0.10.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.2...0.10.3
"}, {"location": "user/changelog/#0102-2023-02-15", "title": "0.10.2 - 2023-02-15", "text": ""}, {"location": "user/changelog/#whats-changed_84", "title": "What's changed", "text": "Create deployment documentation for Word Count pipeline by @sujuka99 in #107
Delete leftover pipeline prefix config by @disrupted in #111
Remove poetry run from Quickstart doc by @sujuka99 in #114
Fix incomplete inflate component by @disrupted in #105
Bump version 0.10.1 \u2192 0.10.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.1...0.10.2
"}, {"location": "user/changelog/#0101-2023-02-13", "title": "0.10.1 - 2023-02-13", "text": ""}, {"location": "user/changelog/#whats-changed_85", "title": "What's changed", "text": "Add name to connector dry-run diff by @philipp94831 in #108
Bump version 0.10.0 \u2192 0.10.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.0...0.10.1
"}, {"location": "user/changelog/#0100-2023-02-13", "title": "0.10.0 - 2023-02-13", "text": ""}, {"location": "user/changelog/#whats-changed_86", "title": "What's changed", "text": "Fix diff not shown for new Helm releases by @disrupted in #92
Fix ATM fraud example by @disrupted in #95
Fix kpops version in pyproject.toml by @raminqaf in #99
Clean up dry-run logging by @philipp94831 in #100
Refactor integration test by @disrupted in #96
Refactor change calculation by @disrupted in #88
Support printing final Kubernetes resources with kpops generate by @sujuka99 in #69
Set Kafka Connect config name from component by @irux in #98
Add prefix as an option to customize by @irux in #97
Bump version 0.9.0 \u2192 0.10.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.9.0...0.10.0
"}, {"location": "user/changelog/#090-2023-02-03", "title": "0.9.0 - 2023-02-03", "text": ""}, {"location": "user/changelog/#whats-changed_87", "title": "What's changed", "text": "Remove mike set-default command by @raminqaf in #86
Add --create-namespace option to helm by @raminqaf in #91
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.3...0.9.0
"}, {"location": "user/changelog/#083-2023-02-01", "title": "0.8.3 - 2023-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_88", "title": "What's changed", "text": "Correct push flag of mike by @raminqaf in #84
Bump version 0.8.2 \u2192 0.8.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.2...0.8.3
"}, {"location": "user/changelog/#082-2023-02-01", "title": "0.8.2 - 2023-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_89", "title": "What's changed", "text": "Add --push flag to mike by @raminqaf in #83
Bump version 0.8.1 \u2192 0.8.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.1...0.8.2
"}, {"location": "user/changelog/#081-2023-02-01", "title": "0.8.1 - 2023-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_90", "title": "What's changed", "text": "Tidy user guide by @disrupted in #81
Fix typo and metrics replication factor in Kafka values by @yannick-roeder in #82
Bump version 0.8.0 \u2192 0.8.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.0...0.8.1
"}, {"location": "user/changelog/#080-2023-01-30", "title": "0.8.0 - 2023-01-30", "text": ""}, {"location": "user/changelog/#whats-changed_91", "title": "What's changed", "text": "Generate schema for pipeline.yaml and config.yaml by @disrupted in #70
Bump version 0.7.0 \u2192 0.8.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.7.0...0.8.0
"}, {"location": "user/changelog/#070-2023-01-19", "title": "0.7.0 - 2023-01-19", "text": ""}, {"location": "user/changelog/#whats-changed_92", "title": "What's changed", "text": "Update setup.cfg by @sujuka99 in #65
Refactor component configs by @raminqaf in #63
Bump version 0.6.1 \u2192 0.7.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.6.1...0.7.0
"}, {"location": "user/changelog/#061-2023-01-12", "title": "0.6.1 - 2023-01-12", "text": ""}, {"location": "user/changelog/#whats-changed_93", "title": "What's changed", "text": "Refactor Kubernetes app properties by @disrupted in #60
Fix Helm release name trimming of cleanup jobs by @disrupted in #61
Bump version 0.6.0 \u2192 0.6.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.6.0...0.6.1
"}, {"location": "user/changelog/#060-2023-01-09", "title": "0.6.0 - 2023-01-09", "text": ""}, {"location": "user/changelog/#whats-changed_94", "title": "What's changed", "text": "Separate clean, reset, and destroy logic by @raminqaf in #57
Fix trigger CI job once on release workflow by @raminqaf in #58
Fix double push of docs to GitHub pages by @raminqaf in #59
Bump version 0.5.0 \u2192 0.6.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.5.0...0.6.0
"}, {"location": "user/changelog/#050-2023-01-05", "title": "0.5.0 - 2023-01-05", "text": ""}, {"location": "user/changelog/#whats-changed_95", "title": "What's changed", "text": "Fix release version for TestPyPI by @philipp94831 in #48
Change topic_name variable to output_topic_name by @MichaelKora in #50
Improve exception output for integration tests by @disrupted in #51
Refactor usage of Pydantic aliases by @disrupted in #52
Add MyPy plugin for Pydantic by @disrupted in #56
Use component name instead of type to set default output topic name by @MichaelKora in #53
Refactor Helm Wrapper by @raminqaf in #47
Bump version 0.4.1 \u2192 0.5.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.4.1...0.5.0
"}, {"location": "user/changelog/#041-2022-12-22", "title": "0.4.1 - 2022-12-22", "text": ""}, {"location": "user/changelog/#whats-changed_96", "title": "What's changed", "text": "Fix link for getting started in readme by @torbsto in #34
Use new Helm repositories for streams-bootstrap and Kafka Connect resetter by @philipp94831 in #36
Fix spelling of PyPI by @disrupted in #33
Fix typo in docs by @disrupted in #38
Fix broken links in the documentation by @raminqaf in #39
Fix generate connecting to Kafka REST proxy by @disrupted in #41
Bump version 0.4.0 \u2192 0.4.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.4.0...0.4.1
"}, {"location": "user/changelog/#040-2022-12-21", "title": "0.4.0 - 2022-12-21", "text": ""}, {"location": "user/changelog/#whats-changed_97", "title": "What's changed", "text": "Add installation instructions to README by @raminqaf in #30
Fix usage of template workflow for Poetry release by @disrupted in #25
Set default value of retain clean jobs flag to false by @raminqaf in #31
Refactor component handlers by @disrupted in #3
Bump version 0.3.0 \u2192 0.3.1 by @bakdata-bot
Bump version 0.3.1 \u2192 0.4.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.3.0...0.4.0
"}, {"location": "user/changelog/#030-2022-12-21", "title": "0.3.0 - 2022-12-21", "text": ""}, {"location": "user/changelog/#whats-changed_98", "title": "What's changed", "text": "Initial commit by @philipp94831
Add source code of KPOps by @raminqaf in #1
Add GitHub action by @philipp94831 in #2
Update project version by @raminqaf in #4
Update project version by @raminqaf in #5
Remove workflow and add release actions by @raminqaf in #8
Fix env variable in GitHub actions by @raminqaf in #9
Bump version 0.2.2 \u2192 0.2.3 by @bakdata-bot
Remove credential flag from checkout in update docs by @raminqaf in #10
Bump version 0.2.3 \u2192 0.2.4 by @bakdata-bot
Update version in actions readme by @JakobEdding in #11
Bump version 0.2.4 \u2192 0.2.5 by @bakdata-bot
Remove push tag step by @raminqaf in #13
Bump version 0.2.5 \u2192 0.2.6 by @bakdata-bot
Bump version 0.2.6 \u2192 0.3.0 by @bakdata-bot
With a couple of easy commands in the shell, and a pipeline.yaml of under 30 lines, KPOps can not only deploy a Kafka pipeline1 to a Kubernetes cluster, but also reset, clean or destroy it!
- type: producer-app\n name: data-producer\n values:\n image: bakdata/kpops-demo-sentence-producer\n imageTag: \"2.0.0\"\n\n- type: streams-app\n name: word-count-app\n values:\n image: bakdata/kpops-demo-word-count-app\n imageTag: \"2.0.0\"\n replicaCount: 1\n to:\n topics:\n ${output_topic_name}:\n type: output\n configs:\n cleanup.policy: compact\n\n- type: kafka-sink-connector\n name: redis-sink-connector\n config:\n connector.class: com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector\n redis.hosts: redis-headless:6379\n redis.database: 0\n tasks.max: 1\n key.converter: org.apache.kafka.connect.storage.StringConverter\n value.converter: org.apache.kafka.connect.storage.StringConverter\nA Kafka pipeline can consist of consecutive streaming applications, producers, and connectors.\u00a0\u21a9
KPOps reads its global configuration that is unrelated to a pipeline's components from config.yaml.
Consider enabling KPOps' editor integration feature to enjoy the benefits of autocompletion and validation when configuring your pipeline.
To learn about any of the available settings, take a look at the example below.
config.yaml # CONFIGURATION\n#\n# Base directory to the pipelines (default is current working directory)\npipeline_base_dir: .\n# The Kafka brokers address.\n# REQUIRED\nkafka_brokers: \"http://broker1:9092,http://broker2:9092\"\n# Configure the topic name variables you can use in the pipeline definition.\ntopic_name_config:\n # Configures the value for the variable ${output_topic_name}\n default_output_topic_name: ${pipeline.name}-${component.name}\n # Configures the value for the variable ${error_topic_name}\n default_error_topic_name: ${pipeline.name}-${component.name}-error\n# Configuration for Schema Registry.\nschema_registry:\n # Whether the Schema Registry handler should be initialized.\n enabled: false\n # Address of the Schema Registry.\n url: \"http://localhost:8081\"\n# Configuration for the Kafka REST Proxy.\nkafka_rest:\n # Address of the Kafka REST Proxy.\n url: \"http://localhost:8082\"\n# Configuration for Kafka Connect.\nkafka_connect:\n # Address of Kafka Connect.\n url: \"http://localhost:8083\"\n# Flag for `helm upgrade --install`.\n# Create the release namespace if not present.\ncreate_namespace: false\n# Global flags for Helm.\nhelm_config:\n # Name of kubeconfig context (`--kube-context`)\n context: name\n # Run Helm in Debug mode.\n debug: false\n # Kubernetes API version used for Capabilities.APIVersions\n api_version: null\n# Configure Helm Diff.\nhelm_diff_config:\n # Set of keys that should not be checked.\n ignore:\n - name\n - imageTag\n# Whether to retain clean up jobs in the cluster or uninstall the, after\n# completion.\nretain_clean_jobs: false\nEnvironment-specific pipeline definitions
Similarly to defaults, it is possible to have an unlimited amount of additional environment-specific pipeline definitions. The naming convention is the same: add a suffix of the form _{environment} to the filename.
KPOps has a very efficient way of dealing with repeating settings which manifests as defaults.yaml. This file provides the user with the power to set defaults for any and all components, thus omitting the need to repeat the same settings in pipeline.yaml.
See real-world examples for defaults.
An important mechanic of KPOps is that defaults set for a component apply to all components that inherit from it.
It is possible, although not recommended, to add settings that are specific to a component's subclass. An example would be configuring offset_topic under kafka-connector instead of kafka-source-connector.
KPOps allows using multiple default values. The defaults.yaml (or defaults_<env>.yaml) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single pipeline.yaml file. KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the pipeline_base_dir is defined.
The deepest defaults.yaml file in the folder hierarchy (i.e., the closest one to the pipeline.yaml) overwrites the higher-level defaults' values.
It is important to note that defaults_{environment}.yaml overrides only the settings that are explicitly set to be different from the ones in the base defaults file.
Imagine the following folder structure, where the pipeline_base_dir is configured to pipelines:
\u2514\u2500 pipelines\n \u2514\u2500\u2500 distributed-defaults\n \u251c\u2500\u2500 defaults.yaml\n \u251c\u2500\u2500 defaults_dev.yaml\n \u2514\u2500\u2500 pipeline-deep\n \u251c\u2500\u2500 defaults.yaml\n \u2514\u2500\u2500 pipeline.yaml\nKPOps picks up the defaults in the following order (high to low priority):
./pipelines/distributed-defaults/pipeline-deep/defaults.yaml./pipelines/distributed-defaults/defaults_dev.yaml./pipelines/distributed-defaults/defaults.yamlThe defaults codeblocks in this section contain the full set of settings that are specific to the component. If a setting already exists in a parent config, it will not be included in the child's.
defaults.yaml # Base Kubernetes App\n#\n# Parent of: HelmApp\n# Child of: PipelineComponent\nkubernetes-app:\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n values: # required\n image: exampleImage # Example\n debug: false # Example\n commandLine: {} # Example\ndefaults.yaml # StreamsApp component that configures a streams bootstrap app.\n#\n# Child of: KafkaApp\n# More documentation on StreamsApp: https://github.com/bakdata/streams-bootstrap\nstreams-app:\n # No arbitrary keys are allowed under `app`here\n # Allowed configs:\n # https://github.com/bakdata/streams-bootstrap/tree/master/charts/streams-app\n values: # required\n # Streams Bootstrap streams section\n streams: # required, streams-app-specific\n brokers: ${config.kafka_brokers} # required\n schemaRegistryUrl: ${config.schema_registry.url}\n inputTopics:\n - topic1\n - topic2\n outputTopic: output-topic\n inputPattern: input-pattern\n extraInputTopics:\n input_role1:\n - input_topic1\n - input_topic2\n input_role2:\n - input_topic3\n - input_topic4\n extraInputPatterns:\n pattern_role1: input_pattern1\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\n errorTopic: error-topic\n config:\n my.streams.config: my.value\n nameOverride: override-with-this-name # streams-app-specific\n autoscaling: # streams-app-specific\n consumerGroup: consumer-group # required\n lagThreshold: 0 # Average target value to trigger scaling actions.\n enabled: false # Whether to enable auto-scaling using KEDA.\n # This is the interval to check each trigger on.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#pollinginterval\n pollingInterval: 30\n # The period to wait after the last trigger reported active before scaling\n # the resource back to 0. https://keda.sh/docs/2.9/concepts/scaling-deployments/#cooldownperiod\n cooldownPeriod: 300\n # The offset reset policy for the consumer if the the consumer group is\n # not yet subscribed to a partition.\n offsetResetPolicy: earliest\n # This setting is passed to the HPA definition that KEDA will create for a\n # given resource and holds the maximum number of replicas of the target resouce.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#maxreplicacount\n maxReplicas: 1\n # Minimum number of replicas KEDA will scale the resource down to.\n # https://keda.sh/docs/2.7/concepts/scaling-deployments/#minreplicacount\n minReplicas: 0\n # If this property is set, KEDA will scale the resource down to this\n # number of replicas.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#idlereplicacount\n idleReplicas: 0\n topics: # List of auto-generated Kafka Streams topics used by the streams app.\n - topic1\n - topic2\ndefaults.yaml \ndefaults.yaml # Kafka connector\n#\n# Parent of: KafkaSinkConnector, KafkaSourceConnector\n# Child of: PipelineComponent\nkafka-connector:\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs\n config: # required\n tasks.max: 1\n # Overriding Kafka Connect Resetter Helm values. E.g. to override the\n # Image Tag etc.\n resetter_values:\n imageTag: \"1.2.3\"\ndefaults.yaml # Kafka source connector\n#\n# Child of: KafkaConnector\nkafka-source-connector:\n # The source connector has no `from` section\n # from:\n # offset.storage.topic\n # https://kafka.apache.org/documentation/#connect_running\n offset_topic: offset_topic\ndefaults.yaml # Kafka sink connector\n#\n# Child of: KafkaConnector\nkafka-sink-connector:\n # No settings differ from `kafka-connector`\nKPOps supports three operation modes\u2014managed, manifest, and argo. These modes determine how resources are managed and allow users to tailor their deployment strategy.
pipeline.yaml.You can configure the operation mode using one of the following methods:
Command-Line Option: Pass the --operation-mode <OPERATION> flag when running a CLI command. Refer to the CLI commands documentation for more details.
Environment Variable: Set the operation mode by defining the KPOPS_OPERATION_MODE environment variable.
deploy", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode", "title": "Manifest Mode", "text": "Job, Deployment, ConfigMap, and Service resources.KafkaTopic CRDs.Job, Deployment, ConfigMap, and Service resources.sync-wave annotation to ensure Kafka topics are created first (default sync-wave=0) before deploying apps (lower priority sync-wave>0). All components of each sync wave are deployed in parallel by Argo.KafkaTopic CRDs.Job resources configured with ArgoCD PostDelete hooks, ensuring cleanup tasks are executed after ArgoCD application deletion.reset", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode_1", "title": "Manifest Mode", "text": "KafkaTopic CRDs.Job resources for resetting Kafka Streams application states.KafkaTopic CRDs.Job resources without ArgoCD PostDelete hooks, providing a simpler reset process.clean", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode_2", "title": "Manifest Mode", "text": "Job resources for cleaning up temporary resources or artifacts using application container images.clean command is not supported in Argo mode. The clean is instead achieved through cleanup job hooks during the deploy command.destroy", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode_3", "title": "Manifest Mode", "text": "KafkaTopic CRDs.KafkaTopic CRDs.deploy reset clean destroy Producer Apps Manifest: Generated N/A N/A N/A Argo: Generated Streams Apps Manifest: Generated N/A N/A N/A Argo: Generated Topics Manifest: Generated Manifest: Generated N/A Manifest: Generated Argo: Generated Argo: Generated Argo: Generated Cleanup Jobs Manifest: N/A N/A Manifest: Generated N/A Argo: With PostDelete hooks N/A N/A N/A Reset Jobs Manifest: N/A Manifest: Generated N/A N/A Argo: Without PostDelete hooks"}, {"location": "user/core-concepts/components/helm-app/", "title": "HelmApp", "text": ""}, {"location": "user/core-concepts/components/helm-app/#usage", "title": "Usage", "text": "Can be used to deploy any app in Kubernetes using Helm, for example, a REST service that serves Kafka data.
"}, {"location": "user/core-concepts/components/helm-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # Kubernetes app managed through Helm with an associated Helm chart\n- type: helm-app\n name: helm-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n values: # required\n image: exampleImage # Example\n debug: false # Example\n commandLine: {} # Example\n # Helm repository configuration (optional)\n # If not set the helm repo add will not be called. Useful when using local Helm charts\n repo_config:\n repository_name: bakdata-streams-bootstrap # required\n url: https://bakdata.github.io/streams-bootstrap/ # required\n repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"1.0.0\" # Helm chart version\nDeploy using Helm.
"}, {"location": "user/core-concepts/components/helm-app/#destroy", "title": "destroy", "text": "Uninstall Helm release.
"}, {"location": "user/core-concepts/components/helm-app/#reset", "title": "reset", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/helm-app/#clean", "title": "clean", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/kafka-connector/", "title": "KafkaConnector", "text": "KafkaConnector is a component that deploys Kafka Connectors. Since a connector cannot be different from sink or source it is not recommended to use KafkaConnector for deployment in pipeline.yaml. Instead, KafkaConnector should be used in defaults.yaml to set defaults for all connectors in the pipeline as they can share some common settings.
Subclass of KafkaConnector.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#usage", "title": "Usage", "text": "Lets other systems pull data from Apache Kafka.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#configuration", "title": "Configuration", "text": "pipeline.yaml # Kafka sink connector\n- type: kafka-sink-connector\n name: kafka-sink-connector # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs\n config: # required\n tasks.max: 1\n # Overriding Kafka Connect Resetter Helm values. E.g. to override the\n # Image Tag etc.\n resetter_values:\n imageTag: \"1.2.3\"\nThe associated sink connector is removed from the Kafka Connect cluster.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#reset", "title": "reset", "text": "Reset the consumer group offsets using bakdata's sink resetter.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#clean", "title": "clean", "text": "Subclass of KafkaConnector.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#usage", "title": "Usage", "text": "Manages source connectors in your Kafka Connect cluster.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#configuration", "title": "Configuration", "text": "pipeline.yaml # Kafka source connector\n- type: kafka-source-connector # required\n name: kafka-source-connector # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n # The source connector has no `from` section\n # from:\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs\n config: # required\n tasks.max: 1\n # Overriding Kafka Connect Resetter Helm values. E.g. to override the\n # Image Tag etc.\n resetter_values:\n imageTag: \"1.2.3\"\n # offset.storage.topic\n # https://kafka.apache.org/documentation/#connect_running\n offset_topic: offset_topic\nRemove the source connector from the Kafka Connect cluster.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#reset", "title": "reset", "text": "Delete state associated with the connector using bakdata's source resetter.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#clean", "title": "clean", "text": "Can be used to create components for any Kubernetes app.
"}, {"location": "user/core-concepts/components/kubernetes-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # Base Kubernetes App\n- type: kubernetes-app\n name: kubernetes-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n values: # required\n image: exampleImage # Example\n debug: false # Example\n commandLine: {} # Example\nDo nothing.
"}, {"location": "user/core-concepts/components/kubernetes-app/#destroy", "title": "destroy", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/kubernetes-app/#reset", "title": "reset", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/kubernetes-app/#clean", "title": "clean", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/overview/", "title": "Overview", "text": "This section explains the different components of KPOps, their usage and configuration in the pipeline definition pipeline.yaml.
flowchart BT\n KubernetesApp --> PipelineComponent\n HelmApp --> KubernetesApp\n StreamsBootstrap --> HelmApp\n StreamsApp --> StreamsBootstrap\n ProducerApp --> StreamsBootstrap\n KafkaConnector --> PipelineComponent\n KafkaSourceConnector --> KafkaConnector\n KafkaSinkConnector --> KafkaConnector\n\n click KubernetesApp \"./../kubernetes-app\"\n click HelmApp \"./../helm-app\"\n click StreamsBootstrap \"./../streams-bootstrap\"\n click StreamsApp \"./../streams-app\"\n click ProducerApp \"./../producer-app\"\n click KafkaConnector \"./../kafka-connector\"\n click KafkaSourceConnector \"./../kafka-source-connector\"\n click KafkaSinkConnector \"./../kafka-sink-connector\"KPOps component hierarchy
"}, {"location": "user/core-concepts/components/producer-app/", "title": "ProducerApp", "text": "Subclass of StreamsBootstrap.
"}, {"location": "user/core-concepts/components/producer-app/#usage", "title": "Usage", "text": "Configures a streams-bootstrap Kafka producer app
"}, {"location": "user/core-concepts/components/producer-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # Holds configuration to use as values for the streams bootstrap producer-app Helm\n# chart.\n# More documentation on ProducerApp:\n# https://github.com/bakdata/streams-bootstrap\n- type: producer-app\n name: producer-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n # from: # While the producer-app does inherit from kafka-app, it does not need a\n # `from` section, hence it does not support it.\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n # Allowed configs:\n # https://github.com/bakdata/streams-bootstrap/tree/master/charts/producer-app\n values: # required\n streams: # required, producer-app-specific\n brokers: ${config.kafka_brokers} # required\n schemaRegistryUrl: ${config.schema_registry.url}\n outputTopic: output_topic\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\n nameOverride: override-with-this-name # kafka-app-specific\n # Helm repository configuration (optional)\n # If not set the helm repo add will not be called. Useful when using local Helm charts\n repo_config:\n repository_name: bakdata-streams-bootstrap # required\n url: https://bakdata.github.io/streams-bootstrap/ # required\n repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"2.12.0\" # Helm chart version\nIn addition to KubernetesApp's deploy:
Uninstall Helm release.
"}, {"location": "user/core-concepts/components/producer-app/#reset", "title": "reset", "text": "Do nothing, producers are stateless.
"}, {"location": "user/core-concepts/components/producer-app/#clean", "title": "clean", "text": "Subclass of and StreamsBootstrap.
"}, {"location": "user/core-concepts/components/streams-app/#usage", "title": "Usage", "text": "Configures a streams-bootstrap Kafka Streams app
"}, {"location": "user/core-concepts/components/streams-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # StreamsApp component that configures a streams bootstrap app.\n# More documentation on StreamsApp: https://github.com/bakdata/streams-bootstrap\n- type: streams-app # required\n name: streams-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n # No arbitrary keys are allowed under `app`here\n # Allowed configs:\n # https://github.com/bakdata/streams-bootstrap/tree/master/charts/streams-app\n values: # required\n # Streams Bootstrap streams section\n streams: # required, streams-app-specific\n brokers: ${config.kafka_brokers} # required\n schemaRegistryUrl: ${config.schema_registry.url}\n inputTopics:\n - topic1\n - topic2\n outputTopic: output-topic\n inputPattern: input-pattern\n extraInputTopics:\n input_role1:\n - input_topic1\n - input_topic2\n input_role2:\n - input_topic3\n - input_topic4\n extraInputPatterns:\n pattern_role1: input_pattern1\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\n errorTopic: error-topic\n config:\n my.streams.config: my.value\n nameOverride: override-with-this-name # streams-app-specific\n autoscaling: # streams-app-specific\n consumerGroup: consumer-group # required\n lagThreshold: 0 # Average target value to trigger scaling actions.\n enabled: false # Whether to enable auto-scaling using KEDA.\n # This is the interval to check each trigger on.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#pollinginterval\n pollingInterval: 30\n # The period to wait after the last trigger reported active before scaling\n # the resource back to 0. https://keda.sh/docs/2.9/concepts/scaling-deployments/#cooldownperiod\n cooldownPeriod: 300\n # The offset reset policy for the consumer if the the consumer group is\n # not yet subscribed to a partition.\n offsetResetPolicy: earliest\n # This setting is passed to the HPA definition that KEDA will create for a\n # given resource and holds the maximum number of replicas of the target resouce.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#maxreplicacount\n maxReplicas: 1\n # Minimum number of replicas KEDA will scale the resource down to.\n # https://keda.sh/docs/2.7/concepts/scaling-deployments/#minreplicacount\n minReplicas: 0\n # If this property is set, KEDA will scale the resource down to this\n # number of replicas.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#idlereplicacount\n idleReplicas: 0\n topics: # List of auto-generated Kafka Streams topics used by the streams app.\n - topic1\n - topic2\n # Helm repository configuration (optional)\n # If not set the helm repo add will not be called. Useful when using local Helm charts\n repo_config:\n repository_name: bakdata-streams-bootstrap # required\n url: https://bakdata.github.io/streams-bootstrap/ # required\n repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"2.12.0\" # Helm chart version\nIn addition to KubernetesApp's deploy:
Uninstall Helm release.
"}, {"location": "user/core-concepts/components/streams-app/#reset", "title": "reset", "text": "Similar to reset with to additional steps:
Subclass of HelmApp.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#usage", "title": "Usage", "text": "Defines a streams-bootstrap component
Often used in defaults.yaml
Deploy using Helm.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#destroy", "title": "destroy", "text": "Uninstall Helm release.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#reset", "title": "reset", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#clean", "title": "clean", "text": "Do nothing.
"}, {"location": "user/core-concepts/variables/environment_variables/", "title": "Environment variables", "text": "Environment variables can be set by using the export command in Linux or the set command in Windows.
dotenv files
KPOps currently supports .env files only for variables related to the config. Full support for .env files is on the roadmap. One of the possible ways to use one and export the contents manually is with the following command: export $(xargs < .env). This would work in bash suppose there are no spaces inside the values.
These variables take precedence over the settings in config.yaml. Variables marked as required can instead be set in the global config.
helm upgrade --install. Create the release namespace if not present. create_namespace KPOPS_HELM_CONFIG__CONTEXT False Name of kubeconfig context (--kube-context) helm_config.context KPOPS_HELM_CONFIG__DEBUG False False Run Helm in Debug mode helm_config.debug KPOPS_HELM_CONFIG__API_VERSION False Kubernetes API version used for Capabilities.APIVersions helm_config.api_version KPOPS_HELM_DIFF_CONFIG__IGNORE True Set of keys that should not be checked. helm_diff_config.ignore KPOPS_RETAIN_CLEAN_JOBS False False Whether to retain clean up jobs in the cluster or uninstall the, after completion. retain_clean_jobs KPOPS_STRIMZI_TOPIC False Configuration for Strimzi Kafka Topics. strimzi_topic KPOPS_OPERATION_MODE managed False The operation mode of KPOps (managed, manifest, argo). operation_mode config_env_vars.env Exhaustive list of all config-related environment variables# Global config environment variables\n#\n# The default setup is shown. These variables take precedence over the\n# settings in `config.yaml`. Variables marked as required can instead\n# be set in the global config.\n#\n# pipeline_base_dir\n# Base directory to the pipelines (default is current working\n# directory)\nKPOPS_PIPELINE_BASE_DIR=.\n# kafka_brokers\n# The comma separated Kafka brokers address.\nKPOPS_KAFKA_BROKERS # No default value, required\n# topic_name_config.default_output_topic_name\n# Configures the value for the variable ${output_topic_name}\nKPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME=${pipeline.name}-${component.name}\n# topic_name_config.default_error_topic_name\n# Configures the value for the variable ${error_topic_name}\nKPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME=${pipeline.name}-${component.name}-error\n# schema_registry.enabled\n# Whether the Schema Registry handler should be initialized.\nKPOPS_SCHEMA_REGISTRY__ENABLED=False\n# schema_registry.url\n# Address of the Schema Registry.\nKPOPS_SCHEMA_REGISTRY__URL=http://localhost:8081/\n# schema_registry.timeout\n# Operation timeout in seconds.\nKPOPS_SCHEMA_REGISTRY__TIMEOUT=30\n# kafka_rest.url\n# Address of the Kafka REST Proxy.\nKPOPS_KAFKA_REST__URL=http://localhost:8082/\n# kafka_rest.timeout\n# Operation timeout in seconds.\nKPOPS_KAFKA_REST__TIMEOUT=30\n# kafka_connect.url\n# Address of Kafka Connect.\nKPOPS_KAFKA_CONNECT__URL=http://localhost:8083/\n# kafka_connect.timeout\n# Operation timeout in seconds.\nKPOPS_KAFKA_CONNECT__TIMEOUT=30\n# create_namespace\n# Flag for `helm upgrade --install`. Create the release namespace if\n# not present.\nKPOPS_CREATE_NAMESPACE=False\n# helm_config.context\n# Name of kubeconfig context (`--kube-context`)\nKPOPS_HELM_CONFIG__CONTEXT # No default value, not required\n# helm_config.debug\n# Run Helm in Debug mode\nKPOPS_HELM_CONFIG__DEBUG=False\n# helm_config.api_version\n# Kubernetes API version used for `Capabilities.APIVersions`\nKPOPS_HELM_CONFIG__API_VERSION # No default value, not required\n# helm_diff_config.ignore\n# Set of keys that should not be checked.\nKPOPS_HELM_DIFF_CONFIG__IGNORE # No default value, required\n# retain_clean_jobs\n# Whether to retain clean up jobs in the cluster or uninstall the,\n# after completion.\nKPOPS_RETAIN_CLEAN_JOBS=False\n# strimzi_topic\n# Configuration for Strimzi Kafka Topics.\nKPOPS_STRIMZI_TOPIC # No default value, not required\n# operation_mode\n# The operation mode of KPOps (managed, manifest, argo).\nKPOPS_OPERATION_MODE=managed\nThese variables take precedence over the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags.
Name Default Value Required Description KPOPS_CONFIG_PATH . False Path to the dir containing config.yaml files KPOPS_DOTENV_PATH False Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. KPOPS_ENVIRONMENT False The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). KPOPS_OPERATION_MODE managed False How KPOps should operate. KPOPS_PIPELINE_PATHS True Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. KPOPS_PIPELINE_STEPS False Comma separated list of steps to apply the command on cli_env_vars.env Exhaustive list of all cli-related environment variables# CLI Environment variables\n#\n# The default setup is shown. These variables take precedence over the\n# commands' flags. If a variable is set, the corresponding flag does\n# not have to be specified in commands. Variables marked as required\n# can instead be set as flags.\n#\n# Path to the dir containing config.yaml files\nKPOPS_CONFIG_PATH=.\n# Path to dotenv file. Multiple files can be provided. The files will\n# be loaded in order, with each file overriding the previous one.\nKPOPS_DOTENV_PATH # No default value, not required\n# The environment you want to generate and deploy the pipeline to.\n# Suffix your environment files with this value (e.g.\n# defaults_development.yaml for environment=development).\nKPOPS_ENVIRONMENT # No default value, not required\n# How KPOps should operate.\nKPOPS_OPERATION_MODE=managed\n# Paths to dir containing 'pipeline.yaml' or files named\n# 'pipeline.yaml'.\nKPOPS_PIPELINE_PATHS # No default value, required\n# Comma separated list of steps to apply the command on\nKPOPS_PIPELINE_STEPS # No default value, not required\nKPOps supports the usage of placeholders and environment variables in pipeline definition and defaults.
"}, {"location": "user/core-concepts/variables/substitution/#component-specific-variables", "title": "Component-specific variables", "text": "These variables can be used in a component's definition to refer to any of its attributes, including ones that the user has defined in the defaults.
All of them are prefixed with component. and follow the following form: component.{attribute_name}. If the attribute itself contains attributes, they can be referred to like this: component.{attribute_name}.{subattribute_name}.
- type: scheduled-producer\n values:\n labels:\n app_type: \"${component.type}\"\n app_name: \"${component.name}\"\n app_schedule: \"${component.values.schedule}\"\n commandLine:\n FAKE_ARG: \"fake-arg-value\"\n schedule: \"30 3/8 * * *\"\n- type: converter\n values:\n commandLine:\n CONVERT_XML: true\n resources:\n limits:\n memory: 2G\n requests:\n memory: 2G\n- type: filter\n name: \"filter-app\"\n values:\n labels:\n app_type: \"${component.type}\"\n app_name: \"${component.name}\"\n app_resources_requests_memory: \"${component.values.resources.requests.memory}\"\n ${component.type}: \"${component.values.labels.app_name}-${component.values.labels.app_type}\"\n test_placeholder_in_placeholder: \"${component.values.labels.${component.type}}\"\n commandLine:\n TYPE: \"nothing\"\n resources:\n requests:\n memory: 3G\n replicaCount: 4\n autoscaling:\n minReplicas: 4\n maxReplicas: 4\nThese variables include all fields in the config and refer to the pipeline configuration that is independent of the components.
All such variables are prefixed with config. and are of the same form as the component-specific variables.
Info
error_topic_name is an alias for config.topic_name_config.default_error_topic_name output_topic_name is an alias for config.topic_name_config.default_output_topic_name
Environment variables such as $PATH can be used in the pipeline definition and defaults without any transformation following the form ${ENV_VAR_NAME}. This, of course, includes variables like the ones relevant to the KPOps cli that are exported by the user.
See all KPOps environment variables
"}, {"location": "user/core-concepts/variables/substitution/#pipeline-name-variables", "title": "Pipeline name variables", "text": "These are special variables that refer to the name and path of a pipeline.
${pipeline.name}: Concatenated path of the parent directory where pipeline.yaml is defined in. For instance, ./data/pipelines/v1/pipeline.yaml, here the value for the variable would be data-pipelines-v1.
${pipeline_name_<level>}: Similar to the previous variable, each <level> contains a part of the path to the pipeline.yaml file. Consider the previous example, ${pipeline_name_0} would be data, ${pipeline_name_1} would be pipelines, and ${pipeline_name_2} equals to v1.
ATM fraud is a demo pipeline for ATM fraud detection. The original by Confluent is written in KSQL and outlined in this blogpost. The one used in this example is re-built from scratch using bakdata's streams-bootstrap library.
Completed all steps in the setup.
"}, {"location": "user/examples/atm-fraud-pipeline/#setup-and-deployment", "title": "Setup and deployment", "text": ""}, {"location": "user/examples/atm-fraud-pipeline/#postgresql", "title": "PostgreSQL", "text": "Deploy PostgreSQL using the Bitnami Helm chart: Add the helm repository:
helm repo add bitnami https://charts.bitnami.com/bitnami && \\\nhelm repo update\nInstall the PostgreSQL with helm:
helm upgrade --install -f ./postgresql.yaml \\\n--namespace kpops \\\npostgresql bitnami/postgresql\npostgresql.yaml) auth:\n database: app_db\n enablePostgresUser: true\n password: AppPassword\n postgresPassword: StrongPassword\n username: app1\nprimary:\n persistence:\n enabled: false\n existingClaim: postgresql-data-claim\nvolumePermissions:\n enabled: true\nBefore we deploy the pipeline, we need to forward the ports of kafka-rest-proxy and kafka-connect. Run the following commands in two different terminals.
kubectl port-forward --namespace kpops service/k8kafka-cp-rest 8082:8082\nkubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:8083\nClone the kpops-examples repository and cd into the directory.
Install KPOps pip install -r requirements.txt.
Export environment variables in your terminal:
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nDeploy the pipeline
kpops deploy atm-fraud/pipeline.yaml --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be deployed correctly.
You can use the Streams Explorer to see the deployed pipeline. To do so, port-forward the service in a separate terminal session using the command below:
kubectl port-forward -n kpops service/streams-explorer 8080:8080\nAfter that open http://localhost:8080 in your browser. You should be able to see pipeline shown in the image below:
An overview of ATM fraud pipeline shown in Streams Explorer
Attention
Kafka Connect needs some time to set up the connector. Moreover, Streams Explorer needs a while to scrape the information from Kafka connect. Therefore, it might take a bit until you see the whole graph.
"}, {"location": "user/examples/atm-fraud-pipeline/#teardown-resources", "title": "Teardown resources", "text": ""}, {"location": "user/examples/atm-fraud-pipeline/#postrgresql", "title": "PostrgreSQL", "text": "PostgreSQL can be uninstalled by running the following command:
helm --namespace kpops uninstall postgresql\nExport environment variables in your terminal.
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nRemove the pipeline
kpops clean atm-fraud/pipeline.yaml --verbose --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be destroyed correctly.
Attention
If you face any issues destroying this example see Teardown for manual deletion.
"}, {"location": "user/examples/atm-fraud-pipeline/#common-errors", "title": "Common errors", "text": "deploy fails:clean.deploy --dry-run to avoid havig to clean again. If an error is dropped, start over from step 1.deploy.clean fails:clean.clean fails, follow the steps in teardown.Word-count is a demo pipeline consisting of a producer producing words to Kafka, a Kafka streams app counting the number of times each word occurs, and finally a Redis database into which the words are exported.
"}, {"location": "user/getting-started/quick-start/#what-this-will-demonstrate", "title": "What this will demonstrate", "text": "Completed all steps in the setup.
"}, {"location": "user/getting-started/quick-start/#setup-and-deployment", "title": "Setup and deployment", "text": ""}, {"location": "user/getting-started/quick-start/#redis", "title": "Redis", "text": "Deploy Redis using the Bitnami Helm chart: Add the Helm repository:
helm repo add bitnami https://charts.bitnami.com/bitnami && \\\nhelm repo update\nInstall Redis with Helm:
helm upgrade --install -f ./values-redis.yaml \\\n--namespace kpops \\\nredis bitnami/redis\nvalues-redis.yaml) architecture: standalone\nauth:\n enabled: false\nmaster:\n count: 1\n configuration: \"databases 1\"\nimage:\n tag: 7.0.8\nBefore we deploy the pipeline, we need to forward the ports of kafka-rest-proxy and kafka-connect. Run the following commands in two different terminals.
kubectl port-forward --namespace kpops service/k8kafka-cp-rest 8082:8082\nkubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:8083\nClone the kpops-examples repository and cd into the directory.
Install KPOps pip install -r requirements.txt.
Export environment variables in your terminal:
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nDeploy the pipeline
kpops deploy word-count/pipeline.yaml --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be deployed correctly.
You can use the Streams Explorer to inspect the deployed pipeline. To do so, port-forward the service in a separate terminal session using the command below:
kubectl port-forward -n kpops service/streams-explorer 8080:8080\nAfter that open http://localhost:8080 in your browser.
You should be able to see pipeline shown in the image below:
An overview of Word-count pipeline shown in Streams Explorer
Attention
Kafka Connect needs some time to set up the connector. Moreover, Streams Explorer needs a while to scrape the information from Kafka Connect. Therefore, it might take a bit until you see the whole graph.
"}, {"location": "user/getting-started/quick-start/#teardown-resources", "title": "Teardown resources", "text": ""}, {"location": "user/getting-started/quick-start/#redis_1", "title": "Redis", "text": "Redis can be uninstalled by running the following command:
helm --namespace kpops uninstall redis\nExport environment variables in your terminal.
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nRemove the pipeline
kpops clean word-count/pipeline.yaml --verbose --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be destroyed correctly.
Attention
If you face any issues destroying this example see Teardown for manual deletion.
"}, {"location": "user/getting-started/quick-start/#common-errors", "title": "Common errors", "text": "deploy fails:clean.deploy --dry-run to avoid having to clean again. If an error is dropped, start over from step 1.deploy.clean fails:clean.clean fails, follow the steps in teardown.In this part, you will set up KPOps. This includes:
If you don't have access to an existing Kubernetes cluster, this section will guide you through creating a local cluster. We recommend the lightweight Kubernetes distribution k3s for this. k3d is a wrapper around k3s in Docker that lets you get started fast.
You can install k3d with its installation script:
wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.4.6/install.sh | bash\nFor other ways of installing k3d, you can have a look at their installation guide.
The Kafka deployment needs a modified Docker image. In that case the image is built and pushed to a Docker registry that holds it. If you do not have access to an existing Docker registry, you can use k3d's Docker registry:
k3d registry create kpops-registry.localhost --port 12345\nNow you can create a new cluster called kpops that uses the previously created Docker registry:
k3d cluster create kpops --k3s-arg \"--no-deploy=traefik@server:*\" --registry-use k3d-kpops-registry.localhost:12345\nNote
Creating a new k3d cluster automatically configures kubectl to connect to the local cluster by modifying your ~/.kube/config. In case you manually set the KUBECONFIG variable or don't want k3d to modify your config, k3d offers many other options.
You can check the cluster status with kubectl get pods -n kube-system. If all returned elements have a STATUS of Running or Completed, then the cluster is up and running.
Kafka is an open-source data streaming platform. More information about Kafka can be found in the documentation. To deploy Kafka, this guide uses Confluent's Helm chart.
To allow connectivity to other systems Kafka Connect needs to be extended with drivers. You can install a JDBC driver for Kafka Connect by creating a new Docker image:
Create a Dockerfile with the following content:
FROM confluentinc/cp-kafka-connect:7.1.3\n\nRUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0\nBuild and push the modified image to your private Docker registry:
docker build . --tag localhost:12345/kafka-connect-jdbc:7.1.3 && \\\ndocker push localhost:12345/kafka-connect-jdbc:7.1.3\nDetailed instructions on building, tagging and pushing a docker image can be found in Docker docs.
Add Confluent's Helm chart repository and update the index:
helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ && \nhelm repo update\nInstall Kafka, Zookeeper, Confluent's Schema Registry, Kafka Rest Proxy, and Kafka Connect. A single Helm chart installs all five components. Below you can find an example for the --values ./kafka.yaml file configuring the deployment accordingly. Deploy the services:
helm upgrade \\\n --install \\\n --version 0.6.1 \\\n --values ./kafka.yaml \\\n --namespace kpops \\\n --create-namespace \\\n --wait \\\n k8kafka confluentinc/cp-helm-charts\nkafka.yaml) An example value configuration for Confluent's Helm chart. This configuration deploys a single Kafka Broker, a Schema Registry, Zookeeper, Kafka Rest Proxy, and Kafka Connect with minimal resources.
cp-zookeeper:\n enabled: true\n servers: 1\n imageTag: 7.1.3\n heapOptions: \"-Xms124M -Xmx124M\"\n overrideGroupId: k8kafka\n fullnameOverride: \"k8kafka-cp-zookeeper\"\n resources:\n requests:\n cpu: 50m\n memory: 0.2G\n limits:\n cpu: 250m\n memory: 0.2G\n prometheus:\n jmx:\n enabled: false\n\ncp-kafka:\n enabled: true\n brokers: 1\n imageTag: 7.1.3\n podManagementPolicy: Parallel\n configurationOverrides:\n \"auto.create.topics.enable\": false\n \"offsets.topic.replication.factor\": 1\n \"transaction.state.log.replication.factor\": 1\n \"transaction.state.log.min.isr\": 1\n \"confluent.metrics.reporter.topic.replicas\": 1\n resources:\n requests:\n cpu: 50m\n memory: 0.5G\n limits:\n cpu: 250m\n memory: 0.5G\n prometheus:\n jmx:\n enabled: false\n persistence:\n enabled: false\n\ncp-schema-registry:\n enabled: true\n imageTag: 7.1.3\n fullnameOverride: \"k8kafka-cp-schema-registry\"\n overrideGroupId: k8kafka\n kafka:\n bootstrapServers: \"PLAINTEXT://k8kafka-cp-kafka-headless:9092\"\n resources:\n requests:\n cpu: 50m\n memory: 0.25G\n limits:\n cpu: 250m\n memory: 0.25G\n prometheus:\n jmx:\n enabled: false\n\ncp-kafka-connect:\n enabled: true\n replicaCount: 1\n image: k3d-kpops-registry.localhost:12345/kafka-connect-jdbc\n imageTag: 7.1.3\n fullnameOverride: \"k8kafka-cp-kafka-connect\"\n overrideGroupId: k8kafka\n kafka:\n bootstrapServers: \"PLAINTEXT://k8kafka-cp-kafka-headless:9092\"\n heapOptions: \"-Xms256M -Xmx256M\"\n resources:\n requests:\n cpu: 500m\n memory: 0.25G\n limits:\n cpu: 500m\n memory: 0.25G\n configurationOverrides:\n \"consumer.max.poll.records\": \"10\"\n \"consumer.max.poll.interval.ms\": \"900000\"\n \"config.storage.replication.factor\": \"1\"\n \"offset.storage.replication.factor\": \"1\"\n \"status.storage.replication.factor\": \"1\"\n cp-schema-registry:\n url: http://k8kafka-cp-schema-registry:8081\n prometheus:\n jmx:\n enabled: false\n\ncp-kafka-rest:\n enabled: true\n imageTag: 7.1.3\n fullnameOverride: \"k8kafka-cp-rest\"\n heapOptions: \"-Xms256M -Xmx256M\"\n resources:\n requests:\n cpu: 50m\n memory: 0.25G\n limits:\n cpu: 250m\n memory: 0.5G\n prometheus:\n jmx:\n enabled: false\n\ncp-ksql-server:\n enabled: false\ncp-control-center:\n enabled: false\nStreams Explorer allows examining Apache Kafka data pipelines in a Kubernetes cluster including the inspection of schemas and monitoring of metrics. First, add the Helm repository:
helm repo add streams-explorer https://bakdata.github.io/streams-explorer && \\\nhelm repo update\nBelow you can find an example for the --values ./streams-explorer.yaml file configuring the deployment accordingly. Now, deploy the service:
helm upgrade \\\n --install \\\n --version 0.2.3 \\\n --values ./streams-explorer.yaml \\\n --namespace kpops \\\n streams-explorer streams-explorer/streams-explorer\nstreams-explorer.yaml) An example value configuration for Steams Explorer Helm chart.
imageTag: \"v2.1.2\"\nconfig:\n K8S__deployment__cluster: true\n SCHEMAREGISTRY__url: http://k8kafka-cp-schema-registry.kpops.svc.cluster.local:8081\n KAFKACONNECT__url: http://k8kafka-cp-kafka-connect.kpops.svc.cluster.local:8083\nresources:\n requests:\n cpu: 200m\n memory: 300Mi\n limits:\n cpu: 200m\n memory: 300Mi\nNow we will check if all the pods are running in our namespace. You can list all pods in the namespace with this command:
kubectl --namespace kpops get pods\nThen you should see the following output in your terminal:
NAME READY STATUS RESTARTS AGE\nk8kafka-cp-kafka-connect-8fc7d544f-8pjnt 1/1 Running 0 15m\nk8kafka-cp-zookeeper-0 1/1 Running 0 15m\nk8kafka-cp-kafka-0 1/1 Running 0 15m\nk8kafka-cp-schema-registry-588f8c65db-jdwbq 1/1 Running 0 15m\nk8kafka-cp-rest-6bbfd7b645-nwkf8 1/1 Running 0 15m\nstreams-explorer-54db878c67-s8wbz 1/1 Running 0 15m\nPay attention to the STATUS row. The pods should have a status of Running.
KPOps comes as a PyPI package. You can install it with pip:
pip install kpops\ndestroy: Removes Kubernetes resources.reset: Runs destroy, resets the states of Kafka Streams apps and resets offsets to zero.clean: Runs reset and removes all Kafka resources.The kpops CLI can be used to destroy a pipeline that was previously deployed with KPOps. In case that doesn't work, the pipeline can always be taken down manually with helm (see section Infrastructure).
Export environment variables.
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nNavigate to the examples folder. Replace the <name-of-the-example-directory> with the example you want to tear down. For example the atm-fraud-detection.
Remove the pipeline
# Uncomment 1 line to either destroy, reset or clean.\n\n# kpops destroy <name-of-the-example-directory>/pipeline.yaml \\\n# kpops reset <name-of-the-example-directory>/pipeline.yaml \\\n# kpops clean <name-of-the-example-directory>/pipeline.yaml \\\n--config <name-of-the-example-directory>/config.yaml \\\n--execute\nDelete namespace:
kubectl delete namespace kpops\nNote
In case kpops destroy is not working one can uninstall the pipeline services one by one. This is equivalent to running kpops destroy. In case a clean uninstall (like the one kpops clean does) is needed, one needs to also delete the topics and schemas created by deployment of the pipeline.
Delete local cluster:
k3d cluster delete kpops\nDelete local registry:
k3d registry delete k3d-kpops-registry.localhost\nKPOps automatically infers the component type from the class name. Therefore, the type and schema_type attributes can be removed from your custom components. By convention the type would be the lower, and kebab cased name of the class.
class MyCoolStreamApp(StreamsApp):\n- type = \"my-cool-stream-app\"\n+ ...\nBecause of this new convention producer has been renamed to producer-app. This must be addressed in your pipeline.yaml and defaults.yaml.
- producer:\n+ producer-app:\n app:\n streams:\n outputTopic: output_topic\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\nIn the to section these have changed:
outputrole is set, type is inferred to be extraerror needs to be defined explicitly to:\n topics:\n ${pipeline_name}-topic-1:\n- type: extra\n role: \"role-1\"\n ...\n ${pipeline_name}-topic-2:\n- type: output\n ...\n ${pipeline_name}-topic-3:\n type: error\n ...\nIn the from section these have changed:
inputinput-pattern type is replaced by patternrole is set, type is inferred to be extrarole is set, type is explicitly set to pattern, this would be inferred type extra-pattern from:\n topics:\n ${pipeline_name}-input-topic:\n- type: input\n ...\n ${pipeline_name}-extra-topic:\n- type: extra\n role: topic-role\n ...\n ${pipeline_name}-input-pattern-topic:\n- type: input-pattern\n+ type: pattern\n ...\n ${pipeline_name}-extra-pattern-topic:\n- type: extra-pattern\n+ type: pattern\n role: some-role\n ...\nAll the internal KPOps models are now snake_case, and only Helm/Kubernetes values require camel casing. You can find an example of a pipeline.yaml in the following. Notice that the app section here remains untouched.
...\ntype: streams-app\n name: streams-app\n namespace: namespace\n app:\n streams:\n brokers: ${brokers}\n schemaRegistryUrl: ${schema_registry_url}\n autoscaling:\n consumerGroup: consumer-group\n lagThreshold: 0\n enabled: false\n pollingInterval: 30\n\n to:\n topics:\n ${pipeline_name}-output-topic:\n type: error\n- keySchema: key-schema\n+ key_schema: key-schema\n- valueSchema: value-schema\n+ value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs:\n cleanup.policy: compact\n models:\n model: model\n prefix: ${pipeline_name}-\n- repoConfig:\n+ repo_config:\n- repositoryName: bakdata-streams-bootstrap\n+ repository_name: bakdata-streams-bootstrap\n url: https://bakdata.github.io/streams-bootstrap/\n- repoAuthFlags:\n+ repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"1.0.4\"\n...\nIf you are using the KubernetesApp class to define your own Kubernetes resource to deploy, the abstract function get_helm_chart that returns the chart for deploying the app using Helm is now a Python property and renamed to helm_chart.
class MyCoolApp(KubernetesApp):\n\n+ @property\n @override\n- def get_helm_chart(self) -> str:\n+ def helm_chart(self) -> str:\n return \"./charts/charts-folder\"\nSince you can pass a comma separated string of broker address, the broker field in KPOps is now plural. The pluralization has affected multiple areas:
"}, {"location": "user/migration-guide/v1-v2/#configyaml", "title": "config.yaml", "text": " environment: development\n- broker: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n+ brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n kafka_connect_host: \"http://localhost:8083\"\n kafka_rest_host: \"http://localhost:8082\"\n schema_registry_url: \"http://localhost:8081\"\nThe variable is now called brokers.
...\n app:\n streams:\n- brokers: ${broker}\n+ brokers: ${brokers}\n schemaRegistryUrl: ${schema_registry_url}\n nameOverride: override-with-this-name\n imageTag: \"1.0.0\"\n...\nPreviously, if you set the environment variable KPOPS_KAFKA_BROKER, you need to replace that now with KPOPS_KAFKA_BROKERS.
Jump to the summary
"}, {"location": "user/migration-guide/v2-v3/#use-hash-and-trim-long-helm-release-names-instead-of-only-trimming", "title": "Use hash and trim long Helm release names instead of only trimming", "text": "KPOps handles long (more than 53 characters) Helm releases names differently. Helm will not find your (long) old release names anymore. Therefore, it is recommended that you should once destroy your pipeline with KPOps v2 to remove old Helm release names. After a clean destroy, re-deploy your pipeline with the KPOps v3.
For example if you have a component with the Helm release name example-component-name-too-long-fake-fakefakefakefakefake. The new release name will shorten the original name to 53 characters and then replace the last 6 characters of the trimmed name with the first 5 characters of the result of SHA-1(helm_release_name).
example-component-name-too-long-fake-fakefakef-0a7fc ----> 53 chars\n---------------------------------------------- -----\n ^Shortened helm_release_name ^first 5 characters of SHA1(helm_release_name)\nAll Helm-specific parts of the built-in KubernetesApp have been extracted to a new child component that is more appropriately named HelmApp. It has to be renamed in your existing pipeline defintions and custom components module.
-- type: kubernetes-app\n+- type: helm-app\n name: foo\n- from kpops.components import KubernetesApp\n+ from kpops.components import HelmApp\n\n\n- class CustomHelmApp(KubernetesApp):\n+ class CustomHelmApp(HelmApp):\n ...\nPreviously the default KafkaApp component configured the streams-bootstrap Helm Charts. Now, this component is no longer tied to Helm (or Kubernetes). Instead, there is a new StreamsBootstrap component that configures the Helm Chart repository for the components that use it, e.g. StreamsApp and ProducerApp. If you are using non-default values for the Helm Chart repository or version, it has to be updated as shown below.
kafka-app:\n app:\n streams: ...\n\n+ streams-bootstrap:\n repo_config: ...\n version: ...\nInternally, the Kafka Connector resetter is now its own standard HelmApp, removing a lot of the shared code. It is configured using the resetter_namespace (formerly namespace) and resetter_values attributes.
kafka-connector:\n- namespace: my-namespace\n+ resetter_namespace: my-namespace\nThe breaking changes target the config.yaml file:
The schema_registry_url is replaced with schema_registry.url (default http://localhost:8081) and schema_registry.enabled (default false).
kafka_rest_host is renamed to kafka_rest.url (default http://localhost:8082).
kafka_connect_host is replaced with kafka_connect.url (default http://localhost:8083).
brokers is renamed to kafka_brokers.
The environment variable names of these config fields changed respectively. Please refer to the environment variables documentation page to see the newest changes.
"}, {"location": "user/migration-guide/v2-v3/#configyaml", "title": "config.yaml", "text": " environment: development\n- brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n- kafka_rest_host: \"http://my-custom-rest.url:8082\"\n- kafka_connect_host: \"http://my-custom-connect.url:8083\"\n- schema_registry_url: \"http://my-custom-sr.url:8081\"\n+ kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n+ kafka_rest:\n+ url: \"http://my-custom-rest.url:8082\"\n+ kafka_connect:\n+ url: \"http://my-custom-connect.url:8083\"\n+ schema_registry:\n+ enabled: true\n+ url: \"http://my-custom-sr.url:8081\"\nThe variable is now called kafka_brokers.
...\n app:\n streams:\n- brokers: ${brokers}\n+ brokers: ${kafka_brokers}\n schemaRegistryUrl: ${schema_registry_url}\n nameOverride: override-with-this-name\n imageTag: \"1.0.0\"\n...\nWarning
The previous CLI parameters have been removed.
The options for a custom components_module and pipeline_base_dir are now global settings, defined in config.yaml.
kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n environment: development\n+ components_module: components\n+ pipeline_base_dir: pipelines\nThe location of the GitHub action has changed, and it's now available directly as bakdata/kpops.
You'll need to change it in your GitHub CI workflows.
steps:\n - name: kpops deploy\n- uses: bakdata/kpops/actions/kpops-runner@main\n+ uses: bakdata/kpops@main\n with:\n command: deploy --execute\n # ...\nSpecifying the environment is no longer mandatory. If not defined, only the global files will be used.
environment is no longer specified in config.yaml. Instead, it can be either set via the CLI flag --environment or with the environment variable KPOPS_ENVIRONMENT.
The --config flag in the CLI now points to the directory that contains config*.yaml files. The files to be used are resolved based on the provided (or not) environment.
- environment: development\n kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n schema_registry:\n enabled: true\n url: \"http://my-custom-sr.url:8081\"\n.", "text": "The delimiter in the substitution variables is changed to ..
steps:\n - type: scheduled-producer\n app:\n labels:\n- app_type: \"${component_type}\"\n- app_name: \"${component_name}\"\n- app_schedule: \"${component_app_schedule}\"\n+ app_type: \"${component.type}\"\n+ app_name: \"${component.name}\"\n+ app_schedule: \"${component.app.schedule}\"\ntopic_name_config:\n- default_error_topic_name: \"${pipeline_name}-${component_name}-dead-letter-topic\"\n- default_output_topic_name: \"${pipeline_name}-${component_name}-topic\"\n+ default_error_topic_name: \"${pipeline_name}-${component.name}-dead-letter-topic\"\n+ default_output_topic_name: \"${pipeline_name}-${component.name}-topic\"\nThe template method of every pipeline component has been renamed to manifest as it is no longer strictly tied to Helm template. Instead, it can be used to render the final resources of a component, such as Kubernetes manifests.
There is also a new kpops manifest command replacing the existing kpops generate --template flag.
If you're using this functionality in your custom components, it needs to be updated.
from kpops.components.base_components.models.resource import Resource\n\n @override\n- def template(self) -> None:\n+ def manifest(self) -> Resource:\n \"\"\"Render final component resources, e.g. Kubernetes manifests.\"\"\"\n return [] # list of manifests\nThe global configuration variables are now namespaced under the config key, such as ${config.kafka_brokers}, ${config.schema_registry.url}. Same with pipeline variables, e.g. ${pipeline_name} \u2192 ${pipeline.name}. This would make it more uniform with the existing ${component.<key>} variables.
name: kafka-app\n- prefix: ${pipeline_name}-\n+ prefix: ${pipeline.name}-\n app:\n streams:\n- brokers: ${kafka_brokers}\n- schemaRegistryUrl: ${schema_registry.url}\n+ brokers: ${config.kafka_brokers}\n+ schemaRegistryUrl: ${config.schema_registry.url}\nWarning
Helm will not find your (long) old release names anymore.
defaults.yaml kafka-app:\n app:\n streams: ...\n\n+ streams-bootstrap:\n repo_config: ...\n version: ...\n- - type: kubernetes-app\n+ - type: helm-app\n ...\n - type: kafka-app\n app:\n- brokers: ${brokers}\n+ brokers: ${config.kafka_brokers}\n labels:\n- app_schedule: \"${component_app_schedule}\"\n+ app_schedule: \"${component.app.schedule}\"\n ...\n - type: kafka-connector\n- namespace: my-namespace\n+ resetter_namespace: my-namespace\n ...\n- environment: development\n\n+ components_module: components\n\n+ pipeline_base_dir: pipelines\n\n- brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n+ kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n\n- kafka_rest_host: \"http://my-custom-rest.url:8082\"\n+ kafka_rest:\n+ url: \"http://my-custom-rest.url:8082\"\n\n- kafka_connect_host: \"http://my-custom-connect.url:8083\"\n+ kafka_connect:\n+ url: \"http://my-custom-connect.url:8083\"\n\n- schema_registry_url: \"http://my-custom-sr.url:8081\"\n+ schema_registry:\n+ enabled: true\n+ url: \"http://my-custom-sr.url:8081\"\n\n topic_name_config:\n- default_error_topic_name: \"${pipeline_name}-${component_name}-dead-letter-topic\"\n+ default_error_topic_name: \"${pipeline.name}-${component.name}-dead-letter-topic\"\n ...\n- from kpops.components import KubernetesApp\n+ from kpops.components import HelmApp\n+ from kpops.components.base_components.models.resource import Resource\n\n- class CustomHelmApp(KubernetesApp):\n+ class CustomHelmApp(HelmApp):\n\n @override\n- def template(self) -> None:\n+ def manifest(self) -> Resource:\n \"\"\"Render final component resources, e.g. Kubernetes manifests.\"\"\"\n return [] # list of manifests\n ...\n steps:\n - name: ...\n- uses: bakdata/kpops/actions/kpops-runner@main\n+ uses: bakdata/kpops@main\n ...\nWarning
The --defaults flag is removed
It is possible now to use multiple default values. The defaults.yaml (or defaults_<env>.yaml) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single pipeline.yaml file. KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the pipeline_base_dir is defined.
For example, imagine the following folder structure:
\u2514\u2500 pipelines\n \u2514\u2500\u2500 distributed-defaults\n \u251c\u2500\u2500 defaults.yaml\n \u251c\u2500\u2500 defaults_dev.yaml\n \u2514\u2500\u2500 pipeline-deep\n \u251c\u2500\u2500 defaults.yaml\n \u2514\u2500\u2500 pipeline.yaml\nThe pipeline_base_dir is configured to pipelines. Now if we generate this pipeline with the following command:
kpops generate \\\n --environment dev\n ./pipelines/distributed-defaults/pipeline-deep/pipeline.yaml\nThe defaults would be picked in the following order (high to low priority):
./pipelines/distributed-defaults/pipeline-deep/defaults.yaml./pipelines/distributed-defaults/defaults_dev.yaml./pipelines/distributed-defaults/defaults.yamlThe deepest defaults.yaml file in the folder hierarchy (i.e., the closest one to the pipeline.yaml) overwrites the higher-level defaults' values.
The global timeout setting has been removed. Instead, an individual timeout can be set for each external service. The default is 30 seconds.
- timeout: 300\n\n kafka_rest:\n url: \"http://my-custom-rest.url:8082\"\n+ timeout: 30\n kafka_connect:\n url: \"http://my-custom-connect.url:8083\"\n+ timeout: 30\n schema_registry:\n enabled: true\n url: \"http://my-custom-sr.url:8081\"\n+ timeout: 30\nKPOps can now deploy multiple pipelines in a single command. It is possible to pass one or many pipeline.yaml files or pass a directory with many pipeline.yaml files within it.
The environment variable KPOPS_PIPELINE_PATH is changed to KPOPS_PIPELINE_PATHS.
Read more:
KPops Python API is now stable and separated from the CLI! \ud83c\udf89
"}, {"location": "user/migration-guide/v6-v7/", "title": "Migrate from V6 to V7", "text": ""}, {"location": "user/migration-guide/v6-v7/#automatic-loading-of-namespaced-custom-components", "title": "Automatic loading of namespaced custom components", "text": "KPOps is now distributed as a Python namespace package (as defined by PEP 420). This allows us to standardize the namespace kpops.components for both builtin and custom pipeline components.
As a result of the restructure, some imports need to be adjusted:
KPOps Python API
- import kpops\n+ import kpops.api as kpops\nbuiltin KPOps components
- from kpops.components import (\n- HelmApp,\n- KafkaApp,\n- KafkaConnector,\n- KafkaSinkConnector,\n- KafkaSourceConnector,\n- KubernetesApp,\n- StreamsBootstrap,\n- ProducerApp,\n- StreamsApp,\n- PipelineComponent,\n- StreamsApp,\n- ProducerApp,\n- )\n+ from kpops.components.base_components import (\n+ HelmApp,\n+ KafkaApp,\n+ KafkaConnector,\n+ KafkaSinkConnector,\n+ KafkaSourceConnector,\n+ KubernetesApp,\n+ PipelineComponent,\n+ )\n+ from kpops.components.streams_bootstrap import (\n+ StreamsBootstrap,\n+ StreamsApp,\n+ ProducerApp,\n+ )\n- components_module: components\n- components/__init__.py\n+ kpops/components/custom/__init__.py\nThe app attribute of the builtin KPOps components has been renamed to better differentiate them. Both your pipeline.yaml and defaults.yaml files have to be updated, e.g.:
kubernetes-app:\n- app: {}\n+ values: {}\n\n helm-app:\n- app: {}\n+ values: {}\n\n kafka-app:\n- app: {}\n+ values: {}\n\n streams-app:\n- app: {}\n+ values: {}\n\n producer-app:\n- app: {}\n+ values: {}\n\n kafka-connector:\n- app: {}\n+ config: {}\n\n kafka-source-connector:\n- app: {}\n+ config: {}\n\n kafka-sink-connector:\n- app: {}\n+ config: {}\nBefore v7, the KPOps CLI executed destroy before running reset/clean to ensure the component was destroyed.
This logic has changed. The destroy method is now called within the PipelineComponent's reset/clean.
During migrating to v7, you should check your custom components and see if they override the reset/clean methods. If so, you need to call the supermethod reset/clean to trigger the destroy inside the parent class. Alternatively, if you are implementing the PipelineComponent class, you need to call the destroy method at the beginning of the method.
For example, when creating a custom StreamsApp or ProducerApp (or any other custom component), you must call the supermethod reset/clean to execute the destroy in the parent class. Otherwise, the logic of destroy will not be executed!
class MyStreamsApp(StreamsApp):\n\n @override\n async def clean(self, dry_run: bool) -> None:\n+ await super().clean(dry_run)\n # Some custom clean logic\n # ...\n ```diff\n \n \nclass MyCustomComponent(PipelineComponent):\n \n @override\n async def destroy(self, dry_run: bool) -> None:\n # Some custom destroy logic\n # ...\n\n @override\n async def clean(self, dry_run: bool) -> None:\n+ await super().clean(dry_run)\n # Some custom clean logic\n # ...\nFrom now on KPOps supports streams-bootstrap v3 as its default component. The previous streams-bootstrap version (below 3.x.x) is marked as deprecated and will be removed in a future version of KPOps. If you don't want to migrate your producer or streams app to v3, you should suffix your components with -v2. Here is an example of a pipeline.yaml file.
- - type: producer-app\n+ - type: producer-app-v2\n\n- - type: streams-app\n+ - type: streams-app-v2\n\n# rest of your pipeline\n- class MyStreamsApp(StreamsApp):\n+ class MyStreamsApp(StreamsAppV2):\n ...\nInfo
The streams-boostrap, streams-app, and producer-app now all take the Helm values of streams-bootstrap version 3. You can find these values under the Helm charts documentation or by referring to the Base model definitions.
The keyword role is renamed to label. You need to replace it in your pipeline.yaml, defaults.yaml, and the Python components definition files. Here is a simple example of the defaults.yaml.
streams-app-v2:\n values:\n streams:\n brokers: localhost:9092\n from:\n topics:\n my-labeled-input-topic:\n- role: my-input-topic-label\n+ label: my-input-topic-label\n my-labeled-input-pattern:\n type: pattern\n- role: my-input-topic-labeled-pattern\n+ label: my-input-topic-labeled-pattern\n\n to:\n topics:\n my-labeled-topic-output:\n- role: my-output-topic-label\n+ label: my-output-topic-label\n\n# rest of your pipeline\nThe KafkaApp component now only contains the deployment logic of the stream-bootstrap applications (streams-app, producer-app). It should not be used in the defaults.yaml nor the pipeline.yaml. If you are using it, it should be replaced by streams-bootstrap.
- kafka-app:\n+ streams-bootstrap-v2:\n values:\n streams:\n brokers: 127.0.0.1:9092\n schemaRegistryUrl: 127.0.0.1:8081\nThe kpops manifest command and kpops.manifest() API have been removed.
Resource manifesting is now integrated into the operation commands (deploy, destroy, reset, clean) through the new operation mode feature.
To manifest resources, you can:
--operation-mode manifest when executing kpops commands.KPOPS_OPERATION_MODE environment variable.KPOps now supports generating valid Kubernetes KafkaTopic resources compatible with Strimzi. When using manifest or argo as the operation_mode, you must specify the Strimzi cluster label to ensure the topics are recognized by the deployed Strimzi Topic Operator.
operation_mode: manifest\n\n+ strimzi_topic:\n+ label:\n+ strimzi.io/cluster: my-cluster\n\n# rest of your config\nInfo
Refer to the Strimzi documentation on deploying a standalone topic operator for more details.
"}, {"location": "user/migration-guide/v8-v9/#drop-support-for-python-310", "title": "Drop support for Python 3.10", "text": "KPOps V9 no longer supports Python 3.10. Ensure your environment is running Python 3.11 to 3.12.
"}, {"location": "user/migration-guide/v8-v9/#action-required", "title": "Action Required:", "text": "Upgrade your Python version to a supported version (3.11 or 3.12). Update your virtual environments and CI pipelines to reflect this change.
"}, {"location": "user/references/cli-commands/", "title": "CLI Usage", "text": "Usage:
$ kpops [OPTIONS] COMMAND [ARGS]...\nOptions:
-V, --version: Print KPOps version--install-completion: Install completion for the current shell.--show-completion: Show completion for the current shell, to copy it or customize the installation.--help: Show this message and exit.Commands:
init: Initialize a new KPOps project.generate: Generate enriched pipeline representationdeploy: Deploy pipeline stepsdestroy: Destroy pipeline stepsreset: Reset pipeline stepsclean: Clean pipeline stepsschema: Generate JSON schema.kpops init", "text": "Initialize a new KPOps project.
Usage:
$ kpops init [OPTIONS] PATH\nArguments:
PATH: Path for a new KPOps project. It should lead to an empty (or non-existent) directory. The part of the path that doesn't exist will be created. [required]Options:
--config-include-optional / --no-config-include-optional: Whether to include non-required settings in the generated 'config.yaml' [default: no-config-include-optional]--help: Show this message and exit.kpops generate", "text": "Enrich pipeline steps with defaults. The enriched pipeline is used for all KPOps operations (deploy, destroy, ...).
Usage:
$ kpops generate [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--help: Show this message and exit.kpops deploy", "text": "Deploy pipeline steps
Usage:
$ kpops deploy [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops destroy", "text": "Destroy pipeline steps
Usage:
$ kpops destroy [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops reset", "text": "Reset pipeline steps
Usage:
$ kpops reset [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops clean", "text": "Clean pipeline steps
Usage:
$ kpops clean [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops schema", "text": "Generate JSON schema.
The schemas can be used to enable support for KPOps files in a text editor.
Usage:
$ kpops schema [OPTIONS] SCOPE:{pipeline|defaults|config}\nArguments:
SCOPE:{pipeline|defaults|config}: Scope of the generated schema- pipeline: Schema of PipelineComponents for KPOps pipeline.yaml\n\n- defaults: Schema of PipelineComponents for KPOps defaults.yaml\n\n- config: Schema for KPOps config.yaml [required]\nOptions:
--help: Show this message and exit.We are working towards first-class editor support by providing plugins that work out of the box.
settings.json {\n \"yaml.schemas\": {\n \"https://bakdata.github.io/kpops/4.0/schema/pipeline.json\": [\n \"pipeline.yaml\",\n \"pipeline_*.yaml\"\n ],\n \"https://bakdata.github.io/kpops/4.0/schema/defaults.json\": [\n \"defaults.yaml\",\n \"defaults_*.yaml\"\n ],\n \"https://bakdata.github.io/kpops/4.0/schema/config.json\": [\n \"config.yaml\",\n \"config_*.yaml\"\n ]\n }\n}\nAdvanced usage
It is possible to generate schemas with the kpops schema command. Useful for including custom components or when using a pre-release version of KPOps.
KPOps provides JSON schemas that enable autocompletion and validation for all YAML files that the user must work with.
"}, {"location": "user/references/editor-integration/#supported-files", "title": "Supported files", "text": "pipeline.yamldefaults.yamlconfig.yamlWe provided a GitHub composite action bakdata/kpops that installs and executes KPOps commands with the given parameters.
steps:\n # ...\n # This step is useful for debugging reasons\n - name: Generate Kafka pipeline\n uses: bakdata/kpops@main\n with:\n command: generate\n working-directory: home/my-kpops-root-dir\n pipeline: pipelines/my-pipeline-file.yaml\n kpops-version: 1.2.3\n\n # It is possible to use a pre-release KPOps version from TestPyPI https://test.pypi.org/project/kpops/#history\n - name: Deploy Kafka pipeline\n uses: bakdata/kpops@main\n with:\n command: deploy --execute\n working-directory: home/my-kpops-root-dir\n pipeline: pipelines/my-pipeline-file.yaml\n kpops-version: 1.2.5.dev20230707132709\n # ...\nclean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nClean pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Clean pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def clean_runner(component: PipelineComponent):\n log_action(\"Clean\", component)\n await component.clean(dry_run)\n\n async def async_clean():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(clean_runner, reverse=True)\n await pipeline_tasks\n else:\n for component in reversed(pipeline.components):\n await clean_runner(component)\n\n asyncio.run(async_clean())\ndeploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nDeploy pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def deploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Deploy pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def deploy_runner(component: PipelineComponent):\n log_action(\"Deploy\", component)\n await component.deploy(dry_run)\n\n async def async_deploy():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(deploy_runner)\n await pipeline_tasks\n else:\n for component in pipeline.components:\n await deploy_runner(component)\n\n asyncio.run(async_deploy())\ndestroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nDestroy pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def destroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Destroy pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def destroy_runner(component: PipelineComponent):\n log_action(\"Destroy\", component)\n await component.destroy(dry_run)\n\n async def async_destroy():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(\n destroy_runner, reverse=True\n )\n await pipeline_tasks\n else:\n for component in reversed(pipeline.components):\n await destroy_runner(component)\n\n asyncio.run(async_destroy())\ngenerate(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = False,\n operation_mode: OperationMode = OperationMode.MANAGED,\n) -> Pipeline\nGenerate enriched pipeline representation.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: False
operation_mode How KPOps should operate.
TYPE: OperationMode DEFAULT: MANAGED
Pipeline Generated Pipeline object.
kpops/api/__init__.py def generate(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = False,\n operation_mode: OperationMode = OperationMode.MANAGED,\n) -> Pipeline:\n \"\"\"Generate enriched pipeline representation.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param operation_mode: How KPOps should operate.\n :return: Generated `Pipeline` object.\n \"\"\"\n kpops_config = KpopsConfig.create(\n config, dotenv, environment, verbose, operation_mode\n )\n pipeline = _create_pipeline(pipeline_path, kpops_config, environment)\n log.info(f\"Picked up pipeline '{pipeline_path.parent.name}'\")\n if steps:\n component_names = steps\n log.debug(\n f\"KPOPS_PIPELINE_STEPS is defined with values: {component_names} and filter type of {filter_type.value}\"\n )\n\n predicate = filter_type.create_default_step_names_filter_predicate(\n component_names\n )\n pipeline.filter(predicate)\n log.info(f\"Filtered pipeline:\\n{pipeline.step_names}\")\n return pipeline\ninit(path: Path, config_include_optional: bool = False)\nInitiate a default empty project.
PARAMETER DESCRIPTIONpath Directory in which the project should be initiated.
TYPE: Path
config_include_optional Whether to include non-required settings in the generated config file.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def init(\n path: Path,\n config_include_optional: bool = False,\n):\n \"\"\"Initiate a default empty project.\n\n :param path: Directory in which the project should be initiated.\n :param config_include_optional: Whether to include non-required settings\n in the generated config file.\n \"\"\"\n if not path.exists():\n path.mkdir(parents=False)\n elif next(path.iterdir(), False):\n log.warning(\"Please provide a path to an empty directory.\")\n return\n init_project(path, config_include_optional)\nmanifest_clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_clean(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_clean()\n yield resource\nmanifest_deploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_deploy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_deploy()\n yield resource\nmanifest_destroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_destroy(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_destroy()\n yield resource\nmanifest_reset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]\nkpops/api/__init__.py def manifest_reset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n verbose: bool = True,\n operation_mode: OperationMode = OperationMode.MANIFEST,\n) -> Iterator[tuple[KubernetesManifest, ...]]:\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n operation_mode=operation_mode,\n )\n for component in pipeline.components:\n resource = component.manifest_reset()\n yield resource\nreset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n)\nReset pipeline steps.
PARAMETER DESCRIPTIONpipeline_path Path to pipeline definition yaml file.
TYPE: Path
dotenv Paths to dotenv files.
TYPE: list[Path] | None DEFAULT: None
config Path to the dir containing config.yaml files.
TYPE: Path DEFAULT: Path()
steps Set of steps (components) to apply the command on.
TYPE: set[str] | None DEFAULT: None
filter_type Whether steps should include/exclude the steps.
TYPE: FilterType DEFAULT: INCLUDE
dry_run Whether to dry run the command or execute it.
TYPE: bool DEFAULT: True
environment The environment to generate and deploy the pipeline to.
TYPE: str | None DEFAULT: None
verbose Enable verbose printing.
TYPE: bool DEFAULT: True
parallel Enable or disable parallel execution of pipeline steps.
TYPE: bool DEFAULT: False
kpops/api/__init__.py def reset(\n pipeline_path: Path,\n dotenv: list[Path] | None = None,\n config: Path = Path(),\n steps: set[str] | None = None,\n filter_type: FilterType = FilterType.INCLUDE,\n environment: str | None = None,\n dry_run: bool = True,\n verbose: bool = True,\n parallel: bool = False,\n):\n \"\"\"Reset pipeline steps.\n\n :param pipeline_path: Path to pipeline definition yaml file.\n :param dotenv: Paths to dotenv files.\n :param config: Path to the dir containing config.yaml files.\n :param steps: Set of steps (components) to apply the command on.\n :param filter_type: Whether `steps` should include/exclude the steps.\n :param dry_run: Whether to dry run the command or execute it.\n :param environment: The environment to generate and deploy the pipeline to.\n :param verbose: Enable verbose printing.\n :param parallel: Enable or disable parallel execution of pipeline steps.\n \"\"\"\n pipeline = generate(\n pipeline_path=pipeline_path,\n dotenv=dotenv,\n config=config,\n steps=steps,\n filter_type=filter_type,\n environment=environment,\n verbose=verbose,\n )\n\n async def reset_runner(component: PipelineComponent):\n log_action(\"Reset\", component)\n await component.reset(dry_run)\n\n async def async_reset():\n if parallel:\n pipeline_tasks = pipeline.build_execution_graph(reset_runner, reverse=True)\n await pipeline_tasks\n else:\n for component in reversed(pipeline.components):\n await reset_runner(component)\n\n asyncio.run(async_reset())\n Bases: BaseModel
Pipeline representation.
Source code inkpops/pipeline/__init__.py class Pipeline(BaseModel):\n \"\"\"Pipeline representation.\"\"\"\n\n _component_index: dict[str, PipelineComponent] = {}\n _graph: nx.DiGraph = nx.DiGraph()\n\n model_config = ConfigDict(arbitrary_types_allowed=True)\n\n @property\n def step_names(self) -> list[str]:\n return [step.name for step in self.components]\n\n @computed_field(title=\"Components\")\n @property\n def components(self) -> list[SerializeAsAny[PipelineComponent]]:\n return list(self._component_index.values())\n\n @property\n def last(self) -> PipelineComponent:\n return self.components[-1]\n\n def add(self, component: PipelineComponent) -> None:\n if self._component_index.get(component.id) is not None:\n msg = (\n f\"Pipeline steps must have unique id, '{component.id}' already exists.\"\n )\n raise ValidationError(msg)\n self._component_index[component.id] = component\n self.__add_to_graph(component)\n\n def remove(self, component_id: str) -> None:\n self._component_index.pop(component_id)\n\n def get(self, component_id: str) -> PipelineComponent | None:\n return self._component_index.get(component_id)\n\n def find(self, predicate: ComponentFilterPredicate) -> Iterator[PipelineComponent]:\n \"\"\"Find pipeline components matching a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n :returns: Iterator of components matching the predicate\n \"\"\"\n for component in self.components:\n if predicate(component):\n yield component\n\n def filter(self, predicate: ComponentFilterPredicate) -> None:\n \"\"\"Filter pipeline components using a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n \"\"\"\n for component in self.components:\n # filter out components not matching the predicate\n if not predicate(component):\n self.remove(component.id)\n\n def validate(self) -> None: # pyright: ignore [reportIncompatibleMethodOverride]\n self.__validate_graph()\n\n def to_yaml(self) -> str:\n return yaml.dump(\n self.model_dump(mode=\"json\", by_alias=True, exclude_none=True)[\n \"components\"\n ],\n Dumper=CustomSafeDumper,\n )\n\n def build_execution_graph(\n self,\n runner: Callable[[PipelineComponent], Coroutine[Any, Any, None]],\n /,\n reverse: bool = False,\n ) -> Awaitable[None]:\n async def run_parallel_tasks(\n coroutines: list[Coroutine[Any, Any, None]],\n ) -> None:\n tasks: list[asyncio.Task[None]] = []\n for coro in coroutines:\n tasks.append(asyncio.create_task(coro))\n await asyncio.gather(*tasks)\n\n async def run_graph_tasks(pending_tasks: list[Awaitable[None]]) -> None:\n for pending_task in pending_tasks:\n await pending_task\n\n graph: nx.DiGraph = self._graph.copy() # pyright: ignore[reportAssignmentType, reportGeneralTypeIssues] imprecise type hint in networkx\n\n # We add an extra node to the graph, connecting all the leaf nodes to it\n # in that way we make this node the root of the graph, avoiding backtracking\n root_node = \"root_node_bfs\"\n graph.add_node(root_node)\n\n for node in graph:\n predecessors = list(graph.predecessors(node))\n if not predecessors:\n graph.add_edge(root_node, node)\n\n layers_graph: list[list[str]] = list(nx.bfs_layers(graph, root_node))\n\n sorted_tasks: list[Awaitable[None]] = []\n for layer in layers_graph[1:]:\n if parallel_tasks := self.__get_parallel_tasks_from(layer, runner):\n sorted_tasks.append(run_parallel_tasks(parallel_tasks))\n\n if reverse:\n sorted_tasks.reverse()\n\n return run_graph_tasks(sorted_tasks)\n\n def __getitem__(self, component_id: str) -> PipelineComponent:\n try:\n return self._component_index[component_id]\n except KeyError as exc:\n msg = f\"Component {component_id} not found\"\n raise ValueError(msg) from exc\n\n def __bool__(self) -> bool:\n return bool(self._component_index)\n\n def __iter__(self) -> Iterator[PipelineComponent]: # pyright: ignore [reportIncompatibleMethodOverride]\n yield from self._component_index.values()\n\n def __len__(self) -> int:\n return len(self.components)\n\n def __add_to_graph(self, component: PipelineComponent):\n self._graph.add_node(component.id)\n\n for input_topic in component.inputs:\n self.__add_input(input_topic.id, component.id)\n\n for output_topic in component.outputs:\n self.__add_output(output_topic.id, component.id)\n\n def __add_output(self, topic_id: str, source: str) -> None:\n self._graph.add_node(topic_id)\n self._graph.add_edge(source, topic_id)\n\n def __add_input(self, topic_id: str, target: str) -> None:\n self._graph.add_node(topic_id)\n self._graph.add_edge(topic_id, target)\n\n def __get_parallel_tasks_from(\n self,\n layer: list[str],\n runner: Callable[[PipelineComponent], Coroutine[Any, Any, None]],\n ) -> list[Coroutine[Any, Any, None]]:\n def gen_parallel_tasks():\n for node_in_layer in layer:\n # check if component, skip topics\n if (component := self._component_index.get(node_in_layer)) is not None:\n yield runner(component)\n\n return list(gen_parallel_tasks())\n\n def __validate_graph(self) -> None:\n if not nx.is_directed_acyclic_graph(self._graph):\n msg = \"Pipeline is not a valid DAG.\"\n raise ValueError(msg)\nproperty", "text": "components: list[SerializeAsAny[PipelineComponent]]\nproperty", "text": "last: PipelineComponent\nproperty", "text": "step_names: list[str]\nadd(component: PipelineComponent) -> None\nkpops/pipeline/__init__.py def add(self, component: PipelineComponent) -> None:\n if self._component_index.get(component.id) is not None:\n msg = (\n f\"Pipeline steps must have unique id, '{component.id}' already exists.\"\n )\n raise ValidationError(msg)\n self._component_index[component.id] = component\n self.__add_to_graph(component)\nbuild_execution_graph(\n runner: Callable[\n [PipelineComponent], Coroutine[Any, Any, None]\n ],\n /,\n reverse: bool = False,\n) -> Awaitable[None]\nkpops/pipeline/__init__.py def build_execution_graph(\n self,\n runner: Callable[[PipelineComponent], Coroutine[Any, Any, None]],\n /,\n reverse: bool = False,\n) -> Awaitable[None]:\n async def run_parallel_tasks(\n coroutines: list[Coroutine[Any, Any, None]],\n ) -> None:\n tasks: list[asyncio.Task[None]] = []\n for coro in coroutines:\n tasks.append(asyncio.create_task(coro))\n await asyncio.gather(*tasks)\n\n async def run_graph_tasks(pending_tasks: list[Awaitable[None]]) -> None:\n for pending_task in pending_tasks:\n await pending_task\n\n graph: nx.DiGraph = self._graph.copy() # pyright: ignore[reportAssignmentType, reportGeneralTypeIssues] imprecise type hint in networkx\n\n # We add an extra node to the graph, connecting all the leaf nodes to it\n # in that way we make this node the root of the graph, avoiding backtracking\n root_node = \"root_node_bfs\"\n graph.add_node(root_node)\n\n for node in graph:\n predecessors = list(graph.predecessors(node))\n if not predecessors:\n graph.add_edge(root_node, node)\n\n layers_graph: list[list[str]] = list(nx.bfs_layers(graph, root_node))\n\n sorted_tasks: list[Awaitable[None]] = []\n for layer in layers_graph[1:]:\n if parallel_tasks := self.__get_parallel_tasks_from(layer, runner):\n sorted_tasks.append(run_parallel_tasks(parallel_tasks))\n\n if reverse:\n sorted_tasks.reverse()\n\n return run_graph_tasks(sorted_tasks)\nfilter(predicate: ComponentFilterPredicate) -> None\nFilter pipeline components using a custom predicate.
PARAMETER DESCRIPTIONpredicate Filter function, returns boolean value whether the component should be kept or removed
TYPE: ComponentFilterPredicate
kpops/pipeline/__init__.py def filter(self, predicate: ComponentFilterPredicate) -> None:\n \"\"\"Filter pipeline components using a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n \"\"\"\n for component in self.components:\n # filter out components not matching the predicate\n if not predicate(component):\n self.remove(component.id)\nfind(\n predicate: ComponentFilterPredicate,\n) -> Iterator[PipelineComponent]\nFind pipeline components matching a custom predicate.
PARAMETER DESCRIPTIONpredicate Filter function, returns boolean value whether the component should be kept or removed
TYPE: ComponentFilterPredicate
Iterator[PipelineComponent] Iterator of components matching the predicate
Source code inkpops/pipeline/__init__.py def find(self, predicate: ComponentFilterPredicate) -> Iterator[PipelineComponent]:\n \"\"\"Find pipeline components matching a custom predicate.\n\n :param predicate: Filter function,\n returns boolean value whether the component should be kept or removed\n :returns: Iterator of components matching the predicate\n \"\"\"\n for component in self.components:\n if predicate(component):\n yield component\nget(component_id: str) -> PipelineComponent | None\nkpops/pipeline/__init__.py def get(self, component_id: str) -> PipelineComponent | None:\n return self._component_index.get(component_id)\nremove(component_id: str) -> None\nkpops/pipeline/__init__.py def remove(self, component_id: str) -> None:\n self._component_index.pop(component_id)\nto_yaml() -> str\nkpops/pipeline/__init__.py def to_yaml(self) -> str:\n return yaml.dump(\n self.model_dump(mode=\"json\", by_alias=True, exclude_none=True)[\n \"components\"\n ],\n Dumper=CustomSafeDumper,\n )\nvalidate() -> None\nkpops/pipeline/__init__.py def validate(self) -> None: # pyright: ignore [reportIncompatibleMethodOverride]\n self.__validate_graph()\nAuto generation happens mostly with Git hooks. You can find the lefthook configuration here. These pre-commit hooks call different Python scripts to auto generate code for the documentation.
cli_env_vars.env -- All CLI environment variables in a dotenv file.cli_env_vars.md -- All CLI environment variables in a table.config_env_vars.env -- Almost all pipeline config environment variables in a dotenv file. The script checks for each field in KpopsConfig whether it has an env attribute defined. The script is currently unable to visit the classes of fields like topic_name_config, hence any environment variables defined there would remain unknown to it.config_env_vars.env -- Almost all pipeline config environment variables in a table.variable_substitution.yaml -- A copy of ./tests/pipeline/resources/component-type-substitution/pipeline.yaml used as an example of substitution.Generated by typer-cli from the code in main.py. It is called with Python's subprocess module.
Generates example pipeline.yaml and defaults.yaml for each individual component, stores them and also concatenates them into 1 big pipeline definition and 1 big pipeline defaults definition.
User input
headers/*\\.yaml -- The top of each example. Includes a description comment, type and name. The headers for pipeline.yaml reside in the pipeline-components dir and the defaults.yaml headers reside in the pipeline-defaults dir. The names of the files must be equal to the respective component type.sections/*\\.yaml -- Each YAML file contains a single section (component attribute) definition. The intention is to keep the minimal set of definitions there from which any component definition can be built. The names of the files must be equal to the respective component type and the attribute name. The sections are used for both defaults.yaml and pipeline.yaml generation and reside in the pipeline-components dir.Generated
pipeline-components/dependencies/* Cached information about KPOps componentspipeline_component_dependencies.yaml -- Specifies per component which files in the sections dir should be used for the pipeline.yaml generation.defaults_pipeline_component_dependencies.yaml -- Specifies per component which files in the sections dir should be used for the defaults.yaml generation.kpops_structure.yaml -- Specifies the inheritance hierarchy of the components and what sections exist in each component.pipeline-components/*\\.yaml -- All single-component pipeline definitions and one big (complete) pipeline.yaml that contains all of them.pipeline-defaults/*\\.yaml -- All single-component defaults definitions and one big (complete) defaults.yaml that contains all of them.Welcome! We are glad to have you visit our contributing guide!
If you find any bugs or have suggestions for improvements, please open an issue and optionally a pull request (PR). In the case of a PR, we would appreciate it if you preface it with an issue outlining your goal and means of achieving it.
"}, {"location": "developer/contributing/#git", "title": "git", "text": "We are using git submodules to import the KPOps examples repository. You need to fetch the repository locally on your machine. To do so use this command:
git submodule init\ngit submodule update --recursive\nThis will fetch the resources under the examples folder.
We advise that you stick to our Git hooks for code linting, formatting, and auto-generation of documentation. After you install them using lefthook install they're triggered automatically during git operations, such as commit or checkout. Additionally, you can manually invoke them with lefthook run pre-commit --all-files. Please also install the dprint formatter.
Welcome! We are glad to have you visit our developer guide! If you find any bugs or have suggestions for improvements, please open an issue and optionally a pull request (PR). In the case of a PR, we would appreciate it if you preface it with an issue outlining your goal and means of achieving it.
Find more about our code-style or insights into KPOps' code base here in our developer guide.
Work in progress
The developer guide is still under construction. If you have a question left unanswered here, feel free to ask it by opening an issue.
"}, {"location": "user/changelog/", "title": "Changelog", "text": "All notable changes to this project will be documented in this file.
"}, {"location": "user/changelog/#921-2025-01-15", "title": "9.2.1 - 2025-01-15", "text": ""}, {"location": "user/changelog/#whats-changed", "title": "What's changed", "text": "Full Changelog: https://github.com/bakdata/kpops/compare/9.2.0-dev...9.2.1
"}, {"location": "user/changelog/#920-dev-2025-01-14", "title": "9.2.0-dev - 2025-01-14", "text": ""}, {"location": "user/changelog/#whats-changed_1", "title": "What's changed", "text": "Improve Pyright matcher by @disrupted in #579
Migrate from Poetry to uv by @disrupted in #578
Fix circular imports when running individual tests by @disrupted in #583
Configure Pyright to report import cycles by @disrupted in #585
Fix kpops package build by @disrupted in #588
Fail if streams-boostrap v3 model is instantiated with v2 attribute by @disrupted in #587
Bump version 9.1.0 \u2192 9.2.0-dev by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/9.1.0...9.2.0-dev
"}, {"location": "user/changelog/#910-2025-01-07", "title": "9.1.0 - 2025-01-07", "text": ""}, {"location": "user/changelog/#whats-changed_2", "title": "What's changed", "text": "Update CODEOWNERS by @raminqaf in #572
Update test components to streams-bootstrap v3 by @disrupted in #576
Silence deprecation warnings for streams-bootstrap v2 in tests by @disrupted in #577
Represent multiline strings using YAML block style by @disrupted in #574
Indent sequence items to follow style recommendations by @disrupted in #575
Bump version 9.0.1 \u2192 9.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/9.0.1...9.1.0
"}, {"location": "user/changelog/#901-2024-12-20", "title": "9.0.1 - 2024-12-20", "text": ""}, {"location": "user/changelog/#whats-changed_3", "title": "What's changed", "text": "Add operation-mode documentation to mkdocs index by @raminqaf in #573
Bump version 9.0.0 \u2192 9.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/9.0.0...9.0.1
"}, {"location": "user/changelog/#900-2024-12-20", "title": "9.0.0 - 2024-12-20", "text": ""}, {"location": "user/changelog/#whats-changed_4", "title": "What's changed", "text": "Merge main by @raminqaf
Add topic manifestation of ProducerApps for reset command by @raminqaf in #566
Add documentation for operation-mode in KPOps by @raminqaf in #565
Merge branch 'main' into v9 by @raminqaf
Merge branch 'v9' of github.com:bakdata/kpops into v9 by @raminqaf
Set Python target version to 3.11 by @disrupted
Hide operation_mode from KPOps config by @raminqaf in #571
Add migration guide v8-v9 by @raminqaf in #562
KPOps V9 by @raminqaf in #558
Bump version 8.4.0 \u2192 9.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.4.0...9.0.0
"}, {"location": "user/changelog/#840-2024-12-18", "title": "8.4.0 - 2024-12-18", "text": ""}, {"location": "user/changelog/#whats-changed_5", "title": "What's changed", "text": "Create generic SerializeAsOptional type for Pydantic by @disrupted in #564
Bump version 8.3.2 \u2192 8.4.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.3.2...8.4.0
"}, {"location": "user/changelog/#832-2024-12-17", "title": "8.3.2 - 2024-12-17", "text": ""}, {"location": "user/changelog/#whats-changed_6", "title": "What's changed", "text": "Fix allow optional resources requests and limits by @disrupted in #570
Bump version 8.3.1 \u2192 8.3.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.3.1...8.3.2
"}, {"location": "user/changelog/#831-2024-12-17", "title": "8.3.1 - 2024-12-17", "text": ""}, {"location": "user/changelog/#whats-changed_7", "title": "What's changed", "text": "Fix Kubernetes memory not accepting decimal values by @disrupted in #568
Add ephemeral storage to Kubernetes resource requests and limits by @disrupted in #569
Bump version 8.3.0 \u2192 8.3.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.3.0...8.3.1
"}, {"location": "user/changelog/#830-2024-12-17", "title": "8.3.0 - 2024-12-17", "text": ""}, {"location": "user/changelog/#whats-changed_8", "title": "What's changed", "text": "Merge branch 'main' into v9 by @raminqaf
Drop support for Python 3.10 by @disrupted in #561
Manifest Kubernetes resources for reset command by @raminqaf in #563
Add Kubernetes affinity and tolerations to streams-bootstrap v2 values by @disrupted in #567
Bump version 8.2.0 \u2192 8.3.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.2.0...8.3.0
"}, {"location": "user/changelog/#820-2024-12-12", "title": "8.2.0 - 2024-12-12", "text": ""}, {"location": "user/changelog/#whats-changed_9", "title": "What's changed", "text": "merge by @raminqaf
Manifest toSection with Strimzi KafkaTopic by @raminqaf in #545
Manifest Kubernetes resources for destroy command by @raminqaf in #552
Bump streams-bootstrap to 3.1.0 by @disrupted in #557
Merge branch 'main' into v9 by @raminqaf
Manifest Kubernetes resources for clean command by @raminqaf in #559
Update KPOps example snapshots and fix broken link to defaults.yaml by @raminqaf in #560
Merge branch 'main' into v9 by @raminqaf
Add Pydantic models for Kubernetes Affinity by @disrupted in #555
Bump version 8.1.4 \u2192 8.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.4...8.2.0
"}, {"location": "user/changelog/#814-2024-12-09", "title": "8.1.4 - 2024-12-09", "text": ""}, {"location": "user/changelog/#whats-changed_10", "title": "What's changed", "text": "Fix kpops --version by @disrupted in #551
Trim Helm name override for Producer CronJob to 52 characters by @disrupted in #550
Bump version 8.1.3 \u2192 8.1.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.3...8.1.4
"}, {"location": "user/changelog/#813-2024-12-05", "title": "8.1.3 - 2024-12-05", "text": ""}, {"location": "user/changelog/#whats-changed_11", "title": "What's changed", "text": "Merge branch 'main' of github.com:bakdata/kpops into v9 by @raminqaf
Remove repeated defaults from streams-bootstrap values by @disrupted in #547
Bump version 8.1.2 \u2192 8.1.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.2...8.1.3
"}, {"location": "user/changelog/#812-2024-12-04", "title": "8.1.2 - 2024-12-04", "text": ""}, {"location": "user/changelog/#whats-changed_12", "title": "What's changed", "text": "Introduce KPOps operation and manifest resources for deployment by @raminqaf in #541
Define Pydantic model to representing Kubernetes manifest by @raminqaf in #546
Convert all values of Kafka connector and topic config to string by @disrupted in #544
Bump version 8.1.1 \u2192 8.1.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.1...8.1.2
"}, {"location": "user/changelog/#811-2024-12-02", "title": "8.1.1 - 2024-12-02", "text": ""}, {"location": "user/changelog/#whats-changed_13", "title": "What's changed", "text": "Fix files field value type in Streamsboostrap component by @raminqaf in #542
Fix: Use enum values when dumping models by @raminqaf in #543
Bump version 8.1.0 \u2192 8.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.1.0...8.1.1
"}, {"location": "user/changelog/#810-2024-10-25", "title": "8.1.0 - 2024-10-25", "text": ""}, {"location": "user/changelog/#whats-changed_14", "title": "What's changed", "text": "Upgrade typer to support union types by @raminqaf in #533
Extend StreamsBootstrap model by @raminqaf in #534
Bump version 8.0.1 \u2192 8.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.0.1...8.1.0
"}, {"location": "user/changelog/#801-2024-08-22", "title": "8.0.1 - 2024-08-22", "text": ""}, {"location": "user/changelog/#whats-changed_15", "title": "What's changed", "text": "Fix changelog in docs by @raminqaf in #532
Bump version 8.0.0 \u2192 8.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/8.0.0...8.0.1
"}, {"location": "user/changelog/#800-2024-08-21", "title": "8.0.0 - 2024-08-21", "text": ""}, {"location": "user/changelog/#whats-changed_16", "title": "What's changed", "text": "Make KafkaApp responsible of deploying/cleaning streams bootstrap components (#522) by @raminqaf
Add support for streams-bootstrap v3 (#519) by @raminqaf
Rename role to label (#525) by @raminqaf
Fix Pyright warning about type override without default value (#524) by @disrupted
Remove v3 and suffix old streams bootstrap with v2 (#526) by @raminqaf
KPOps 8.0.0 by @raminqaf in #531
Bump version 7.1.0 \u2192 8.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/7.1.0...8.0.0
"}, {"location": "user/changelog/#710-2024-08-15", "title": "7.1.0 - 2024-08-15", "text": ""}, {"location": "user/changelog/#whats-changed_17", "title": "What's changed", "text": "Improve incomplete type hints by @disrupted in #515
Fallback to user defined model when the validation of cluster model fails by @raminqaf in #521
Fix incorrect parameter type annotation by @disrupted in #523
Update pytest by @disrupted in #527
Replace kubernetes-asyncio with lightkube by @disrupted in #517
Bump version 7.0.0 \u2192 7.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/7.0.0...7.1.0
"}, {"location": "user/changelog/#700-2024-07-23", "title": "7.0.0 - 2024-07-23", "text": ""}, {"location": "user/changelog/#whats-changed_18", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v7 by @disrupted
Call destroy from inside of reset or clean by @raminqaf in #501
clean/reset streams-bootstrap components with cluster values by @raminqaf in #498
Rename app field by @disrupted in #506
Fix circular dependency when running individual tests by @raminqaf
Add tests for global config & handlers by @disrupted
Update examples by @disrupted
Bump version 6.1.0 \u2192 7.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.1.0...7.0.0
"}, {"location": "user/changelog/#610-2024-07-09", "title": "6.1.0 - 2024-07-09", "text": ""}, {"location": "user/changelog/#whats-changed_19", "title": "What's changed", "text": "Add image tag field to streams-bootstrap app values by @raminqaf in #499
Automatic loading of namespaced custom components by @disrupted in #500
Improve dataclass instance check by @disrupted in #507
Delete ignored keys from diff by @disrupted in #510
Bump version 6.0.2 \u2192 6.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.0.2...6.1.0
"}, {"location": "user/changelog/#602-2024-07-04", "title": "6.0.2 - 2024-07-04", "text": ""}, {"location": "user/changelog/#whats-changed_20", "title": "What's changed", "text": "Update codeowners by @disrupted in #504
Generate developer docs for Python API by @sujuka99 in #503
Bump version 6.0.1 \u2192 6.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.0.1...6.0.2
"}, {"location": "user/changelog/#601-2024-06-12", "title": "6.0.1 - 2024-06-12", "text": ""}, {"location": "user/changelog/#whats-changed_21", "title": "What's changed", "text": "Fix connector resetter offset topic by @disrupted in #497
Bump version 6.0.0 \u2192 6.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/6.0.0...6.0.1
"}, {"location": "user/changelog/#600-2024-06-06", "title": "6.0.0 - 2024-06-06", "text": ""}, {"location": "user/changelog/#whats-changed_22", "title": "What's changed", "text": "KPOps 6.0.0 by @raminqaf in #496
Bump version 5.1.1 \u2192 6.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.1.1...6.0.0
"}, {"location": "user/changelog/#511-2024-05-22", "title": "5.1.1 - 2024-05-22", "text": ""}, {"location": "user/changelog/#whats-changed_23", "title": "What's changed", "text": "Add YAML separator (---) to stdout by @raminqaf in #491
Bump version 5.1.0 \u2192 5.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.1.0...5.1.1
"}, {"location": "user/changelog/#510-2024-05-22", "title": "5.1.0 - 2024-05-22", "text": ""}, {"location": "user/changelog/#whats-changed_24", "title": "What's changed", "text": "Add computed field for Helm release name and name override by @disrupted in #490
Bump version 5.0.1 \u2192 5.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.0.1...5.1.0
"}, {"location": "user/changelog/#501-2024-05-15", "title": "5.0.1 - 2024-05-15", "text": ""}, {"location": "user/changelog/#whats-changed_25", "title": "What's changed", "text": "Fix missing await on Kubernetes API by @raminqaf in #488
Bump version 5.0.0 \u2192 5.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/5.0.0...5.0.1
"}, {"location": "user/changelog/#500-2024-05-02", "title": "5.0.0 - 2024-05-02", "text": ""}, {"location": "user/changelog/#whats-changed_26", "title": "What's changed", "text": "Update examples for v4 by @disrupted in #486
Allow custom timeout for external services by @disrupted in #485
Bump version 4.2.1 \u2192 5.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.2.1...5.0.0
"}, {"location": "user/changelog/#421-2024-04-25", "title": "4.2.1 - 2024-04-25", "text": ""}, {"location": "user/changelog/#whats-changed_27", "title": "What's changed", "text": "Add support for cleaning StatefulSets with PVCs by @raminqaf in #482
Bump version 4.2.0 \u2192 4.2.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.2.0...4.2.1
"}, {"location": "user/changelog/#420-2024-04-25", "title": "4.2.0 - 2024-04-25", "text": ""}, {"location": "user/changelog/#whats-changed_28", "title": "What's changed", "text": "Update Ruff by @disrupted in #475
Improve type annotations for parallel pipeline jobs by @disrupted in #476
Set Pyright to warn on unknown types by @disrupted in #480
Quiet faker debug logs in tests by @disrupted in #483
Add pyright matcher by @sujuka99 in #481
Bump version 4.1.2 \u2192 4.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.1.2...4.2.0
"}, {"location": "user/changelog/#412-2024-03-11", "title": "4.1.2 - 2024-03-11", "text": ""}, {"location": "user/changelog/#whats-changed_29", "title": "What's changed", "text": "fix(docs): Correct from.components.<component-name>.type to input by @raminqaf in #473
Bump version 4.1.1 \u2192 4.1.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.1.1...4.1.2
"}, {"location": "user/changelog/#411-2024-03-11", "title": "4.1.1 - 2024-03-11", "text": ""}, {"location": "user/changelog/#whats-changed_30", "title": "What's changed", "text": "Update httpx by @disrupted in #471
Fix import errors by @sujuka99 in #472
Bump version 4.1.0 \u2192 4.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.1.0...4.1.1
"}, {"location": "user/changelog/#410-2024-03-07", "title": "4.1.0 - 2024-03-07", "text": ""}, {"location": "user/changelog/#whats-changed_31", "title": "What's changed", "text": "Document precedence between env vars and config.yaml by @JakobEdding in #465
Create init command by @sujuka99 in #394
Bump version 4.0.2 \u2192 4.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.0.2...4.1.0
"}, {"location": "user/changelog/#402-2024-03-04", "title": "4.0.2 - 2024-03-04", "text": ""}, {"location": "user/changelog/#whats-changed_32", "title": "What's changed", "text": "Add support for Python 3.12 by @disrupted in #467
Update Pyright by @disrupted in #468
Remove package classifiers that are automatically assigned by Poetry by @disrupted in #469
Reference editor plugin for Neovim in docs by @disrupted in #464
Validate autoscaling mandatory fields when enabled by @raminqaf in #470
Bump version 4.0.1 \u2192 4.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.0.1...4.0.2
"}, {"location": "user/changelog/#401-2024-02-29", "title": "4.0.1 - 2024-02-29", "text": ""}, {"location": "user/changelog/#whats-changed_33", "title": "What's changed", "text": "Set supported Python cutoff to 3.11 by @disrupted in #466
Bump version 4.0.0 \u2192 4.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/4.0.0...4.0.1
"}, {"location": "user/changelog/#400-2024-02-27", "title": "4.0.0 - 2024-02-27", "text": ""}, {"location": "user/changelog/#whats-changed_34", "title": "What's changed", "text": "Distribute defaults across multiple files by @raminqaf in #438
Bump version 3.2.4 \u2192 4.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.4...4.0.0
"}, {"location": "user/changelog/#324-2024-02-26", "title": "3.2.4 - 2024-02-26", "text": ""}, {"location": "user/changelog/#whats-changed_35", "title": "What's changed", "text": "Refactor Kafka topics by @disrupted in #447
Fix docs CI to include the latest changes to a tagged version in the changelog by @sujuka99 in #459
Refactor PipelineGenerator to use component ids by @disrupted in #460
Fix tempfile creation by @sujuka99 in #461
Fix symbolic link to CONTRIBUTING.md and parallel option in action.yaml by @raminqaf in #462
Bump version 3.2.3 \u2192 3.2.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.3...3.2.4
"}, {"location": "user/changelog/#323-2024-02-19", "title": "3.2.3 - 2024-02-19", "text": ""}, {"location": "user/changelog/#whats-changed_36", "title": "What's changed", "text": "Trim and hash Helm name override to 63 characters by @disrupted in #456
Bump version 3.2.2 \u2192 3.2.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.2...3.2.3
"}, {"location": "user/changelog/#322-2024-02-12", "title": "3.2.2 - 2024-02-12", "text": ""}, {"location": "user/changelog/#whats-changed_37", "title": "What's changed", "text": "Fix nested substitution by @sujuka99 in #451
Bump version 3.2.1 \u2192 3.2.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.1...3.2.2
"}, {"location": "user/changelog/#321-2024-02-08", "title": "3.2.1 - 2024-02-08", "text": ""}, {"location": "user/changelog/#whats-changed_38", "title": "What's changed", "text": "Simplify execution graph logic by @disrupted in #446
Fix order of pipeline steps for clean/reset by @disrupted in #450
Fix substitution by @sujuka99 in #449
Fix cleaner inheritance, parent model should be aliased during instantiation by @disrupted in #452
Bump version 3.2.0 \u2192 3.2.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.2.0...3.2.1
"}, {"location": "user/changelog/#320-2024-02-01", "title": "3.2.0 - 2024-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_39", "title": "What's changed", "text": "Improve Sphinx docs highlighting using RST markup by @disrupted in #443
Refactor enrichment using Pydantic model validator by @disrupted in #444
Refactor pipeline filter and add to public API by @disrupted in #405
Bump version 3.1.0 \u2192 3.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.1.0...3.2.0
"}, {"location": "user/changelog/#310-2024-01-30", "title": "3.1.0 - 2024-01-30", "text": ""}, {"location": "user/changelog/#whats-changed_40", "title": "What's changed", "text": "Simplify loading of defaults by @disrupted in #435
Update poetry publish workflow version to latest by @raminqaf in #430
Add support for pipeline steps parallelization by @irux in #312
Add custom PascalCase to snake_case alias generator by @disrupted in #436
Add parallel flag support to kpops runner by @irux in #439
Bump version 3.0.2 \u2192 3.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.0.2...3.1.0
"}, {"location": "user/changelog/#302-2024-01-23", "title": "3.0.2 - 2024-01-23", "text": ""}, {"location": "user/changelog/#whats-changed_41", "title": "What's changed", "text": "Add step for submodule initialization on the docs by @irux in #431
Add message if examples git submodule is not initialized by @disrupted in #432
Update type annotation for deserialized pipeline by @disrupted in #433
Fix Helm diff output by @disrupted in #434
Bump version 3.0.1 \u2192 3.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.0.1...3.0.2
"}, {"location": "user/changelog/#301-2024-01-19", "title": "3.0.1 - 2024-01-19", "text": ""}, {"location": "user/changelog/#whats-changed_42", "title": "What's changed", "text": "Update pydantic dependency by @sujuka99 in #422
Update docs of word-count example for v3 & new folder structure by @disrupted in #423
Move ATM fraud to examples repo by @disrupted in #425
Fix broken doc link by @raminqaf in #427
Add warning log if SR handler is disabled but URL is set by @raminqaf in #428
Add git submodule instructions to the contributing.md by @raminqaf in #429
Bump version 3.0.0 \u2192 3.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/3.0.0...3.0.1
"}, {"location": "user/changelog/#300-2024-01-17", "title": "3.0.0 - 2024-01-17", "text": ""}, {"location": "user/changelog/#whats-changed_43", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v3 by @disrupted
Fix test by @disrupted
Add missing HelmApp docs by @disrupted
Replace black with ruff by @sujuka99 in #365
Add toml formatter to dprint by @sujuka99 in #386
Add malva to dprint by @sujuka99 in #385
Merge branch 'main' of github.com:bakdata/kpops into v3 by @raminqaf
Migrate to Pydantic v2 by @sujuka99 in #347
Allow overriding config files by @sujuka99 in #391
Change substitution variables separator to . by @sujuka99 in #388
Refactor pipeline generator & representation by @disrupted in #392
Define custom components module & pipeline base dir globally by @disrupted in #387
Update KPOps runner with the new options by @raminqaf in #395
Add steps for KubernetesApp->HelmApp to migration guide by @disrupted
Fix KPOps action to get package from testPyPI by @raminqaf in #396
Use hash and trim long Helm release names instead of only trimming by @raminqaf in #390
Refactor Helm nameOverride by @disrupted in #397
Mark component type as computed Pydantic field by @disrupted in #399
Fix missing component type in pipeline schema by @disrupted in #401
Refactor generate template for Python API usage by @disrupted in #380
Generate defaults schema by @disrupted in #402
Update docs for substitution variable usage in v3 by @sujuka99 in #409
Namespace substitution vars by @sujuka99 in #408
Support multiple inheritance for doc generation by @sujuka99 in #406
Refactor streams-bootstrap cleanup jobs as individual HelmApp by @disrupted in #398
Update docs for v3 by @sujuka99 in #416
Refactor Kafka Connector resetter as individual HelmApp by @disrupted in #400
Update tests resources by @sujuka99 in #417
Fix enrichment of nested Pydantic BaseModel by @disrupted in #415
Summarize all breaking changes in diffs at the top of the migration guide by @sujuka99 in #419
Fix wrong Helm release name character limit by @disrupted in #418
KPOps 3.0 by @disrupted in #420
Update release workflow template to support custom changelog file path by @disrupted in #421
Bump version 2.0.11 \u2192 3.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.11...3.0.0
"}, {"location": "user/changelog/#2011-2023-10-24", "title": "2.0.11 - 2023-10-24", "text": ""}, {"location": "user/changelog/#whats-changed_44", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v3 by @disrupted
Create HelmApp component by @disrupted in #370
Fix early exit upon Helm exit code 1 (#376) by @sujuka99
Migrate deprecated mkdocs-material-extensions (#378) by @disrupted
Fix docs setup page list indentation (#377) by @sujuka99
Exclude resources from docs search (#371) by @disrupted
Bump version 2.0.10 \u2192 2.0.11 by @bakdata-bot
Fix early exit upon Helm exit code 1 by @sujuka99 in #376
Migrate deprecated mkdocs-material-extensions by @disrupted in #378
Fix docs setup page list indentation by @sujuka99 in #377
Exclude resources from docs search by @disrupted in #371
Bump version 2.0.10 \u2192 2.0.11 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.10...2.0.11
"}, {"location": "user/changelog/#2010-2023-10-12", "title": "2.0.10 - 2023-10-12", "text": ""}, {"location": "user/changelog/#whats-changed_45", "title": "What's changed", "text": "Fix environment variables documentation generation by @sujuka99 in #362
Merge branch 'main' of github.com:bakdata/kpops into v3 by @raminqaf
Make Kafka REST Proxy & Kafka Connect hosts default and improve Schema Registry config by @raminqaf in #354
Introduce ruff by @sujuka99 in #363
Print details on connector name mismatch error by @disrupted in #369
Enable transparent OS environment lookups from internal environment by @disrupted in #368
Bump version 2.0.9 \u2192 2.0.10 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.9...2.0.10
"}, {"location": "user/changelog/#209-2023-09-19", "title": "2.0.9 - 2023-09-19", "text": ""}, {"location": "user/changelog/#whats-changed_46", "title": "What's changed", "text": "Move GitHub action to repository root by @disrupted in #356
Fix link to kpops-examples by @sujuka99 in #357
Fix Kafka connect config name for deletion by @raminqaf in #361
Bump version 2.0.8 \u2192 2.0.9 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.8...2.0.9
"}, {"location": "user/changelog/#208-2023-09-06", "title": "2.0.8 - 2023-09-06", "text": ""}, {"location": "user/changelog/#whats-changed_47", "title": "What's changed", "text": "Refactor component prefix & name by @disrupted in #326
Remove unnecessary condition during inflate by @disrupted in #328
Fix config.yaml overriding environment variables by @sujuka99 in #353
Bump version 2.0.7 \u2192 2.0.8 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.7...2.0.8
"}, {"location": "user/changelog/#207-2023-08-31", "title": "2.0.7 - 2023-08-31", "text": ""}, {"location": "user/changelog/#whats-changed_48", "title": "What's changed", "text": "Print only rendered templates when --template flag is set by @raminqaf in #350
Add migration guide by @raminqaf in #352
Bump version 2.0.6 \u2192 2.0.7 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.6...2.0.7
"}, {"location": "user/changelog/#206-2023-08-30", "title": "2.0.6 - 2023-08-30", "text": ""}, {"location": "user/changelog/#whats-changed_49", "title": "What's changed", "text": "Simplify deployment with local Helm charts by @raminqaf in #349
Bump version 2.0.5 \u2192 2.0.6 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.5...2.0.6
"}, {"location": "user/changelog/#205-2023-08-30", "title": "2.0.5 - 2023-08-30", "text": ""}, {"location": "user/changelog/#whats-changed_50", "title": "What's changed", "text": "Fix versioning of docs when releasing by @raminqaf in #346
Bump version 2.0.4 \u2192 2.0.5 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.4...2.0.5
"}, {"location": "user/changelog/#204-2023-08-29", "title": "2.0.4 - 2023-08-29", "text": ""}, {"location": "user/changelog/#whats-changed_51", "title": "What's changed", "text": "Exclude abstract components from pipeline schema by @disrupted in #332
Add dprint as the markdown formatter by @raminqaf in #337
Publish pre-release docs for PRs & main branch by @raminqaf in #339
Fix GitHub ref variable for pushing docs to main branch by @raminqaf in #343
Align docs colours by @raminqaf in #345
Bump version 2.0.3 \u2192 2.0.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.3...2.0.4
"}, {"location": "user/changelog/#203-2023-08-24", "title": "2.0.3 - 2023-08-24", "text": ""}, {"location": "user/changelog/#whats-changed_52", "title": "What's changed", "text": "Lint GitHub action by @disrupted in #342
Fix GitHub action error in non-Python projects by @disrupted in #340
Bump version 2.0.2 \u2192 2.0.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.2...2.0.3
"}, {"location": "user/changelog/#202-2023-08-23", "title": "2.0.2 - 2023-08-23", "text": ""}, {"location": "user/changelog/#whats-changed_53", "title": "What's changed", "text": "Add version dropdown to the documentation by @raminqaf in #336
Break the documentation down into smaller subsection by @raminqaf in #329
Bump version 2.0.1 \u2192 2.0.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.1...2.0.2
"}, {"location": "user/changelog/#201-2023-08-22", "title": "2.0.1 - 2023-08-22", "text": ""}, {"location": "user/changelog/#whats-changed_54", "title": "What's changed", "text": "Fix optional flags in GitHub action by @disrupted in #334
Bump version 2.0.0 \u2192 2.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/2.0.0...2.0.1
"}, {"location": "user/changelog/#200-2023-08-17", "title": "2.0.0 - 2023-08-17", "text": ""}, {"location": "user/changelog/#whats-changed_55", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v2 by @disrupted
v2 by @disrupted in #321
Bump version 1.7.2 \u2192 2.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.7.2...2.0.0
"}, {"location": "user/changelog/#172-2023-08-16", "title": "1.7.2 - 2023-08-16", "text": ""}, {"location": "user/changelog/#whats-changed_56", "title": "What's changed", "text": "Merge remote-tracking branch 'origin/main' into v2 by @disrupted
Refactor input/output types by @sujuka99 in #232
Fix editor integration example in docs by @sujuka99 in #273
Add KPOps Runner GitHub Action to the documentation by @raminqaf in #325
Refactor Kafka Connect handler by @disrupted in #322
Remove :type and :rtype from docstrings by @raminqaf in #324
Merge remote-tracking branch 'origin/main' into v2 by @disrupted
Bump version 1.7.1 \u2192 1.7.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.7.1...1.7.2
"}, {"location": "user/changelog/#171-2023-08-15", "title": "1.7.1 - 2023-08-15", "text": ""}, {"location": "user/changelog/#whats-changed_57", "title": "What's changed", "text": "Modularize and autogenerate examples for the documentation by @sujuka99 in #267
Update the variable documentation by @sujuka99 in #266
Merge remote-tracking branch 'origin/main' into v2 by @disrupted
Update docs generation by @disrupted
Bump version 1.7.0 \u2192 1.7.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.7.0...1.7.1
"}, {"location": "user/changelog/#170-2023-08-15", "title": "1.7.0 - 2023-08-15", "text": ""}, {"location": "user/changelog/#whats-changed_58", "title": "What's changed", "text": "Add flag to exclude pipeline steps by @raminqaf in #300
Bump version 1.6.0 \u2192 1.7.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.6.0...1.7.0
"}, {"location": "user/changelog/#160-2023-08-10", "title": "1.6.0 - 2023-08-10", "text": ""}, {"location": "user/changelog/#whats-changed_59", "title": "What's changed", "text": "Refactor handling of Helm flags by @disrupted in #319
Bump version 1.5.0 \u2192 1.6.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.5.0...1.6.0
"}, {"location": "user/changelog/#150-2023-08-10", "title": "1.5.0 - 2023-08-10", "text": ""}, {"location": "user/changelog/#whats-changed_60", "title": "What's changed", "text": "Remove camel case conversion of internal models by @disrupted in #308
Automatically support schema generation for custom components by @disrupted in #307
Derive component type automatically from class name by @disrupted in #309
Refactor Helm wrapper and add --set-file flag by @disrupted in #311
Set default for ToSection topics by @disrupted in #313
Annotate types for ToSection models mapping by @disrupted in #315
Check Poetry lock file consistency by @disrupted in #316
Bump version 1.4.0 \u2192 1.5.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.4.0...1.5.0
"}, {"location": "user/changelog/#140-2023-08-02", "title": "1.4.0 - 2023-08-02", "text": ""}, {"location": "user/changelog/#whats-changed_61", "title": "What's changed", "text": "Update Black by @disrupted in #294
Fix vulnerability in mkdocs-material by @disrupted in #295
Move breaking changes section upper in the change log config by @raminqaf in #287
Order PipelineComponent fields by @disrupted in #290
Migrate requests to httpx by @irux in #302
Validate unique step names by @disrupted in #292
Refactor CLI using dtyper by @disrupted in #306
Bump version 1.3.2 \u2192 1.4.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.3.2...1.4.0
"}, {"location": "user/changelog/#132-2023-07-13", "title": "1.3.2 - 2023-07-13", "text": ""}, {"location": "user/changelog/#whats-changed_62", "title": "What's changed", "text": "Exclude Helm tests from dry-run diff by @raminqaf in #293
Bump version 1.3.1 \u2192 1.3.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.3.1...1.3.2
"}, {"location": "user/changelog/#131-2023-07-11", "title": "1.3.1 - 2023-07-11", "text": ""}, {"location": "user/changelog/#whats-changed_63", "title": "What's changed", "text": "Update codeowners by @disrupted in #281
Reactivate Windows CI by @irux in #255
Downgrade Poetry version on the Windows CI pipeline by @irux in #286
Remove workaround for pipeline steps by @disrupted in #276
Set ANSI theme for output of kpops generate by @disrupted in #289
Bump version 1.3.0 \u2192 1.3.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.3.0...1.3.1
"}, {"location": "user/changelog/#130-2023-07-07", "title": "1.3.0 - 2023-07-07", "text": ""}, {"location": "user/changelog/#whats-changed_64", "title": "What's changed", "text": "Update KPOps runner readme for dev versions by @raminqaf in #279
Add breaking changes section to change log config by @raminqaf in #280
Plural broker field in pipeline config by @raminqaf in #278
Bump version 1.2.4 \u2192 1.3.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.4...1.3.0
"}, {"location": "user/changelog/#124-2023-06-27", "title": "1.2.4 - 2023-06-27", "text": ""}, {"location": "user/changelog/#whats-changed_65", "title": "What's changed", "text": "Update changelog action to contain miscellaneous PRs by @raminqaf in #269
Bump version 1.2.3 \u2192 1.2.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.3...1.2.4
"}, {"location": "user/changelog/#123-2023-06-22", "title": "1.2.3 - 2023-06-22", "text": ""}, {"location": "user/changelog/#whats-changed_66", "title": "What's changed", "text": "Refactor custom component validation & hide field from kpops output by @disrupted in #265
Bump version 1.2.2 \u2192 1.2.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.2...1.2.3
"}, {"location": "user/changelog/#122-2023-06-21", "title": "1.2.2 - 2023-06-21", "text": ""}, {"location": "user/changelog/#whats-changed_67", "title": "What's changed", "text": "Create workflow to lint CI by @disrupted in #260
Fix update docs when releasing by @irux in #261
Rename change log message for uncategorized issues by @raminqaf in #262
Bump version 1.2.1 \u2192 1.2.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.1...1.2.2
"}, {"location": "user/changelog/#121-2023-06-21", "title": "1.2.1 - 2023-06-21", "text": ""}, {"location": "user/changelog/#whats-changed_68", "title": "What's changed", "text": "Fix update docs in release workflow by @irux in #258
Bump version 1.2.0 \u2192 1.2.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.2.0...1.2.1
"}, {"location": "user/changelog/#120-2023-06-21", "title": "1.2.0 - 2023-06-21", "text": ""}, {"location": "user/changelog/#whats-changed_69", "title": "What's changed", "text": "Add background to docs home page by @disrupted in #236
Remove enable option from helm diff by @raminqaf in #235
add --namespace option to Helm template command by @raminqaf in #237
Add missing type annotation for Pydantic attributes by @disrupted in #238
Add helm repo update <repo-name> for Helm >3.7 by @raminqaf in #239
Fix helm version check by @sujuka99 in #242
Refactor variable substitution by @sujuka99 in #198
Fix Helm Version Check by @sujuka99 in #244
Update Poetry version in CI by @sujuka99 in #247
Add pip cache in KPOps runner action by @raminqaf in #249
Check types using Pyright by @disrupted in #251
Remove MyPy by @disrupted in #252
Disable broken Windows CI temporarily by @sujuka99 in #253
Update release and publish workflows by @irux in #254
Fix import from external module by @disrupted in #256
Fix release & publish workflows by @irux in #257
Bump version 1.1.5 \u2192 1.2.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.5...1.2.0
"}, {"location": "user/changelog/#115-2023-06-07", "title": "1.1.5 - 2023-06-07", "text": ""}, {"location": "user/changelog/#whats-changed_70", "title": "What's changed", "text": "Fix links to ATM-fraud defaults by @sujuka99 in #219
Exclude pytest snapshots from pre-commit hook by @sujuka99 in #226
Add Windows support by @irux in #217
Fix missing extra input topics by @disrupted in #230
Bump version 1.1.4 \u2192 1.1.5 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.4...1.1.5
"}, {"location": "user/changelog/#114-2023-05-22", "title": "1.1.4 - 2023-05-22", "text": ""}, {"location": "user/changelog/#whats-changed_71", "title": "What's changed", "text": "Document environment-specific pipeline definitions by @sujuka99 in #210
Set up Helm inside composite action & install latest KPOps by default by @disrupted in #211
Update example pipeline by @sujuka99 in #216
Bump version 1.1.3 \u2192 1.1.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.3...1.1.4
"}, {"location": "user/changelog/#113-2023-05-04", "title": "1.1.3 - 2023-05-04", "text": ""}, {"location": "user/changelog/#whats-changed_72", "title": "What's changed", "text": "Rewrite bash pre-commit hooks in Python by @sujuka99 in #207
Collapse pip install output for GitHub action by @disrupted in #209
Fix misleading error of 'File or directory not found' by @irux in #208
Bump version 1.1.2 \u2192 1.1.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.2...1.1.3
"}, {"location": "user/changelog/#112-2023-04-27", "title": "1.1.2 - 2023-04-27", "text": ""}, {"location": "user/changelog/#whats-changed_73", "title": "What's changed", "text": "Add titles and descriptions to Pydantic model fields by @sujuka99 in #191
Respect object docstring titles by @sujuka99 in #196
Allow manually running the CI by @sujuka99 in #204
Generate schema in CI by @sujuka99 in #197
Add kpops --version command by @disrupted in #206
Bump version 1.1.1 \u2192 1.1.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.1...1.1.2
"}, {"location": "user/changelog/#111-2023-04-17", "title": "1.1.1 - 2023-04-17", "text": ""}, {"location": "user/changelog/#whats-changed_74", "title": "What's changed", "text": "Expose pipeline component by @irux in #192
Bump version 1.1.0 \u2192 1.1.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.1.0...1.1.1
"}, {"location": "user/changelog/#110-2023-04-11", "title": "1.1.0 - 2023-04-11", "text": ""}, {"location": "user/changelog/#whats-changed_75", "title": "What's changed", "text": "Error when running generate with --steps by @sujuka99 in #169
Make schema generation a builtin CLI command by @sujuka99 in #166
Add CLI Usage doc generation to CI by @sujuka99 in #174
Add new badges to readme and improve KubernetesApp docs by @raminqaf in #186
Read from component by @disrupted in #193
Bump version 1.0.1 \u2192 1.1.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.0.1...1.1.0
"}, {"location": "user/changelog/#101-2023-03-23", "title": "1.0.1 - 2023-03-23", "text": ""}, {"location": "user/changelog/#whats-changed_76", "title": "What's changed", "text": "fix(README): documentation leads to user-guide by @sujuka99 in #163
Fix serialization of pathlib.Path type on model export by @disrupted in #168
Bump version 1.0.0 \u2192 1.0.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/1.0.0...1.0.1
"}, {"location": "user/changelog/#100-2023-03-20", "title": "1.0.0 - 2023-03-20", "text": ""}, {"location": "user/changelog/#whats-changed_77", "title": "What's changed", "text": "Update \"What is KPOps\" section to be more catchy by @sujuka99 in #148
Fix broken links in README by @raminqaf in #160
Update CLI usage Reference by @sujuka99 in #152
Fix config.yaml defaults_path being overridden by CLI by @sujuka99 in #151
Bump version 0.12.0 \u2192 1.0.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.12.0...1.0.0
"}, {"location": "user/changelog/#0120-2023-03-15", "title": "0.12.0 - 2023-03-15", "text": ""}, {"location": "user/changelog/#whats-changed_78", "title": "What's changed", "text": "Create documentation for defaults.yaml by @sujuka99 in #146
Rename kafka-connect to kafka-connector by @sujuka99 in #150
Set schema for Kafka Connect config by @disrupted in #132
Fix missing enum keys in Kafka REST proxy response model by @irux in #135
Bump version 0.11.2 \u2192 0.12.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.11.2...0.12.0
"}, {"location": "user/changelog/#0112-2023-03-07", "title": "0.11.2 - 2023-03-07", "text": ""}, {"location": "user/changelog/#whats-changed_79", "title": "What's changed", "text": "Create documentation of KPOps components by @sujuka99 in #112
Helm diff should not render NOTES.txt by @sujuka99 in #130
Improve inflate example & enum comparison in test by @disrupted in #104
Remove duplicate documentation about CLI environment variables by @disrupted in #140
Provide documentation for editor integration by @sujuka99 in #137
Create documentation of config.yaml by @sujuka99 in #138
Refactor loading of component defaults to independent function by @disrupted in #147
Bump version 0.11.1 \u2192 0.11.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.11.1...0.11.2
"}, {"location": "user/changelog/#0111-2023-02-23", "title": "0.11.1 - 2023-02-23", "text": ""}, {"location": "user/changelog/#whats-changed_80", "title": "What's changed", "text": "Skip FromSection for producers by @disrupted in #125
Fix pipeline environment override by @disrupted in #127
Bump version 0.11.0 \u2192 0.11.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.11.0...0.11.1
"}, {"location": "user/changelog/#0110-2023-02-22", "title": "0.11.0 - 2023-02-22", "text": ""}, {"location": "user/changelog/#whats-changed_81", "title": "What's changed", "text": "Full Changelog: https://github.com/bakdata/kpops/compare/0.10.4...0.11.0
"}, {"location": "user/changelog/#0104-2023-02-22", "title": "0.10.4 - 2023-02-22", "text": ""}, {"location": "user/changelog/#whats-changed_82", "title": "What's changed", "text": "Fix enrichment of inflated components by @disrupted in #118
Assign default reviewers through codeowners by @disrupted in #124
Update streams-bootstrap autoscaling config by @disrupted in #122
Bump version 0.10.3 \u2192 0.10.4 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.3...0.10.4
"}, {"location": "user/changelog/#0103-2023-02-16", "title": "0.10.3 - 2023-02-16", "text": ""}, {"location": "user/changelog/#whats-changed_83", "title": "What's changed", "text": "Update screenshot of word count pipeline by @disrupted in #116
Fix topic name substitution of ${component_name} in ToSection by @disrupted in #117
Bump version 0.10.2 \u2192 0.10.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.2...0.10.3
"}, {"location": "user/changelog/#0102-2023-02-15", "title": "0.10.2 - 2023-02-15", "text": ""}, {"location": "user/changelog/#whats-changed_84", "title": "What's changed", "text": "Create deployment documentation for Word Count pipeline by @sujuka99 in #107
Delete leftover pipeline prefix config by @disrupted in #111
Remove poetry run from Quickstart doc by @sujuka99 in #114
Fix incomplete inflate component by @disrupted in #105
Bump version 0.10.1 \u2192 0.10.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.1...0.10.2
"}, {"location": "user/changelog/#0101-2023-02-13", "title": "0.10.1 - 2023-02-13", "text": ""}, {"location": "user/changelog/#whats-changed_85", "title": "What's changed", "text": "Add name to connector dry-run diff by @philipp94831 in #108
Bump version 0.10.0 \u2192 0.10.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.10.0...0.10.1
"}, {"location": "user/changelog/#0100-2023-02-13", "title": "0.10.0 - 2023-02-13", "text": ""}, {"location": "user/changelog/#whats-changed_86", "title": "What's changed", "text": "Fix diff not shown for new Helm releases by @disrupted in #92
Fix ATM fraud example by @disrupted in #95
Fix kpops version in pyproject.toml by @raminqaf in #99
Clean up dry-run logging by @philipp94831 in #100
Refactor integration test by @disrupted in #96
Refactor change calculation by @disrupted in #88
Support printing final Kubernetes resources with kpops generate by @sujuka99 in #69
Set Kafka Connect config name from component by @irux in #98
Add prefix as an option to customize by @irux in #97
Bump version 0.9.0 \u2192 0.10.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.9.0...0.10.0
"}, {"location": "user/changelog/#090-2023-02-03", "title": "0.9.0 - 2023-02-03", "text": ""}, {"location": "user/changelog/#whats-changed_87", "title": "What's changed", "text": "Remove mike set-default command by @raminqaf in #86
Add --create-namespace option to helm by @raminqaf in #91
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.3...0.9.0
"}, {"location": "user/changelog/#083-2023-02-01", "title": "0.8.3 - 2023-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_88", "title": "What's changed", "text": "Correct push flag of mike by @raminqaf in #84
Bump version 0.8.2 \u2192 0.8.3 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.2...0.8.3
"}, {"location": "user/changelog/#082-2023-02-01", "title": "0.8.2 - 2023-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_89", "title": "What's changed", "text": "Add --push flag to mike by @raminqaf in #83
Bump version 0.8.1 \u2192 0.8.2 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.1...0.8.2
"}, {"location": "user/changelog/#081-2023-02-01", "title": "0.8.1 - 2023-02-01", "text": ""}, {"location": "user/changelog/#whats-changed_90", "title": "What's changed", "text": "Tidy user guide by @disrupted in #81
Fix typo and metrics replication factor in Kafka values by @yannick-roeder in #82
Bump version 0.8.0 \u2192 0.8.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.8.0...0.8.1
"}, {"location": "user/changelog/#080-2023-01-30", "title": "0.8.0 - 2023-01-30", "text": ""}, {"location": "user/changelog/#whats-changed_91", "title": "What's changed", "text": "Generate schema for pipeline.yaml and config.yaml by @disrupted in #70
Bump version 0.7.0 \u2192 0.8.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.7.0...0.8.0
"}, {"location": "user/changelog/#070-2023-01-19", "title": "0.7.0 - 2023-01-19", "text": ""}, {"location": "user/changelog/#whats-changed_92", "title": "What's changed", "text": "Update setup.cfg by @sujuka99 in #65
Refactor component configs by @raminqaf in #63
Bump version 0.6.1 \u2192 0.7.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.6.1...0.7.0
"}, {"location": "user/changelog/#061-2023-01-12", "title": "0.6.1 - 2023-01-12", "text": ""}, {"location": "user/changelog/#whats-changed_93", "title": "What's changed", "text": "Refactor Kubernetes app properties by @disrupted in #60
Fix Helm release name trimming of cleanup jobs by @disrupted in #61
Bump version 0.6.0 \u2192 0.6.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.6.0...0.6.1
"}, {"location": "user/changelog/#060-2023-01-09", "title": "0.6.0 - 2023-01-09", "text": ""}, {"location": "user/changelog/#whats-changed_94", "title": "What's changed", "text": "Separate clean, reset, and destroy logic by @raminqaf in #57
Fix trigger CI job once on release workflow by @raminqaf in #58
Fix double push of docs to GitHub pages by @raminqaf in #59
Bump version 0.5.0 \u2192 0.6.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.5.0...0.6.0
"}, {"location": "user/changelog/#050-2023-01-05", "title": "0.5.0 - 2023-01-05", "text": ""}, {"location": "user/changelog/#whats-changed_95", "title": "What's changed", "text": "Fix release version for TestPyPI by @philipp94831 in #48
Change topic_name variable to output_topic_name by @MichaelKora in #50
Improve exception output for integration tests by @disrupted in #51
Refactor usage of Pydantic aliases by @disrupted in #52
Add MyPy plugin for Pydantic by @disrupted in #56
Use component name instead of type to set default output topic name by @MichaelKora in #53
Refactor Helm Wrapper by @raminqaf in #47
Bump version 0.4.1 \u2192 0.5.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.4.1...0.5.0
"}, {"location": "user/changelog/#041-2022-12-22", "title": "0.4.1 - 2022-12-22", "text": ""}, {"location": "user/changelog/#whats-changed_96", "title": "What's changed", "text": "Fix link for getting started in readme by @torbsto in #34
Use new Helm repositories for streams-bootstrap and Kafka Connect resetter by @philipp94831 in #36
Fix spelling of PyPI by @disrupted in #33
Fix typo in docs by @disrupted in #38
Fix broken links in the documentation by @raminqaf in #39
Fix generate connecting to Kafka REST proxy by @disrupted in #41
Bump version 0.4.0 \u2192 0.4.1 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.4.0...0.4.1
"}, {"location": "user/changelog/#040-2022-12-21", "title": "0.4.0 - 2022-12-21", "text": ""}, {"location": "user/changelog/#whats-changed_97", "title": "What's changed", "text": "Add installation instructions to README by @raminqaf in #30
Fix usage of template workflow for Poetry release by @disrupted in #25
Set default value of retain clean jobs flag to false by @raminqaf in #31
Refactor component handlers by @disrupted in #3
Bump version 0.3.0 \u2192 0.3.1 by @bakdata-bot
Bump version 0.3.1 \u2192 0.4.0 by @bakdata-bot
Full Changelog: https://github.com/bakdata/kpops/compare/0.3.0...0.4.0
"}, {"location": "user/changelog/#030-2022-12-21", "title": "0.3.0 - 2022-12-21", "text": ""}, {"location": "user/changelog/#whats-changed_98", "title": "What's changed", "text": "Initial commit by @philipp94831
Add source code of KPOps by @raminqaf in #1
Add GitHub action by @philipp94831 in #2
Update project version by @raminqaf in #4
Update project version by @raminqaf in #5
Remove workflow and add release actions by @raminqaf in #8
Fix env variable in GitHub actions by @raminqaf in #9
Bump version 0.2.2 \u2192 0.2.3 by @bakdata-bot
Remove credential flag from checkout in update docs by @raminqaf in #10
Bump version 0.2.3 \u2192 0.2.4 by @bakdata-bot
Update version in actions readme by @JakobEdding in #11
Bump version 0.2.4 \u2192 0.2.5 by @bakdata-bot
Remove push tag step by @raminqaf in #13
Bump version 0.2.5 \u2192 0.2.6 by @bakdata-bot
Bump version 0.2.6 \u2192 0.3.0 by @bakdata-bot
With a couple of easy commands in the shell, and a pipeline.yaml of under 30 lines, KPOps can not only deploy a Kafka pipeline1 to a Kubernetes cluster, but also reset, clean or destroy it!
- type: producer-app\n name: data-producer\n values:\n image: bakdata/kpops-demo-sentence-producer\n imageTag: \"2.0.0\"\n\n- type: streams-app\n name: word-count-app\n values:\n image: bakdata/kpops-demo-word-count-app\n imageTag: \"2.0.0\"\n replicaCount: 1\n to:\n topics:\n ${output_topic_name}:\n type: output\n configs:\n cleanup.policy: compact\n\n- type: kafka-sink-connector\n name: redis-sink-connector\n config:\n connector.class: com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector\n redis.hosts: redis-headless:6379\n redis.database: 0\n tasks.max: 1\n key.converter: org.apache.kafka.connect.storage.StringConverter\n value.converter: org.apache.kafka.connect.storage.StringConverter\nA Kafka pipeline can consist of consecutive streaming applications, producers, and connectors.\u00a0\u21a9
KPOps reads its global configuration that is unrelated to a pipeline's components from config.yaml.
Consider enabling KPOps' editor integration feature to enjoy the benefits of autocompletion and validation when configuring your pipeline.
To learn about any of the available settings, take a look at the example below.
config.yaml # CONFIGURATION\n#\n# Base directory to the pipelines (default is current working directory)\npipeline_base_dir: .\n# The Kafka brokers address.\n# REQUIRED\nkafka_brokers: \"http://broker1:9092,http://broker2:9092\"\n# Configure the topic name variables you can use in the pipeline definition.\ntopic_name_config:\n # Configures the value for the variable ${output_topic_name}\n default_output_topic_name: ${pipeline.name}-${component.name}\n # Configures the value for the variable ${error_topic_name}\n default_error_topic_name: ${pipeline.name}-${component.name}-error\n# Configuration for Schema Registry.\nschema_registry:\n # Whether the Schema Registry handler should be initialized.\n enabled: false\n # Address of the Schema Registry.\n url: \"http://localhost:8081\"\n# Configuration for the Kafka REST Proxy.\nkafka_rest:\n # Address of the Kafka REST Proxy.\n url: \"http://localhost:8082\"\n# Configuration for Kafka Connect.\nkafka_connect:\n # Address of Kafka Connect.\n url: \"http://localhost:8083\"\n# Flag for `helm upgrade --install`.\n# Create the release namespace if not present.\ncreate_namespace: false\n# Global flags for Helm.\nhelm_config:\n # Name of kubeconfig context (`--kube-context`)\n context: name\n # Run Helm in Debug mode.\n debug: false\n # Kubernetes API version used for Capabilities.APIVersions\n api_version: null\n# Configure Helm Diff.\nhelm_diff_config:\n # Set of keys that should not be checked.\n ignore:\n - name\n - imageTag\n# Whether to retain clean up jobs in the cluster or uninstall the, after\n# completion.\nretain_clean_jobs: false\nEnvironment-specific pipeline definitions
Similarly to defaults, it is possible to have an unlimited amount of additional environment-specific pipeline definitions. The naming convention is the same: add a suffix of the form _{environment} to the filename.
KPOps has a very efficient way of dealing with repeating settings which manifests as defaults.yaml. This file provides the user with the power to set defaults for any and all components, thus omitting the need to repeat the same settings in pipeline.yaml.
See real-world examples for defaults.
An important mechanic of KPOps is that defaults set for a component apply to all components that inherit from it.
It is possible, although not recommended, to add settings that are specific to a component's subclass. An example would be configuring offset_topic under kafka-connector instead of kafka-source-connector.
KPOps allows using multiple default values. The defaults.yaml (or defaults_<env>.yaml) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single pipeline.yaml file. KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the pipeline_base_dir is defined.
The deepest defaults.yaml file in the folder hierarchy (i.e., the closest one to the pipeline.yaml) overwrites the higher-level defaults' values.
It is important to note that defaults_{environment}.yaml overrides only the settings that are explicitly set to be different from the ones in the base defaults file.
Imagine the following folder structure, where the pipeline_base_dir is configured to pipelines:
\u2514\u2500 pipelines\n \u2514\u2500\u2500 distributed-defaults\n \u251c\u2500\u2500 defaults.yaml\n \u251c\u2500\u2500 defaults_dev.yaml\n \u2514\u2500\u2500 pipeline-deep\n \u251c\u2500\u2500 defaults.yaml\n \u2514\u2500\u2500 pipeline.yaml\nKPOps picks up the defaults in the following order (high to low priority):
./pipelines/distributed-defaults/pipeline-deep/defaults.yaml./pipelines/distributed-defaults/defaults_dev.yaml./pipelines/distributed-defaults/defaults.yamlThe defaults codeblocks in this section contain the full set of settings that are specific to the component. If a setting already exists in a parent config, it will not be included in the child's.
defaults.yaml # Base Kubernetes App\n#\n# Parent of: HelmApp\n# Child of: PipelineComponent\nkubernetes-app:\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n values: # required\n image: exampleImage # Example\n debug: false # Example\n commandLine: {} # Example\ndefaults.yaml # StreamsApp component that configures a streams bootstrap app.\n#\n# Child of: KafkaApp\n# More documentation on StreamsApp: https://github.com/bakdata/streams-bootstrap\nstreams-app:\n # No arbitrary keys are allowed under `app`here\n # Allowed configs:\n # https://github.com/bakdata/streams-bootstrap/tree/master/charts/streams-app\n values: # required\n # Streams Bootstrap streams section\n streams: # required, streams-app-specific\n brokers: ${config.kafka_brokers} # required\n schemaRegistryUrl: ${config.schema_registry.url}\n inputTopics:\n - topic1\n - topic2\n outputTopic: output-topic\n inputPattern: input-pattern\n extraInputTopics:\n input_role1:\n - input_topic1\n - input_topic2\n input_role2:\n - input_topic3\n - input_topic4\n extraInputPatterns:\n pattern_role1: input_pattern1\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\n errorTopic: error-topic\n config:\n my.streams.config: my.value\n nameOverride: override-with-this-name # streams-app-specific\n autoscaling: # streams-app-specific\n consumerGroup: consumer-group # required\n lagThreshold: 0 # Average target value to trigger scaling actions.\n enabled: false # Whether to enable auto-scaling using KEDA.\n # This is the interval to check each trigger on.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#pollinginterval\n pollingInterval: 30\n # The period to wait after the last trigger reported active before scaling\n # the resource back to 0. https://keda.sh/docs/2.9/concepts/scaling-deployments/#cooldownperiod\n cooldownPeriod: 300\n # The offset reset policy for the consumer if the the consumer group is\n # not yet subscribed to a partition.\n offsetResetPolicy: earliest\n # This setting is passed to the HPA definition that KEDA will create for a\n # given resource and holds the maximum number of replicas of the target resouce.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#maxreplicacount\n maxReplicas: 1\n # Minimum number of replicas KEDA will scale the resource down to.\n # https://keda.sh/docs/2.7/concepts/scaling-deployments/#minreplicacount\n minReplicas: 0\n # If this property is set, KEDA will scale the resource down to this\n # number of replicas.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#idlereplicacount\n idleReplicas: 0\n topics: # List of auto-generated Kafka Streams topics used by the streams app.\n - topic1\n - topic2\ndefaults.yaml \ndefaults.yaml # Kafka connector\n#\n# Parent of: KafkaSinkConnector, KafkaSourceConnector\n# Child of: PipelineComponent\nkafka-connector:\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs\n config: # required\n tasks.max: 1\n # Overriding Kafka Connect Resetter Helm values. E.g. to override the\n # Image Tag etc.\n resetter_values:\n imageTag: \"1.2.3\"\ndefaults.yaml # Kafka source connector\n#\n# Child of: KafkaConnector\nkafka-source-connector:\n # The source connector has no `from` section\n # from:\n # offset.storage.topic\n # https://kafka.apache.org/documentation/#connect_running\n offset_topic: offset_topic\ndefaults.yaml # Kafka sink connector\n#\n# Child of: KafkaConnector\nkafka-sink-connector:\n # No settings differ from `kafka-connector`\nKPOps supports three operation modes\u2014managed, manifest, and argo. These modes determine how resources are managed and allow users to tailor their deployment strategy.
pipeline.yaml.You can configure the operation mode using one of the following methods:
Command-Line Option: Pass the --operation-mode <OPERATION> flag when running a CLI command. Refer to the CLI commands documentation for more details.
Environment Variable: Set the operation mode by defining the KPOPS_OPERATION_MODE environment variable.
deploy", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode", "title": "Manifest Mode", "text": "Job, Deployment, ConfigMap, and Service resources.KafkaTopic CRDs.Job, Deployment, ConfigMap, and Service resources.sync-wave annotation to ensure Kafka topics are created first (default sync-wave=0) before deploying apps (lower priority sync-wave>0). All components of each sync wave are deployed in parallel by Argo.KafkaTopic CRDs.Job resources configured with ArgoCD PostDelete hooks, ensuring cleanup tasks are executed after ArgoCD application deletion.reset", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode_1", "title": "Manifest Mode", "text": "KafkaTopic CRDs.Job resources for resetting Kafka Streams application states.KafkaTopic CRDs.Job resources without ArgoCD PostDelete hooks, providing a simpler reset process.clean", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode_2", "title": "Manifest Mode", "text": "Job resources for cleaning up temporary resources or artifacts using application container images.clean command is not supported in Argo mode. The clean is instead achieved through cleanup job hooks during the deploy command.destroy", "text": ""}, {"location": "user/core-concepts/operation-mode/#manifest-mode_3", "title": "Manifest Mode", "text": "KafkaTopic CRDs.KafkaTopic CRDs.deploy reset clean destroy Producer Apps Manifest: Generated N/A N/A N/A Argo: Generated Streams Apps Manifest: Generated N/A N/A N/A Argo: Generated Topics Manifest: Generated Manifest: Generated N/A Manifest: Generated Argo: Generated Argo: Generated Argo: Generated Cleanup Jobs Manifest: N/A N/A Manifest: Generated N/A Argo: With PostDelete hooks N/A N/A N/A Reset Jobs Manifest: N/A Manifest: Generated N/A N/A Argo: Without PostDelete hooks"}, {"location": "user/core-concepts/components/helm-app/", "title": "HelmApp", "text": ""}, {"location": "user/core-concepts/components/helm-app/#usage", "title": "Usage", "text": "Can be used to deploy any app in Kubernetes using Helm, for example, a REST service that serves Kafka data.
"}, {"location": "user/core-concepts/components/helm-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # Kubernetes app managed through Helm with an associated Helm chart\n- type: helm-app\n name: helm-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n values: # required\n image: exampleImage # Example\n debug: false # Example\n commandLine: {} # Example\n # Helm repository configuration (optional)\n # If not set the helm repo add will not be called. Useful when using local Helm charts\n repo_config:\n repository_name: bakdata-streams-bootstrap # required\n url: https://bakdata.github.io/streams-bootstrap/ # required\n repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"1.0.0\" # Helm chart version\nDeploy using Helm.
"}, {"location": "user/core-concepts/components/helm-app/#destroy", "title": "destroy", "text": "Uninstall Helm release.
"}, {"location": "user/core-concepts/components/helm-app/#reset", "title": "reset", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/helm-app/#clean", "title": "clean", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/kafka-connector/", "title": "KafkaConnector", "text": "KafkaConnector is a component that deploys Kafka Connectors. Since a connector cannot be different from sink or source it is not recommended to use KafkaConnector for deployment in pipeline.yaml. Instead, KafkaConnector should be used in defaults.yaml to set defaults for all connectors in the pipeline as they can share some common settings.
Subclass of KafkaConnector.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#usage", "title": "Usage", "text": "Lets other systems pull data from Apache Kafka.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#configuration", "title": "Configuration", "text": "pipeline.yaml # Kafka sink connector\n- type: kafka-sink-connector\n name: kafka-sink-connector # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs\n config: # required\n tasks.max: 1\n # Overriding Kafka Connect Resetter Helm values. E.g. to override the\n # Image Tag etc.\n resetter_values:\n imageTag: \"1.2.3\"\nThe associated sink connector is removed from the Kafka Connect cluster.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#reset", "title": "reset", "text": "Reset the consumer group offsets using bakdata's sink resetter.
"}, {"location": "user/core-concepts/components/kafka-sink-connector/#clean", "title": "clean", "text": "Subclass of KafkaConnector.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#usage", "title": "Usage", "text": "Manages source connectors in your Kafka Connect cluster.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#configuration", "title": "Configuration", "text": "pipeline.yaml # Kafka source connector\n- type: kafka-source-connector # required\n name: kafka-source-connector # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n # The source connector has no `from` section\n # from:\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs\n config: # required\n tasks.max: 1\n # Overriding Kafka Connect Resetter Helm values. E.g. to override the\n # Image Tag etc.\n resetter_values:\n imageTag: \"1.2.3\"\n # offset.storage.topic\n # https://kafka.apache.org/documentation/#connect_running\n offset_topic: offset_topic\nRemove the source connector from the Kafka Connect cluster.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#reset", "title": "reset", "text": "Delete state associated with the connector using bakdata's source resetter.
"}, {"location": "user/core-concepts/components/kafka-source-connector/#clean", "title": "clean", "text": "Can be used to create components for any Kubernetes app.
"}, {"location": "user/core-concepts/components/kubernetes-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # Base Kubernetes App\n- type: kubernetes-app\n name: kubernetes-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n values: # required\n image: exampleImage # Example\n debug: false # Example\n commandLine: {} # Example\nDo nothing.
"}, {"location": "user/core-concepts/components/kubernetes-app/#destroy", "title": "destroy", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/kubernetes-app/#reset", "title": "reset", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/kubernetes-app/#clean", "title": "clean", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/overview/", "title": "Overview", "text": "This section explains the different components of KPOps, their usage and configuration in the pipeline definition pipeline.yaml.
flowchart BT\n KubernetesApp --> PipelineComponent\n HelmApp --> KubernetesApp\n StreamsBootstrap --> HelmApp\n StreamsApp --> StreamsBootstrap\n ProducerApp --> StreamsBootstrap\n KafkaConnector --> PipelineComponent\n KafkaSourceConnector --> KafkaConnector\n KafkaSinkConnector --> KafkaConnector\n\n click KubernetesApp \"./../kubernetes-app\"\n click HelmApp \"./../helm-app\"\n click StreamsBootstrap \"./../streams-bootstrap\"\n click StreamsApp \"./../streams-app\"\n click ProducerApp \"./../producer-app\"\n click KafkaConnector \"./../kafka-connector\"\n click KafkaSourceConnector \"./../kafka-source-connector\"\n click KafkaSinkConnector \"./../kafka-sink-connector\"KPOps component hierarchy
"}, {"location": "user/core-concepts/components/producer-app/", "title": "ProducerApp", "text": "Subclass of StreamsBootstrap.
"}, {"location": "user/core-concepts/components/producer-app/#usage", "title": "Usage", "text": "Configures a streams-bootstrap Kafka producer app
"}, {"location": "user/core-concepts/components/producer-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # Holds configuration to use as values for the streams bootstrap producer-app Helm\n# chart.\n# More documentation on ProducerApp:\n# https://github.com/bakdata/streams-bootstrap\n- type: producer-app\n name: producer-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n # from: # While the producer-app does inherit from kafka-app, it does not need a\n # `from` section, hence it does not support it.\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n # Allowed configs:\n # https://github.com/bakdata/streams-bootstrap/tree/master/charts/producer-app\n values: # required\n streams: # required, producer-app-specific\n brokers: ${config.kafka_brokers} # required\n schemaRegistryUrl: ${config.schema_registry.url}\n outputTopic: output_topic\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\n nameOverride: override-with-this-name # kafka-app-specific\n # Helm repository configuration (optional)\n # If not set the helm repo add will not be called. Useful when using local Helm charts\n repo_config:\n repository_name: bakdata-streams-bootstrap # required\n url: https://bakdata.github.io/streams-bootstrap/ # required\n repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"2.12.0\" # Helm chart version\nIn addition to KubernetesApp's deploy:
Uninstall Helm release.
"}, {"location": "user/core-concepts/components/producer-app/#reset", "title": "reset", "text": "Do nothing, producers are stateless.
"}, {"location": "user/core-concepts/components/producer-app/#clean", "title": "clean", "text": "Subclass of and StreamsBootstrap.
"}, {"location": "user/core-concepts/components/streams-app/#usage", "title": "Usage", "text": "Configures a streams-bootstrap Kafka Streams app
"}, {"location": "user/core-concepts/components/streams-app/#configuration", "title": "Configuration", "text": "pipeline.yaml # StreamsApp component that configures a streams bootstrap app.\n# More documentation on StreamsApp: https://github.com/bakdata/streams-bootstrap\n- type: streams-app # required\n name: streams-app # required\n # Pipeline prefix that will prefix every component name. If you wish to not\n # have any prefix you can specify an empty string.\n prefix: ${pipeline.name}-\n from: # Must not be null\n topics: # read from topic\n ${pipeline.name}-input-topic:\n type: input # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra\n ${pipeline.name}-input-pattern-topic:\n type: pattern # Implied to be an input pattern if `role` is undefined\n ${pipeline.name}-extra-pattern-topic:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n components: # read from specific component\n account-producer:\n type: input # Implied when role is NOT specified\n other-producer:\n role: some-role # Implies `type` to be extra\n component-as-input-pattern:\n type: pattern # Implied to be an input pattern if `role` is undefined\n component-as-extra-pattern:\n type: pattern # Implied to be an extra pattern if `role` is defined\n role: some-role\n # Topic(s) into which the component will write output\n to:\n topics:\n ${pipeline.name}-output-topic:\n type: output # Implied when role is NOT specified\n ${pipeline.name}-extra-topic:\n role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined\n ${pipeline.name}-error-topic:\n type: error\n # Currently KPOps supports Avro and JSON schemas.\n key_schema: key-schema # must implement SchemaProvider to use\n value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs: # https://kafka.apache.org/documentation/#topicconfigs\n cleanup.policy: compact\n models: # SchemaProvider is initiated with the values given here\n model: model\n namespace: namespace # required\n # No arbitrary keys are allowed under `app`here\n # Allowed configs:\n # https://github.com/bakdata/streams-bootstrap/tree/master/charts/streams-app\n values: # required\n # Streams Bootstrap streams section\n streams: # required, streams-app-specific\n brokers: ${config.kafka_brokers} # required\n schemaRegistryUrl: ${config.schema_registry.url}\n inputTopics:\n - topic1\n - topic2\n outputTopic: output-topic\n inputPattern: input-pattern\n extraInputTopics:\n input_role1:\n - input_topic1\n - input_topic2\n input_role2:\n - input_topic3\n - input_topic4\n extraInputPatterns:\n pattern_role1: input_pattern1\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\n errorTopic: error-topic\n config:\n my.streams.config: my.value\n nameOverride: override-with-this-name # streams-app-specific\n autoscaling: # streams-app-specific\n consumerGroup: consumer-group # required\n lagThreshold: 0 # Average target value to trigger scaling actions.\n enabled: false # Whether to enable auto-scaling using KEDA.\n # This is the interval to check each trigger on.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#pollinginterval\n pollingInterval: 30\n # The period to wait after the last trigger reported active before scaling\n # the resource back to 0. https://keda.sh/docs/2.9/concepts/scaling-deployments/#cooldownperiod\n cooldownPeriod: 300\n # The offset reset policy for the consumer if the the consumer group is\n # not yet subscribed to a partition.\n offsetResetPolicy: earliest\n # This setting is passed to the HPA definition that KEDA will create for a\n # given resource and holds the maximum number of replicas of the target resouce.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#maxreplicacount\n maxReplicas: 1\n # Minimum number of replicas KEDA will scale the resource down to.\n # https://keda.sh/docs/2.7/concepts/scaling-deployments/#minreplicacount\n minReplicas: 0\n # If this property is set, KEDA will scale the resource down to this\n # number of replicas.\n # https://keda.sh/docs/2.9/concepts/scaling-deployments/#idlereplicacount\n idleReplicas: 0\n topics: # List of auto-generated Kafka Streams topics used by the streams app.\n - topic1\n - topic2\n # Helm repository configuration (optional)\n # If not set the helm repo add will not be called. Useful when using local Helm charts\n repo_config:\n repository_name: bakdata-streams-bootstrap # required\n url: https://bakdata.github.io/streams-bootstrap/ # required\n repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"2.12.0\" # Helm chart version\nIn addition to KubernetesApp's deploy:
Uninstall Helm release.
"}, {"location": "user/core-concepts/components/streams-app/#reset", "title": "reset", "text": "Similar to reset with to additional steps:
Subclass of HelmApp.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#usage", "title": "Usage", "text": "Defines a streams-bootstrap component
Often used in defaults.yaml
Deploy using Helm.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#destroy", "title": "destroy", "text": "Uninstall Helm release.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#reset", "title": "reset", "text": "Do nothing.
"}, {"location": "user/core-concepts/components/streams-bootstrap/#clean", "title": "clean", "text": "Do nothing.
"}, {"location": "user/core-concepts/variables/environment_variables/", "title": "Environment variables", "text": "Environment variables can be set by using the export command in Linux or the set command in Windows.
dotenv files
KPOps currently supports .env files only for variables related to the config. Full support for .env files is on the roadmap. One of the possible ways to use one and export the contents manually is with the following command: export $(xargs < .env). This would work in bash suppose there are no spaces inside the values.
These variables take precedence over the settings in config.yaml. Variables marked as required can instead be set in the global config.
helm upgrade --install. Create the release namespace if not present. create_namespace KPOPS_HELM_CONFIG__CONTEXT False Name of kubeconfig context (--kube-context) helm_config.context KPOPS_HELM_CONFIG__DEBUG False False Run Helm in Debug mode helm_config.debug KPOPS_HELM_CONFIG__API_VERSION False Kubernetes API version used for Capabilities.APIVersions helm_config.api_version KPOPS_HELM_DIFF_CONFIG__IGNORE True Set of keys that should not be checked. helm_diff_config.ignore KPOPS_RETAIN_CLEAN_JOBS False False Whether to retain clean up jobs in the cluster or uninstall the, after completion. retain_clean_jobs KPOPS_STRIMZI_TOPIC False Configuration for Strimzi Kafka Topics. strimzi_topic KPOPS_OPERATION_MODE managed False The operation mode of KPOps (managed, manifest, argo). operation_mode config_env_vars.env Exhaustive list of all config-related environment variables# Global config environment variables\n#\n# The default setup is shown. These variables take precedence over the\n# settings in `config.yaml`. Variables marked as required can instead\n# be set in the global config.\n#\n# pipeline_base_dir\n# Base directory to the pipelines (default is current working\n# directory)\nKPOPS_PIPELINE_BASE_DIR=.\n# kafka_brokers\n# The comma separated Kafka brokers address.\nKPOPS_KAFKA_BROKERS # No default value, required\n# topic_name_config.default_output_topic_name\n# Configures the value for the variable ${output_topic_name}\nKPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME=${pipeline.name}-${component.name}\n# topic_name_config.default_error_topic_name\n# Configures the value for the variable ${error_topic_name}\nKPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME=${pipeline.name}-${component.name}-error\n# schema_registry.enabled\n# Whether the Schema Registry handler should be initialized.\nKPOPS_SCHEMA_REGISTRY__ENABLED=False\n# schema_registry.url\n# Address of the Schema Registry.\nKPOPS_SCHEMA_REGISTRY__URL=http://localhost:8081/\n# schema_registry.timeout\n# Operation timeout in seconds.\nKPOPS_SCHEMA_REGISTRY__TIMEOUT=30\n# kafka_rest.url\n# Address of the Kafka REST Proxy.\nKPOPS_KAFKA_REST__URL=http://localhost:8082/\n# kafka_rest.timeout\n# Operation timeout in seconds.\nKPOPS_KAFKA_REST__TIMEOUT=30\n# kafka_connect.url\n# Address of Kafka Connect.\nKPOPS_KAFKA_CONNECT__URL=http://localhost:8083/\n# kafka_connect.timeout\n# Operation timeout in seconds.\nKPOPS_KAFKA_CONNECT__TIMEOUT=30\n# create_namespace\n# Flag for `helm upgrade --install`. Create the release namespace if\n# not present.\nKPOPS_CREATE_NAMESPACE=False\n# helm_config.context\n# Name of kubeconfig context (`--kube-context`)\nKPOPS_HELM_CONFIG__CONTEXT # No default value, not required\n# helm_config.debug\n# Run Helm in Debug mode\nKPOPS_HELM_CONFIG__DEBUG=False\n# helm_config.api_version\n# Kubernetes API version used for `Capabilities.APIVersions`\nKPOPS_HELM_CONFIG__API_VERSION # No default value, not required\n# helm_diff_config.ignore\n# Set of keys that should not be checked.\nKPOPS_HELM_DIFF_CONFIG__IGNORE # No default value, required\n# retain_clean_jobs\n# Whether to retain clean up jobs in the cluster or uninstall the,\n# after completion.\nKPOPS_RETAIN_CLEAN_JOBS=False\n# strimzi_topic\n# Configuration for Strimzi Kafka Topics.\nKPOPS_STRIMZI_TOPIC # No default value, not required\n# operation_mode\n# The operation mode of KPOps (managed, manifest, argo).\nKPOPS_OPERATION_MODE=managed\nThese variables take precedence over the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags.
Name Default Value Required Description KPOPS_CONFIG_PATH . False Path to the dir containing config.yaml files KPOPS_DOTENV_PATH False Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. KPOPS_ENVIRONMENT False The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). KPOPS_OPERATION_MODE managed False How KPOps should operate. KPOPS_PIPELINE_PATHS True Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. KPOPS_PIPELINE_STEPS False Comma separated list of steps to apply the command on cli_env_vars.env Exhaustive list of all cli-related environment variables# CLI Environment variables\n#\n# The default setup is shown. These variables take precedence over the\n# commands' flags. If a variable is set, the corresponding flag does\n# not have to be specified in commands. Variables marked as required\n# can instead be set as flags.\n#\n# Path to the dir containing config.yaml files\nKPOPS_CONFIG_PATH=.\n# Path to dotenv file. Multiple files can be provided. The files will\n# be loaded in order, with each file overriding the previous one.\nKPOPS_DOTENV_PATH # No default value, not required\n# The environment you want to generate and deploy the pipeline to.\n# Suffix your environment files with this value (e.g.\n# defaults_development.yaml for environment=development).\nKPOPS_ENVIRONMENT # No default value, not required\n# How KPOps should operate.\nKPOPS_OPERATION_MODE=managed\n# Paths to dir containing 'pipeline.yaml' or files named\n# 'pipeline.yaml'.\nKPOPS_PIPELINE_PATHS # No default value, required\n# Comma separated list of steps to apply the command on\nKPOPS_PIPELINE_STEPS # No default value, not required\nKPOps supports the usage of placeholders and environment variables in pipeline definition and defaults.
"}, {"location": "user/core-concepts/variables/substitution/#component-specific-variables", "title": "Component-specific variables", "text": "These variables can be used in a component's definition to refer to any of its attributes, including ones that the user has defined in the defaults.
All of them are prefixed with component. and follow the following form: component.{attribute_name}. If the attribute itself contains attributes, they can be referred to like this: component.{attribute_name}.{subattribute_name}.
- type: scheduled-producer\n values:\n labels:\n app_type: \"${component.type}\"\n app_name: \"${component.name}\"\n app_schedule: \"${component.values.schedule}\"\n commandLine:\n FAKE_ARG: \"fake-arg-value\"\n schedule: \"30 3/8 * * *\"\n- type: converter\n values:\n commandLine:\n CONVERT_XML: true\n resources:\n limits:\n memory: 2G\n requests:\n memory: 2G\n- type: filter\n name: \"filter-app\"\n values:\n labels:\n app_type: \"${component.type}\"\n app_name: \"${component.name}\"\n app_resources_requests_memory: \"${component.values.resources.requests.memory}\"\n ${component.type}: \"${component.values.labels.app_name}-${component.values.labels.app_type}\"\n test_placeholder_in_placeholder: \"${component.values.labels.${component.type}}\"\n commandLine:\n TYPE: \"nothing\"\n resources:\n requests:\n memory: 3G\n replicaCount: 4\n autoscaling:\n minReplicas: 4\n maxReplicas: 4\nThese variables include all fields in the config and refer to the pipeline configuration that is independent of the components.
All such variables are prefixed with config. and are of the same form as the component-specific variables.
Info
error_topic_name is an alias for config.topic_name_config.default_error_topic_name output_topic_name is an alias for config.topic_name_config.default_output_topic_name
Environment variables such as $PATH can be used in the pipeline definition and defaults without any transformation following the form ${ENV_VAR_NAME}. This, of course, includes variables like the ones relevant to the KPOps cli that are exported by the user.
See all KPOps environment variables
"}, {"location": "user/core-concepts/variables/substitution/#pipeline-name-variables", "title": "Pipeline name variables", "text": "These are special variables that refer to the name and path of a pipeline.
${pipeline.name}: Concatenated path of the parent directory where pipeline.yaml is defined in. For instance, ./data/pipelines/v1/pipeline.yaml, here the value for the variable would be data-pipelines-v1.
${pipeline_name_<level>}: Similar to the previous variable, each <level> contains a part of the path to the pipeline.yaml file. Consider the previous example, ${pipeline_name_0} would be data, ${pipeline_name_1} would be pipelines, and ${pipeline_name_2} equals to v1.
ATM fraud is a demo pipeline for ATM fraud detection. The original by Confluent is written in KSQL and outlined in this blogpost. The one used in this example is re-built from scratch using bakdata's streams-bootstrap library.
Completed all steps in the setup.
"}, {"location": "user/examples/atm-fraud-pipeline/#setup-and-deployment", "title": "Setup and deployment", "text": ""}, {"location": "user/examples/atm-fraud-pipeline/#postgresql", "title": "PostgreSQL", "text": "Deploy PostgreSQL using the Bitnami Helm chart: Add the helm repository:
helm repo add bitnami https://charts.bitnami.com/bitnami && \\\nhelm repo update\nInstall the PostgreSQL with helm:
helm upgrade --install -f ./postgresql.yaml \\\n--namespace kpops \\\npostgresql bitnami/postgresql\npostgresql.yaml) auth:\n database: app_db\n enablePostgresUser: true\n password: AppPassword\n postgresPassword: StrongPassword\n username: app1\nprimary:\n persistence:\n enabled: false\n existingClaim: postgresql-data-claim\nvolumePermissions:\n enabled: true\nBefore we deploy the pipeline, we need to forward the ports of kafka-rest-proxy and kafka-connect. Run the following commands in two different terminals.
kubectl port-forward --namespace kpops service/k8kafka-cp-rest 8082:8082\nkubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:8083\nClone the kpops-examples repository and cd into the directory.
Install KPOps pip install -r requirements.txt.
Export environment variables in your terminal:
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nDeploy the pipeline
kpops deploy atm-fraud/pipeline.yaml --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be deployed correctly.
You can use the Streams Explorer to see the deployed pipeline. To do so, port-forward the service in a separate terminal session using the command below:
kubectl port-forward -n kpops service/streams-explorer 8080:8080\nAfter that open http://localhost:8080 in your browser. You should be able to see pipeline shown in the image below:
An overview of ATM fraud pipeline shown in Streams Explorer
Attention
Kafka Connect needs some time to set up the connector. Moreover, Streams Explorer needs a while to scrape the information from Kafka connect. Therefore, it might take a bit until you see the whole graph.
"}, {"location": "user/examples/atm-fraud-pipeline/#teardown-resources", "title": "Teardown resources", "text": ""}, {"location": "user/examples/atm-fraud-pipeline/#postrgresql", "title": "PostrgreSQL", "text": "PostgreSQL can be uninstalled by running the following command:
helm --namespace kpops uninstall postgresql\nExport environment variables in your terminal.
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nRemove the pipeline
kpops clean atm-fraud/pipeline.yaml --verbose --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be destroyed correctly.
Attention
If you face any issues destroying this example see Teardown for manual deletion.
"}, {"location": "user/examples/atm-fraud-pipeline/#common-errors", "title": "Common errors", "text": "deploy fails:clean.deploy --dry-run to avoid havig to clean again. If an error is dropped, start over from step 1.deploy.clean fails:clean.clean fails, follow the steps in teardown.Word-count is a demo pipeline consisting of a producer producing words to Kafka, a Kafka streams app counting the number of times each word occurs, and finally a Redis database into which the words are exported.
"}, {"location": "user/getting-started/quick-start/#what-this-will-demonstrate", "title": "What this will demonstrate", "text": "Completed all steps in the setup.
"}, {"location": "user/getting-started/quick-start/#setup-and-deployment", "title": "Setup and deployment", "text": ""}, {"location": "user/getting-started/quick-start/#redis", "title": "Redis", "text": "Deploy Redis using the Bitnami Helm chart: Add the Helm repository:
helm repo add bitnami https://charts.bitnami.com/bitnami && \\\nhelm repo update\nInstall Redis with Helm:
helm upgrade --install -f ./values-redis.yaml \\\n--namespace kpops \\\nredis bitnami/redis\nvalues-redis.yaml) architecture: standalone\nauth:\n enabled: false\nmaster:\n count: 1\n configuration: \"databases 1\"\nimage:\n tag: 7.0.8\nBefore we deploy the pipeline, we need to forward the ports of kafka-rest-proxy and kafka-connect. Run the following commands in two different terminals.
kubectl port-forward --namespace kpops service/k8kafka-cp-rest 8082:8082\nkubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:8083\nClone the kpops-examples repository and cd into the directory.
Install KPOps pip install -r requirements.txt.
Export environment variables in your terminal:
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nDeploy the pipeline
kpops deploy word-count/pipeline.yaml --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be deployed correctly.
You can use the Streams Explorer to inspect the deployed pipeline. To do so, port-forward the service in a separate terminal session using the command below:
kubectl port-forward -n kpops service/streams-explorer 8080:8080\nAfter that open http://localhost:8080 in your browser.
You should be able to see pipeline shown in the image below:
An overview of Word-count pipeline shown in Streams Explorer
Attention
Kafka Connect needs some time to set up the connector. Moreover, Streams Explorer needs a while to scrape the information from Kafka Connect. Therefore, it might take a bit until you see the whole graph.
"}, {"location": "user/getting-started/quick-start/#teardown-resources", "title": "Teardown resources", "text": ""}, {"location": "user/getting-started/quick-start/#redis_1", "title": "Redis", "text": "Redis can be uninstalled by running the following command:
helm --namespace kpops uninstall redis\nExport environment variables in your terminal.
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nRemove the pipeline
kpops clean word-count/pipeline.yaml --verbose --execute\nNote
You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be destroyed correctly.
Attention
If you face any issues destroying this example see Teardown for manual deletion.
"}, {"location": "user/getting-started/quick-start/#common-errors", "title": "Common errors", "text": "deploy fails:clean.deploy --dry-run to avoid having to clean again. If an error is dropped, start over from step 1.deploy.clean fails:clean.clean fails, follow the steps in teardown.In this part, you will set up KPOps. This includes:
If you don't have access to an existing Kubernetes cluster, this section will guide you through creating a local cluster. We recommend the lightweight Kubernetes distribution k3s for this. k3d is a wrapper around k3s in Docker that lets you get started fast.
You can install k3d with its installation script:
wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.4.6/install.sh | bash\nFor other ways of installing k3d, you can have a look at their installation guide.
The Kafka deployment needs a modified Docker image. In that case the image is built and pushed to a Docker registry that holds it. If you do not have access to an existing Docker registry, you can use k3d's Docker registry:
k3d registry create kpops-registry.localhost --port 12345\nNow you can create a new cluster called kpops that uses the previously created Docker registry:
k3d cluster create kpops --k3s-arg \"--no-deploy=traefik@server:*\" --registry-use k3d-kpops-registry.localhost:12345\nNote
Creating a new k3d cluster automatically configures kubectl to connect to the local cluster by modifying your ~/.kube/config. In case you manually set the KUBECONFIG variable or don't want k3d to modify your config, k3d offers many other options.
You can check the cluster status with kubectl get pods -n kube-system. If all returned elements have a STATUS of Running or Completed, then the cluster is up and running.
Kafka is an open-source data streaming platform. More information about Kafka can be found in the documentation. To deploy Kafka, this guide uses Confluent's Helm chart.
To allow connectivity to other systems Kafka Connect needs to be extended with drivers. You can install a JDBC driver for Kafka Connect by creating a new Docker image:
Create a Dockerfile with the following content:
FROM confluentinc/cp-kafka-connect:7.1.3\n\nRUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0\nBuild and push the modified image to your private Docker registry:
docker build . --tag localhost:12345/kafka-connect-jdbc:7.1.3 && \\\ndocker push localhost:12345/kafka-connect-jdbc:7.1.3\nDetailed instructions on building, tagging and pushing a docker image can be found in Docker docs.
Add Confluent's Helm chart repository and update the index:
helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ && \nhelm repo update\nInstall Kafka, Zookeeper, Confluent's Schema Registry, Kafka Rest Proxy, and Kafka Connect. A single Helm chart installs all five components. Below you can find an example for the --values ./kafka.yaml file configuring the deployment accordingly. Deploy the services:
helm upgrade \\\n --install \\\n --version 0.6.1 \\\n --values ./kafka.yaml \\\n --namespace kpops \\\n --create-namespace \\\n --wait \\\n k8kafka confluentinc/cp-helm-charts\nkafka.yaml) An example value configuration for Confluent's Helm chart. This configuration deploys a single Kafka Broker, a Schema Registry, Zookeeper, Kafka Rest Proxy, and Kafka Connect with minimal resources.
cp-zookeeper:\n enabled: true\n servers: 1\n imageTag: 7.1.3\n heapOptions: \"-Xms124M -Xmx124M\"\n overrideGroupId: k8kafka\n fullnameOverride: \"k8kafka-cp-zookeeper\"\n resources:\n requests:\n cpu: 50m\n memory: 0.2G\n limits:\n cpu: 250m\n memory: 0.2G\n prometheus:\n jmx:\n enabled: false\n\ncp-kafka:\n enabled: true\n brokers: 1\n imageTag: 7.1.3\n podManagementPolicy: Parallel\n configurationOverrides:\n \"auto.create.topics.enable\": false\n \"offsets.topic.replication.factor\": 1\n \"transaction.state.log.replication.factor\": 1\n \"transaction.state.log.min.isr\": 1\n \"confluent.metrics.reporter.topic.replicas\": 1\n resources:\n requests:\n cpu: 50m\n memory: 0.5G\n limits:\n cpu: 250m\n memory: 0.5G\n prometheus:\n jmx:\n enabled: false\n persistence:\n enabled: false\n\ncp-schema-registry:\n enabled: true\n imageTag: 7.1.3\n fullnameOverride: \"k8kafka-cp-schema-registry\"\n overrideGroupId: k8kafka\n kafka:\n bootstrapServers: \"PLAINTEXT://k8kafka-cp-kafka-headless:9092\"\n resources:\n requests:\n cpu: 50m\n memory: 0.25G\n limits:\n cpu: 250m\n memory: 0.25G\n prometheus:\n jmx:\n enabled: false\n\ncp-kafka-connect:\n enabled: true\n replicaCount: 1\n image: k3d-kpops-registry.localhost:12345/kafka-connect-jdbc\n imageTag: 7.1.3\n fullnameOverride: \"k8kafka-cp-kafka-connect\"\n overrideGroupId: k8kafka\n kafka:\n bootstrapServers: \"PLAINTEXT://k8kafka-cp-kafka-headless:9092\"\n heapOptions: \"-Xms256M -Xmx256M\"\n resources:\n requests:\n cpu: 500m\n memory: 0.25G\n limits:\n cpu: 500m\n memory: 0.25G\n configurationOverrides:\n \"consumer.max.poll.records\": \"10\"\n \"consumer.max.poll.interval.ms\": \"900000\"\n \"config.storage.replication.factor\": \"1\"\n \"offset.storage.replication.factor\": \"1\"\n \"status.storage.replication.factor\": \"1\"\n cp-schema-registry:\n url: http://k8kafka-cp-schema-registry:8081\n prometheus:\n jmx:\n enabled: false\n\ncp-kafka-rest:\n enabled: true\n imageTag: 7.1.3\n fullnameOverride: \"k8kafka-cp-rest\"\n heapOptions: \"-Xms256M -Xmx256M\"\n resources:\n requests:\n cpu: 50m\n memory: 0.25G\n limits:\n cpu: 250m\n memory: 0.5G\n prometheus:\n jmx:\n enabled: false\n\ncp-ksql-server:\n enabled: false\ncp-control-center:\n enabled: false\nStreams Explorer allows examining Apache Kafka data pipelines in a Kubernetes cluster including the inspection of schemas and monitoring of metrics. First, add the Helm repository:
helm repo add streams-explorer https://bakdata.github.io/streams-explorer && \\\nhelm repo update\nBelow you can find an example for the --values ./streams-explorer.yaml file configuring the deployment accordingly. Now, deploy the service:
helm upgrade \\\n --install \\\n --version 0.2.3 \\\n --values ./streams-explorer.yaml \\\n --namespace kpops \\\n streams-explorer streams-explorer/streams-explorer\nstreams-explorer.yaml) An example value configuration for Steams Explorer Helm chart.
imageTag: \"v2.1.2\"\nconfig:\n K8S__deployment__cluster: true\n SCHEMAREGISTRY__url: http://k8kafka-cp-schema-registry.kpops.svc.cluster.local:8081\n KAFKACONNECT__url: http://k8kafka-cp-kafka-connect.kpops.svc.cluster.local:8083\nresources:\n requests:\n cpu: 200m\n memory: 300Mi\n limits:\n cpu: 200m\n memory: 300Mi\nNow we will check if all the pods are running in our namespace. You can list all pods in the namespace with this command:
kubectl --namespace kpops get pods\nThen you should see the following output in your terminal:
NAME READY STATUS RESTARTS AGE\nk8kafka-cp-kafka-connect-8fc7d544f-8pjnt 1/1 Running 0 15m\nk8kafka-cp-zookeeper-0 1/1 Running 0 15m\nk8kafka-cp-kafka-0 1/1 Running 0 15m\nk8kafka-cp-schema-registry-588f8c65db-jdwbq 1/1 Running 0 15m\nk8kafka-cp-rest-6bbfd7b645-nwkf8 1/1 Running 0 15m\nstreams-explorer-54db878c67-s8wbz 1/1 Running 0 15m\nPay attention to the STATUS row. The pods should have a status of Running.
KPOps comes as a PyPI package. You can install it with pip:
pip install kpops\ndestroy: Removes Kubernetes resources.reset: Runs destroy, resets the states of Kafka Streams apps and resets offsets to zero.clean: Runs reset and removes all Kafka resources.The kpops CLI can be used to destroy a pipeline that was previously deployed with KPOps. In case that doesn't work, the pipeline can always be taken down manually with helm (see section Infrastructure).
Export environment variables.
export DOCKER_REGISTRY=bakdata && \\\nexport NAMESPACE=kpops\nNavigate to the examples folder. Replace the <name-of-the-example-directory> with the example you want to tear down. For example the atm-fraud-detection.
Remove the pipeline
# Uncomment 1 line to either destroy, reset or clean.\n\n# kpops destroy <name-of-the-example-directory>/pipeline.yaml \\\n# kpops reset <name-of-the-example-directory>/pipeline.yaml \\\n# kpops clean <name-of-the-example-directory>/pipeline.yaml \\\n--config <name-of-the-example-directory>/config.yaml \\\n--execute\nDelete namespace:
kubectl delete namespace kpops\nNote
In case kpops destroy is not working one can uninstall the pipeline services one by one. This is equivalent to running kpops destroy. In case a clean uninstall (like the one kpops clean does) is needed, one needs to also delete the topics and schemas created by deployment of the pipeline.
Delete local cluster:
k3d cluster delete kpops\nDelete local registry:
k3d registry delete k3d-kpops-registry.localhost\nKPOps automatically infers the component type from the class name. Therefore, the type and schema_type attributes can be removed from your custom components. By convention the type would be the lower, and kebab cased name of the class.
class MyCoolStreamApp(StreamsApp):\n- type = \"my-cool-stream-app\"\n+ ...\nBecause of this new convention producer has been renamed to producer-app. This must be addressed in your pipeline.yaml and defaults.yaml.
- producer:\n+ producer-app:\n app:\n streams:\n outputTopic: output_topic\n extraOutputTopics:\n output_role1: output_topic1\n output_role2: output_topic2\nIn the to section these have changed:
outputrole is set, type is inferred to be extraerror needs to be defined explicitly to:\n topics:\n ${pipeline_name}-topic-1:\n- type: extra\n role: \"role-1\"\n ...\n ${pipeline_name}-topic-2:\n- type: output\n ...\n ${pipeline_name}-topic-3:\n type: error\n ...\nIn the from section these have changed:
inputinput-pattern type is replaced by patternrole is set, type is inferred to be extrarole is set, type is explicitly set to pattern, this would be inferred type extra-pattern from:\n topics:\n ${pipeline_name}-input-topic:\n- type: input\n ...\n ${pipeline_name}-extra-topic:\n- type: extra\n role: topic-role\n ...\n ${pipeline_name}-input-pattern-topic:\n- type: input-pattern\n+ type: pattern\n ...\n ${pipeline_name}-extra-pattern-topic:\n- type: extra-pattern\n+ type: pattern\n role: some-role\n ...\nAll the internal KPOps models are now snake_case, and only Helm/Kubernetes values require camel casing. You can find an example of a pipeline.yaml in the following. Notice that the app section here remains untouched.
...\ntype: streams-app\n name: streams-app\n namespace: namespace\n app:\n streams:\n brokers: ${brokers}\n schemaRegistryUrl: ${schema_registry_url}\n autoscaling:\n consumerGroup: consumer-group\n lagThreshold: 0\n enabled: false\n pollingInterval: 30\n\n to:\n topics:\n ${pipeline_name}-output-topic:\n type: error\n- keySchema: key-schema\n+ key_schema: key-schema\n- valueSchema: value-schema\n+ value_schema: value-schema\n partitions_count: 1\n replication_factor: 1\n configs:\n cleanup.policy: compact\n models:\n model: model\n prefix: ${pipeline_name}-\n- repoConfig:\n+ repo_config:\n- repositoryName: bakdata-streams-bootstrap\n+ repository_name: bakdata-streams-bootstrap\n url: https://bakdata.github.io/streams-bootstrap/\n- repoAuthFlags:\n+ repo_auth_flags:\n username: user\n password: pass\n ca_file: /home/user/path/to/ca-file\n insecure_skip_tls_verify: false\n version: \"1.0.4\"\n...\nIf you are using the KubernetesApp class to define your own Kubernetes resource to deploy, the abstract function get_helm_chart that returns the chart for deploying the app using Helm is now a Python property and renamed to helm_chart.
class MyCoolApp(KubernetesApp):\n\n+ @property\n @override\n- def get_helm_chart(self) -> str:\n+ def helm_chart(self) -> str:\n return \"./charts/charts-folder\"\nSince you can pass a comma separated string of broker address, the broker field in KPOps is now plural. The pluralization has affected multiple areas:
"}, {"location": "user/migration-guide/v1-v2/#configyaml", "title": "config.yaml", "text": " environment: development\n- broker: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n+ brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n kafka_connect_host: \"http://localhost:8083\"\n kafka_rest_host: \"http://localhost:8082\"\n schema_registry_url: \"http://localhost:8081\"\nThe variable is now called brokers.
...\n app:\n streams:\n- brokers: ${broker}\n+ brokers: ${brokers}\n schemaRegistryUrl: ${schema_registry_url}\n nameOverride: override-with-this-name\n imageTag: \"1.0.0\"\n...\nPreviously, if you set the environment variable KPOPS_KAFKA_BROKER, you need to replace that now with KPOPS_KAFKA_BROKERS.
Jump to the summary
"}, {"location": "user/migration-guide/v2-v3/#use-hash-and-trim-long-helm-release-names-instead-of-only-trimming", "title": "Use hash and trim long Helm release names instead of only trimming", "text": "KPOps handles long (more than 53 characters) Helm releases names differently. Helm will not find your (long) old release names anymore. Therefore, it is recommended that you should once destroy your pipeline with KPOps v2 to remove old Helm release names. After a clean destroy, re-deploy your pipeline with the KPOps v3.
For example if you have a component with the Helm release name example-component-name-too-long-fake-fakefakefakefakefake. The new release name will shorten the original name to 53 characters and then replace the last 6 characters of the trimmed name with the first 5 characters of the result of SHA-1(helm_release_name).
example-component-name-too-long-fake-fakefakef-0a7fc ----> 53 chars\n---------------------------------------------- -----\n ^Shortened helm_release_name ^first 5 characters of SHA1(helm_release_name)\nAll Helm-specific parts of the built-in KubernetesApp have been extracted to a new child component that is more appropriately named HelmApp. It has to be renamed in your existing pipeline defintions and custom components module.
-- type: kubernetes-app\n+- type: helm-app\n name: foo\n- from kpops.components import KubernetesApp\n+ from kpops.components import HelmApp\n\n\n- class CustomHelmApp(KubernetesApp):\n+ class CustomHelmApp(HelmApp):\n ...\nPreviously the default KafkaApp component configured the streams-bootstrap Helm Charts. Now, this component is no longer tied to Helm (or Kubernetes). Instead, there is a new StreamsBootstrap component that configures the Helm Chart repository for the components that use it, e.g. StreamsApp and ProducerApp. If you are using non-default values for the Helm Chart repository or version, it has to be updated as shown below.
kafka-app:\n app:\n streams: ...\n\n+ streams-bootstrap:\n repo_config: ...\n version: ...\nInternally, the Kafka Connector resetter is now its own standard HelmApp, removing a lot of the shared code. It is configured using the resetter_namespace (formerly namespace) and resetter_values attributes.
kafka-connector:\n- namespace: my-namespace\n+ resetter_namespace: my-namespace\nThe breaking changes target the config.yaml file:
The schema_registry_url is replaced with schema_registry.url (default http://localhost:8081) and schema_registry.enabled (default false).
kafka_rest_host is renamed to kafka_rest.url (default http://localhost:8082).
kafka_connect_host is replaced with kafka_connect.url (default http://localhost:8083).
brokers is renamed to kafka_brokers.
The environment variable names of these config fields changed respectively. Please refer to the environment variables documentation page to see the newest changes.
"}, {"location": "user/migration-guide/v2-v3/#configyaml", "title": "config.yaml", "text": " environment: development\n- brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n- kafka_rest_host: \"http://my-custom-rest.url:8082\"\n- kafka_connect_host: \"http://my-custom-connect.url:8083\"\n- schema_registry_url: \"http://my-custom-sr.url:8081\"\n+ kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n+ kafka_rest:\n+ url: \"http://my-custom-rest.url:8082\"\n+ kafka_connect:\n+ url: \"http://my-custom-connect.url:8083\"\n+ schema_registry:\n+ enabled: true\n+ url: \"http://my-custom-sr.url:8081\"\nThe variable is now called kafka_brokers.
...\n app:\n streams:\n- brokers: ${brokers}\n+ brokers: ${kafka_brokers}\n schemaRegistryUrl: ${schema_registry_url}\n nameOverride: override-with-this-name\n imageTag: \"1.0.0\"\n...\nWarning
The previous CLI parameters have been removed.
The options for a custom components_module and pipeline_base_dir are now global settings, defined in config.yaml.
kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n environment: development\n+ components_module: components\n+ pipeline_base_dir: pipelines\nThe location of the GitHub action has changed, and it's now available directly as bakdata/kpops.
You'll need to change it in your GitHub CI workflows.
steps:\n - name: kpops deploy\n- uses: bakdata/kpops/actions/kpops-runner@main\n+ uses: bakdata/kpops@main\n with:\n command: deploy --execute\n # ...\nSpecifying the environment is no longer mandatory. If not defined, only the global files will be used.
environment is no longer specified in config.yaml. Instead, it can be either set via the CLI flag --environment or with the environment variable KPOPS_ENVIRONMENT.
The --config flag in the CLI now points to the directory that contains config*.yaml files. The files to be used are resolved based on the provided (or not) environment.
- environment: development\n kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n schema_registry:\n enabled: true\n url: \"http://my-custom-sr.url:8081\"\n.", "text": "The delimiter in the substitution variables is changed to ..
steps:\n - type: scheduled-producer\n app:\n labels:\n- app_type: \"${component_type}\"\n- app_name: \"${component_name}\"\n- app_schedule: \"${component_app_schedule}\"\n+ app_type: \"${component.type}\"\n+ app_name: \"${component.name}\"\n+ app_schedule: \"${component.app.schedule}\"\ntopic_name_config:\n- default_error_topic_name: \"${pipeline_name}-${component_name}-dead-letter-topic\"\n- default_output_topic_name: \"${pipeline_name}-${component_name}-topic\"\n+ default_error_topic_name: \"${pipeline_name}-${component.name}-dead-letter-topic\"\n+ default_output_topic_name: \"${pipeline_name}-${component.name}-topic\"\nThe template method of every pipeline component has been renamed to manifest as it is no longer strictly tied to Helm template. Instead, it can be used to render the final resources of a component, such as Kubernetes manifests.
There is also a new kpops manifest command replacing the existing kpops generate --template flag.
If you're using this functionality in your custom components, it needs to be updated.
from kpops.components.base_components.models.resource import Resource\n\n @override\n- def template(self) -> None:\n+ def manifest(self) -> Resource:\n \"\"\"Render final component resources, e.g. Kubernetes manifests.\"\"\"\n return [] # list of manifests\nThe global configuration variables are now namespaced under the config key, such as ${config.kafka_brokers}, ${config.schema_registry.url}. Same with pipeline variables, e.g. ${pipeline_name} \u2192 ${pipeline.name}. This would make it more uniform with the existing ${component.<key>} variables.
name: kafka-app\n- prefix: ${pipeline_name}-\n+ prefix: ${pipeline.name}-\n app:\n streams:\n- brokers: ${kafka_brokers}\n- schemaRegistryUrl: ${schema_registry.url}\n+ brokers: ${config.kafka_brokers}\n+ schemaRegistryUrl: ${config.schema_registry.url}\nWarning
Helm will not find your (long) old release names anymore.
defaults.yaml kafka-app:\n app:\n streams: ...\n\n+ streams-bootstrap:\n repo_config: ...\n version: ...\n- - type: kubernetes-app\n+ - type: helm-app\n ...\n - type: kafka-app\n app:\n- brokers: ${brokers}\n+ brokers: ${config.kafka_brokers}\n labels:\n- app_schedule: \"${component_app_schedule}\"\n+ app_schedule: \"${component.app.schedule}\"\n ...\n - type: kafka-connector\n- namespace: my-namespace\n+ resetter_namespace: my-namespace\n ...\n- environment: development\n\n+ components_module: components\n\n+ pipeline_base_dir: pipelines\n\n- brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n+ kafka_brokers: \"http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092\"\n\n- kafka_rest_host: \"http://my-custom-rest.url:8082\"\n+ kafka_rest:\n+ url: \"http://my-custom-rest.url:8082\"\n\n- kafka_connect_host: \"http://my-custom-connect.url:8083\"\n+ kafka_connect:\n+ url: \"http://my-custom-connect.url:8083\"\n\n- schema_registry_url: \"http://my-custom-sr.url:8081\"\n+ schema_registry:\n+ enabled: true\n+ url: \"http://my-custom-sr.url:8081\"\n\n topic_name_config:\n- default_error_topic_name: \"${pipeline_name}-${component_name}-dead-letter-topic\"\n+ default_error_topic_name: \"${pipeline.name}-${component.name}-dead-letter-topic\"\n ...\n- from kpops.components import KubernetesApp\n+ from kpops.components import HelmApp\n+ from kpops.components.base_components.models.resource import Resource\n\n- class CustomHelmApp(KubernetesApp):\n+ class CustomHelmApp(HelmApp):\n\n @override\n- def template(self) -> None:\n+ def manifest(self) -> Resource:\n \"\"\"Render final component resources, e.g. Kubernetes manifests.\"\"\"\n return [] # list of manifests\n ...\n steps:\n - name: ...\n- uses: bakdata/kpops/actions/kpops-runner@main\n+ uses: bakdata/kpops@main\n ...\nWarning
The --defaults flag is removed
It is possible now to use multiple default values. The defaults.yaml (or defaults_<env>.yaml) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single pipeline.yaml file. KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the pipeline_base_dir is defined.
For example, imagine the following folder structure:
\u2514\u2500 pipelines\n \u2514\u2500\u2500 distributed-defaults\n \u251c\u2500\u2500 defaults.yaml\n \u251c\u2500\u2500 defaults_dev.yaml\n \u2514\u2500\u2500 pipeline-deep\n \u251c\u2500\u2500 defaults.yaml\n \u2514\u2500\u2500 pipeline.yaml\nThe pipeline_base_dir is configured to pipelines. Now if we generate this pipeline with the following command:
kpops generate \\\n --environment dev\n ./pipelines/distributed-defaults/pipeline-deep/pipeline.yaml\nThe defaults would be picked in the following order (high to low priority):
./pipelines/distributed-defaults/pipeline-deep/defaults.yaml./pipelines/distributed-defaults/defaults_dev.yaml./pipelines/distributed-defaults/defaults.yamlThe deepest defaults.yaml file in the folder hierarchy (i.e., the closest one to the pipeline.yaml) overwrites the higher-level defaults' values.
The global timeout setting has been removed. Instead, an individual timeout can be set for each external service. The default is 30 seconds.
- timeout: 300\n\n kafka_rest:\n url: \"http://my-custom-rest.url:8082\"\n+ timeout: 30\n kafka_connect:\n url: \"http://my-custom-connect.url:8083\"\n+ timeout: 30\n schema_registry:\n enabled: true\n url: \"http://my-custom-sr.url:8081\"\n+ timeout: 30\nKPOps can now deploy multiple pipelines in a single command. It is possible to pass one or many pipeline.yaml files or pass a directory with many pipeline.yaml files within it.
The environment variable KPOPS_PIPELINE_PATH is changed to KPOPS_PIPELINE_PATHS.
Read more:
KPops Python API is now stable and separated from the CLI! \ud83c\udf89
"}, {"location": "user/migration-guide/v6-v7/", "title": "Migrate from V6 to V7", "text": ""}, {"location": "user/migration-guide/v6-v7/#automatic-loading-of-namespaced-custom-components", "title": "Automatic loading of namespaced custom components", "text": "KPOps is now distributed as a Python namespace package (as defined by PEP 420). This allows us to standardize the namespace kpops.components for both builtin and custom pipeline components.
As a result of the restructure, some imports need to be adjusted:
KPOps Python API
- import kpops\n+ import kpops.api as kpops\nbuiltin KPOps components
- from kpops.components import (\n- HelmApp,\n- KafkaApp,\n- KafkaConnector,\n- KafkaSinkConnector,\n- KafkaSourceConnector,\n- KubernetesApp,\n- StreamsBootstrap,\n- ProducerApp,\n- StreamsApp,\n- PipelineComponent,\n- StreamsApp,\n- ProducerApp,\n- )\n+ from kpops.components.base_components import (\n+ HelmApp,\n+ KafkaApp,\n+ KafkaConnector,\n+ KafkaSinkConnector,\n+ KafkaSourceConnector,\n+ KubernetesApp,\n+ PipelineComponent,\n+ )\n+ from kpops.components.streams_bootstrap import (\n+ StreamsBootstrap,\n+ StreamsApp,\n+ ProducerApp,\n+ )\n- components_module: components\n- components/__init__.py\n+ kpops/components/custom/__init__.py\nThe app attribute of the builtin KPOps components has been renamed to better differentiate them. Both your pipeline.yaml and defaults.yaml files have to be updated, e.g.:
kubernetes-app:\n- app: {}\n+ values: {}\n\n helm-app:\n- app: {}\n+ values: {}\n\n kafka-app:\n- app: {}\n+ values: {}\n\n streams-app:\n- app: {}\n+ values: {}\n\n producer-app:\n- app: {}\n+ values: {}\n\n kafka-connector:\n- app: {}\n+ config: {}\n\n kafka-source-connector:\n- app: {}\n+ config: {}\n\n kafka-sink-connector:\n- app: {}\n+ config: {}\nBefore v7, the KPOps CLI executed destroy before running reset/clean to ensure the component was destroyed.
This logic has changed. The destroy method is now called within the PipelineComponent's reset/clean.
During migrating to v7, you should check your custom components and see if they override the reset/clean methods. If so, you need to call the supermethod reset/clean to trigger the destroy inside the parent class. Alternatively, if you are implementing the PipelineComponent class, you need to call the destroy method at the beginning of the method.
For example, when creating a custom StreamsApp or ProducerApp (or any other custom component), you must call the supermethod reset/clean to execute the destroy in the parent class. Otherwise, the logic of destroy will not be executed!
class MyStreamsApp(StreamsApp):\n\n @override\n async def clean(self, dry_run: bool) -> None:\n+ await super().clean(dry_run)\n # Some custom clean logic\n # ...\n ```diff\n \n \nclass MyCustomComponent(PipelineComponent):\n \n @override\n async def destroy(self, dry_run: bool) -> None:\n # Some custom destroy logic\n # ...\n\n @override\n async def clean(self, dry_run: bool) -> None:\n+ await super().clean(dry_run)\n # Some custom clean logic\n # ...\nFrom now on KPOps supports streams-bootstrap v3 as its default component. The previous streams-bootstrap version (below 3.x.x) is marked as deprecated and will be removed in a future version of KPOps. If you don't want to migrate your producer or streams app to v3, you should suffix your components with -v2. Here is an example of a pipeline.yaml file.
- - type: producer-app\n+ - type: producer-app-v2\n\n- - type: streams-app\n+ - type: streams-app-v2\n\n# rest of your pipeline\n- class MyStreamsApp(StreamsApp):\n+ class MyStreamsApp(StreamsAppV2):\n ...\nInfo
The streams-boostrap, streams-app, and producer-app now all take the Helm values of streams-bootstrap version 3. You can find these values under the Helm charts documentation or by referring to the Base model definitions.
The keyword role is renamed to label. You need to replace it in your pipeline.yaml, defaults.yaml, and the Python components definition files. Here is a simple example of the defaults.yaml.
streams-app-v2:\n values:\n streams:\n brokers: localhost:9092\n from:\n topics:\n my-labeled-input-topic:\n- role: my-input-topic-label\n+ label: my-input-topic-label\n my-labeled-input-pattern:\n type: pattern\n- role: my-input-topic-labeled-pattern\n+ label: my-input-topic-labeled-pattern\n\n to:\n topics:\n my-labeled-topic-output:\n- role: my-output-topic-label\n+ label: my-output-topic-label\n\n# rest of your pipeline\nThe KafkaApp component now only contains the deployment logic of the stream-bootstrap applications (streams-app, producer-app). It should not be used in the defaults.yaml nor the pipeline.yaml. If you are using it, it should be replaced by streams-bootstrap.
- kafka-app:\n+ streams-bootstrap-v2:\n values:\n streams:\n brokers: 127.0.0.1:9092\n schemaRegistryUrl: 127.0.0.1:8081\nThe kpops manifest command and kpops.manifest() API have been removed.
Resource manifesting is now integrated into the operation commands (deploy, destroy, reset, clean) through the new operation mode feature.
To manifest resources, you can:
--operation-mode manifest when executing kpops commands.KPOPS_OPERATION_MODE environment variable.KPOps now supports generating valid Kubernetes KafkaTopic resources compatible with Strimzi. When using manifest or argo as the operation_mode, you must specify the Strimzi cluster label to ensure the topics are recognized by the deployed Strimzi Topic Operator.
operation_mode: manifest\n\n+ strimzi_topic:\n+ label:\n+ strimzi.io/cluster: my-cluster\n\n# rest of your config\nInfo
Refer to the Strimzi documentation on deploying a standalone topic operator for more details.
"}, {"location": "user/migration-guide/v8-v9/#drop-support-for-python-310", "title": "Drop support for Python 3.10", "text": "KPOps V9 no longer supports Python 3.10. Ensure your environment is running Python 3.11 to 3.12.
"}, {"location": "user/migration-guide/v8-v9/#action-required", "title": "Action Required:", "text": "Upgrade your Python version to a supported version (3.11 or 3.12). Update your virtual environments and CI pipelines to reflect this change.
"}, {"location": "user/references/cli-commands/", "title": "CLI Usage", "text": "Usage:
$ kpops [OPTIONS] COMMAND [ARGS]...\nOptions:
-V, --version: Print KPOps version--install-completion: Install completion for the current shell.--show-completion: Show completion for the current shell, to copy it or customize the installation.--help: Show this message and exit.Commands:
init: Initialize a new KPOps project.generate: Generate enriched pipeline representationdeploy: Deploy pipeline stepsdestroy: Destroy pipeline stepsreset: Reset pipeline stepsclean: Clean pipeline stepsschema: Generate JSON schema.kpops init", "text": "Initialize a new KPOps project.
Usage:
$ kpops init [OPTIONS] PATH\nArguments:
PATH: Path for a new KPOps project. It should lead to an empty (or non-existent) directory. The part of the path that doesn't exist will be created. [required]Options:
--config-include-optional / --no-config-include-optional: Whether to include non-required settings in the generated 'config.yaml' [default: no-config-include-optional]--help: Show this message and exit.kpops generate", "text": "Enrich pipeline steps with defaults. The enriched pipeline is used for all KPOps operations (deploy, destroy, ...).
Usage:
$ kpops generate [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--help: Show this message and exit.kpops deploy", "text": "Deploy pipeline steps
Usage:
$ kpops deploy [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops destroy", "text": "Destroy pipeline steps
Usage:
$ kpops destroy [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops reset", "text": "Reset pipeline steps
Usage:
$ kpops reset [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops clean", "text": "Clean pipeline steps
Usage:
$ kpops clean [OPTIONS] PIPELINE_PATHS...\nArguments:
PIPELINE_PATHS...: Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. [env var: KPOPS_PIPELINE_PATHS; required]Options:
--dotenv FILE: Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. [env var: KPOPS_DOTENV_PATH]--config DIRECTORY: Path to the dir containing config.yaml files [env var: KPOPS_CONFIG_PATH; default: .]--steps TEXT: Comma separated list of steps to apply the command on [env var: KPOPS_PIPELINE_STEPS]--filter-type [include|exclude]: Whether the --steps option should include/exclude the steps [default: include]--environment TEXT: The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development). [env var: KPOPS_ENVIRONMENT]--dry-run / --execute: Whether to dry run the command or execute it [default: dry-run]--verbose / --no-verbose: Enable verbose printing [default: no-verbose]--parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]--operation-mode [argo|manifest|managed]: How KPOps should operate. [env var: KPOPS_OPERATION_MODE; default: managed]--help: Show this message and exit.kpops schema", "text": "Generate JSON schema.
The schemas can be used to enable support for KPOps files in a text editor.
Usage:
$ kpops schema [OPTIONS] SCOPE:{pipeline|defaults|config}\nArguments:
SCOPE:{pipeline|defaults|config}: Scope of the generated schema- pipeline: Schema of PipelineComponents for KPOps pipeline.yaml\n\n- defaults: Schema of PipelineComponents for KPOps defaults.yaml\n\n- config: Schema for KPOps config.yaml [required]\nOptions:
--help: Show this message and exit.We are working towards first-class editor support by providing plugins that work out of the box.
settings.json {\n \"yaml.schemas\": {\n \"https://bakdata.github.io/kpops/4.0/schema/pipeline.json\": [\n \"pipeline.yaml\",\n \"pipeline_*.yaml\"\n ],\n \"https://bakdata.github.io/kpops/4.0/schema/defaults.json\": [\n \"defaults.yaml\",\n \"defaults_*.yaml\"\n ],\n \"https://bakdata.github.io/kpops/4.0/schema/config.json\": [\n \"config.yaml\",\n \"config_*.yaml\"\n ]\n }\n}\nAdvanced usage
It is possible to generate schemas with the kpops schema command. Useful for including custom components or when using a pre-release version of KPOps.
KPOps provides JSON schemas that enable autocompletion and validation for all YAML files that the user must work with.
"}, {"location": "user/references/editor-integration/#supported-files", "title": "Supported files", "text": "pipeline.yamldefaults.yamlconfig.yamlWe provided a GitHub composite action bakdata/kpops that installs and executes KPOps commands with the given parameters.
steps:\n # ...\n # This step is useful for debugging reasons\n - name: Generate Kafka pipeline\n uses: bakdata/kpops@main\n with:\n command: generate\n working-directory: home/my-kpops-root-dir\n pipeline: pipelines/my-pipeline-file.yaml\n kpops-version: 1.2.3\n\n # It is possible to use a pre-release KPOps version from TestPyPI https://test.pypi.org/project/kpops/#history\n - name: Deploy Kafka pipeline\n uses: bakdata/kpops@main\n with:\n command: deploy --execute\n working-directory: home/my-kpops-root-dir\n pipeline: pipelines/my-pipeline-file.yaml\n kpops-version: 1.2.5.dev20230707132709\n # ...\n