Skip to content

Commit

Permalink
Deployed a90984b to dev with MkDocs 1.5.3 and mike 1.1.2
Browse files Browse the repository at this point in the history
  • Loading branch information
bakdata-bot committed Jan 17, 2024
1 parent f9df428 commit ed250e5
Show file tree
Hide file tree
Showing 3 changed files with 172 additions and 2 deletions.
2 changes: 1 addition & 1 deletion dev/search/search_index.json

Large diffs are not rendered by default.

Binary file modified dev/sitemap.xml.gz
Binary file not shown.
172 changes: 171 additions & 1 deletion dev/user/migration-guide/v2-v3/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -1462,6 +1462,13 @@
</ul>
</nav>

</li>

<li class="md-nav__item">
<a href="#summary" class="md-nav__link">
Summary
</a>

</li>

</ul>
Expand Down Expand Up @@ -1995,6 +2002,13 @@
</ul>
</nav>

</li>

<li class="md-nav__item">
<a href="#summary" class="md-nav__link">
Summary
</a>

</li>

</ul>
Expand All @@ -2016,6 +2030,7 @@


<h1 id="migrate-from-v2-to-v3">Migrate from V2 to V3<a class="headerlink" href="#migrate-from-v2-to-v3" title="Permanent link">&para;</a></h1>
<p><a href="#summary"><strong>Jump to the summary</strong></a></p>
<h2 id="use-hash-and-trim-long-helm-release-names-instead-of-only-trimming"><a href="https://github.com/bakdata/kpops/pull/390">Use hash and trim long Helm release names instead of only trimming</a><a class="headerlink" href="#use-hash-and-trim-long-helm-release-names-instead-of-only-trimming" title="Permanent link">&para;</a></h2>
<p>KPOps handles long (more than 53 characters) Helm releases names differently. Helm will not find your (long) old release names anymore. Therefore, it is recommended that you should once destroy your pipeline with KPOps v2 to remove old Helm release names. After a clean destroy, re-deploy your pipeline with the KPOps v3.</p>
<p>For example if you have a component with the Helm release name <code>example-component-name-too-long-fake-fakefakefakefakefake</code>. The new release name will shorten the original name to 53 characters and then replace the last 6 characters of the trimmed name with the first 5 characters of the result of SHA-1(helm_release_name).</p>
Expand Down Expand Up @@ -2250,7 +2265,7 @@ <h2 id="refactor-generate-template-for-python-api-usage"><a href="https://github
<h2 id="namespace-substitution-vars"><a href="https://github.com/bakdata/kpops/pull/408">Namespace substitution vars</a><a class="headerlink" href="#namespace-substitution-vars" title="Permanent link">&para;</a></h2>
<p>The global configuration variables are now namespaced under the config key, such as <code>${config.kafka_brokers}</code>, <code>${config.schema_registry.url}</code>. Same with pipeline variables, e.g. <code>${pipeline_name} → ${pipeline.name}</code>.
This would make it more uniform with the existing <code>${component.&lt;key&gt;}</code> variables.</p>
<h3 id="pipelineyaml_1">pipeline.yaml<a class="headerlink" href="#pipelineyaml_1" title="Permanent link">&para;</a></h3>
<h4 id="pipelineyaml_1">pipeline.yaml<a class="headerlink" href="#pipelineyaml_1" title="Permanent link">&para;</a></h4>
<div class="language-diff highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-13-1">1</a></span>
<span class="normal"><a href="#__codelineno-13-2">2</a></span>
<span class="normal"><a href="#__codelineno-13-3">3</a></span>
Expand All @@ -2269,6 +2284,161 @@ <h3 id="pipelineyaml_1">pipeline.yaml<a class="headerlink" href="#pipelineyaml_1
</span><span id="__span-13-8"><a id="__codelineno-13-8" name="__codelineno-13-8"></a><span class="gi">+ brokers: ${config.kafka_brokers}</span>
</span><span id="__span-13-9"><a id="__codelineno-13-9" name="__codelineno-13-9"></a><span class="gi">+ schemaRegistryUrl: ${config.schema_registry.url}</span>
</span></code></pre></div></td></tr></table></div>
<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">&para;</a></h2>
<!-- dprint-ignore-start -->

<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p><a href="#use-hash-and-trim-long-helm-release-names-instead-of-only-trimming"><strong>Helm will not find your (long) old release names anymore.</strong></a></p>
</div>
<details class="example">
<summary>defaults.yaml</summary>
<div class="language-diff highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-14-1">1</a></span>
<span class="normal"><a href="#__codelineno-14-2">2</a></span>
<span class="normal"><a href="#__codelineno-14-3">3</a></span>
<span class="normal"><a href="#__codelineno-14-4">4</a></span>
<span class="normal"><a href="#__codelineno-14-5">5</a></span>
<span class="normal"><a href="#__codelineno-14-6">6</a></span>
<span class="normal"><a href="#__codelineno-14-7">7</a></span></pre></div></td><td class="code"><div><pre><span></span><code><span id="__span-14-1"><a id="__codelineno-14-1" name="__codelineno-14-1"></a><span class="w"> </span> kafka-app:
</span><span id="__span-14-2"><a id="__codelineno-14-2" name="__codelineno-14-2"></a><span class="w"> </span> app:
</span><span id="__span-14-3"><a id="__codelineno-14-3" name="__codelineno-14-3"></a><span class="w"> </span> streams: ...
</span><span id="__span-14-4"><a id="__codelineno-14-4" name="__codelineno-14-4"></a>
</span><span id="__span-14-5"><a id="__codelineno-14-5" name="__codelineno-14-5"></a><span class="gi">+ streams-bootstrap:</span>
</span><span id="__span-14-6"><a id="__codelineno-14-6" name="__codelineno-14-6"></a><span class="w"> </span> repo_config: ...
</span><span id="__span-14-7"><a id="__codelineno-14-7" name="__codelineno-14-7"></a><span class="w"> </span> version: ...
</span></code></pre></div></td></tr></table></div>
</details>
<details class="example">
<summary>pipeline.yaml</summary>
<div class="language-diff highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-15-1"> 1</a></span>
<span class="normal"><a href="#__codelineno-15-2"> 2</a></span>
<span class="normal"><a href="#__codelineno-15-3"> 3</a></span>
<span class="normal"><a href="#__codelineno-15-4"> 4</a></span>
<span class="normal"><a href="#__codelineno-15-5"> 5</a></span>
<span class="normal"><a href="#__codelineno-15-6"> 6</a></span>
<span class="normal"><a href="#__codelineno-15-7"> 7</a></span>
<span class="normal"><a href="#__codelineno-15-8"> 8</a></span>
<span class="normal"><a href="#__codelineno-15-9"> 9</a></span>
<span class="normal"><a href="#__codelineno-15-10">10</a></span>
<span class="normal"><a href="#__codelineno-15-11">11</a></span>
<span class="normal"><a href="#__codelineno-15-12">12</a></span>
<span class="normal"><a href="#__codelineno-15-13">13</a></span>
<span class="normal"><a href="#__codelineno-15-14">14</a></span>
<span class="normal"><a href="#__codelineno-15-15">15</a></span></pre></div></td><td class="code"><div><pre><span></span><code><span id="__span-15-1"><a id="__codelineno-15-1" name="__codelineno-15-1"></a><span class="gd">- - type: kubernetes-app</span>
</span><span id="__span-15-2"><a id="__codelineno-15-2" name="__codelineno-15-2"></a><span class="gi">+ - type: helm-app</span>
</span><span id="__span-15-3"><a id="__codelineno-15-3" name="__codelineno-15-3"></a><span class="w"> </span> ...
</span><span id="__span-15-4"><a id="__codelineno-15-4" name="__codelineno-15-4"></a><span class="w"> </span> - type: kafka-app
</span><span id="__span-15-5"><a id="__codelineno-15-5" name="__codelineno-15-5"></a><span class="w"> </span> app:
</span><span id="__span-15-6"><a id="__codelineno-15-6" name="__codelineno-15-6"></a><span class="gd">- brokers: ${brokers}</span>
</span><span id="__span-15-7"><a id="__codelineno-15-7" name="__codelineno-15-7"></a><span class="gi">+ brokers: ${config.kafka_brokers}</span>
</span><span id="__span-15-8"><a id="__codelineno-15-8" name="__codelineno-15-8"></a><span class="w"> </span> labels:
</span><span id="__span-15-9"><a id="__codelineno-15-9" name="__codelineno-15-9"></a><span class="gd">- app_schedule: &quot;${component_app_schedule}&quot;</span>
</span><span id="__span-15-10"><a id="__codelineno-15-10" name="__codelineno-15-10"></a><span class="gi">+ app_schedule: &quot;${component.app.schedule}&quot;</span>
</span><span id="__span-15-11"><a id="__codelineno-15-11" name="__codelineno-15-11"></a><span class="w"> </span> ...
</span><span id="__span-15-12"><a id="__codelineno-15-12" name="__codelineno-15-12"></a><span class="w"> </span> - type: kafka-connector
</span><span id="__span-15-13"><a id="__codelineno-15-13" name="__codelineno-15-13"></a><span class="gd">- namespace: my-namespace</span>
</span><span id="__span-15-14"><a id="__codelineno-15-14" name="__codelineno-15-14"></a><span class="gi">+ resetter_namespace: my-namespace</span>
</span><span id="__span-15-15"><a id="__codelineno-15-15" name="__codelineno-15-15"></a><span class="w"> </span> ...
</span></code></pre></div></td></tr></table></div>
</details>
<details class="example">
<summary>config.yaml</summary>
<div class="language-diff highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-16-1"> 1</a></span>
<span class="normal"><a href="#__codelineno-16-2"> 2</a></span>
<span class="normal"><a href="#__codelineno-16-3"> 3</a></span>
<span class="normal"><a href="#__codelineno-16-4"> 4</a></span>
<span class="normal"><a href="#__codelineno-16-5"> 5</a></span>
<span class="normal"><a href="#__codelineno-16-6"> 6</a></span>
<span class="normal"><a href="#__codelineno-16-7"> 7</a></span>
<span class="normal"><a href="#__codelineno-16-8"> 8</a></span>
<span class="normal"><a href="#__codelineno-16-9"> 9</a></span>
<span class="normal"><a href="#__codelineno-16-10">10</a></span>
<span class="normal"><a href="#__codelineno-16-11">11</a></span>
<span class="normal"><a href="#__codelineno-16-12">12</a></span>
<span class="normal"><a href="#__codelineno-16-13">13</a></span>
<span class="normal"><a href="#__codelineno-16-14">14</a></span>
<span class="normal"><a href="#__codelineno-16-15">15</a></span>
<span class="normal"><a href="#__codelineno-16-16">16</a></span>
<span class="normal"><a href="#__codelineno-16-17">17</a></span>
<span class="normal"><a href="#__codelineno-16-18">18</a></span>
<span class="normal"><a href="#__codelineno-16-19">19</a></span>
<span class="normal"><a href="#__codelineno-16-20">20</a></span>
<span class="normal"><a href="#__codelineno-16-21">21</a></span>
<span class="normal"><a href="#__codelineno-16-22">22</a></span>
<span class="normal"><a href="#__codelineno-16-23">23</a></span>
<span class="normal"><a href="#__codelineno-16-24">24</a></span>
<span class="normal"><a href="#__codelineno-16-25">25</a></span>
<span class="normal"><a href="#__codelineno-16-26">26</a></span></pre></div></td><td class="code"><div><pre><span></span><code><span id="__span-16-1"><a id="__codelineno-16-1" name="__codelineno-16-1"></a><span class="gd">- environment: development</span>
</span><span id="__span-16-2"><a id="__codelineno-16-2" name="__codelineno-16-2"></a>
</span><span id="__span-16-3"><a id="__codelineno-16-3" name="__codelineno-16-3"></a><span class="gi">+ components_module: components</span>
</span><span id="__span-16-4"><a id="__codelineno-16-4" name="__codelineno-16-4"></a>
</span><span id="__span-16-5"><a id="__codelineno-16-5" name="__codelineno-16-5"></a><span class="gi">+ pipeline_base_dir: pipelines</span>
</span><span id="__span-16-6"><a id="__codelineno-16-6" name="__codelineno-16-6"></a>
</span><span id="__span-16-7"><a id="__codelineno-16-7" name="__codelineno-16-7"></a><span class="gd">- brokers: &quot;http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092&quot;</span>
</span><span id="__span-16-8"><a id="__codelineno-16-8" name="__codelineno-16-8"></a><span class="gi">+ kafka_brokers: &quot;http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092&quot;</span>
</span><span id="__span-16-9"><a id="__codelineno-16-9" name="__codelineno-16-9"></a>
</span><span id="__span-16-10"><a id="__codelineno-16-10" name="__codelineno-16-10"></a><span class="gd">- kafka_rest_host: &quot;http://my-custom-rest.url:8082&quot;</span>
</span><span id="__span-16-11"><a id="__codelineno-16-11" name="__codelineno-16-11"></a><span class="gi">+ kafka_rest:</span>
</span><span id="__span-16-12"><a id="__codelineno-16-12" name="__codelineno-16-12"></a><span class="gi">+ url: &quot;http://my-custom-rest.url:8082&quot;</span>
</span><span id="__span-16-13"><a id="__codelineno-16-13" name="__codelineno-16-13"></a>
</span><span id="__span-16-14"><a id="__codelineno-16-14" name="__codelineno-16-14"></a><span class="gd">- kafka_connect_host: &quot;http://my-custom-connect.url:8083&quot;</span>
</span><span id="__span-16-15"><a id="__codelineno-16-15" name="__codelineno-16-15"></a><span class="gi">+ kafka_connect:</span>
</span><span id="__span-16-16"><a id="__codelineno-16-16" name="__codelineno-16-16"></a><span class="gi">+ url: &quot;http://my-custom-connect.url:8083&quot;</span>
</span><span id="__span-16-17"><a id="__codelineno-16-17" name="__codelineno-16-17"></a>
</span><span id="__span-16-18"><a id="__codelineno-16-18" name="__codelineno-16-18"></a><span class="gd">- schema_registry_url: &quot;http://my-custom-sr.url:8081&quot;</span>
</span><span id="__span-16-19"><a id="__codelineno-16-19" name="__codelineno-16-19"></a><span class="gi">+ schema_registry:</span>
</span><span id="__span-16-20"><a id="__codelineno-16-20" name="__codelineno-16-20"></a><span class="gi">+ enabled: true</span>
</span><span id="__span-16-21"><a id="__codelineno-16-21" name="__codelineno-16-21"></a><span class="gi">+ url: &quot;http://my-custom-sr.url:8081&quot;</span>
</span><span id="__span-16-22"><a id="__codelineno-16-22" name="__codelineno-16-22"></a>
</span><span id="__span-16-23"><a id="__codelineno-16-23" name="__codelineno-16-23"></a><span class="w"> </span> topic_name_config:
</span><span id="__span-16-24"><a id="__codelineno-16-24" name="__codelineno-16-24"></a><span class="gd">- default_error_topic_name: &quot;${pipeline_name}-${component_name}-dead-letter-topic&quot;</span>
</span><span id="__span-16-25"><a id="__codelineno-16-25" name="__codelineno-16-25"></a><span class="gi">+ default_error_topic_name: &quot;${pipeline.name}-${component.name}-dead-letter-topic&quot;</span>
</span><span id="__span-16-26"><a id="__codelineno-16-26" name="__codelineno-16-26"></a><span class="w"> </span> ...
</span></code></pre></div></td></tr></table></div>
</details>
<details class="example">
<summary>custom_module.py</summary>
<div class="language-diff highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-17-1"> 1</a></span>
<span class="normal"><a href="#__codelineno-17-2"> 2</a></span>
<span class="normal"><a href="#__codelineno-17-3"> 3</a></span>
<span class="normal"><a href="#__codelineno-17-4"> 4</a></span>
<span class="normal"><a href="#__codelineno-17-5"> 5</a></span>
<span class="normal"><a href="#__codelineno-17-6"> 6</a></span>
<span class="normal"><a href="#__codelineno-17-7"> 7</a></span>
<span class="normal"><a href="#__codelineno-17-8"> 8</a></span>
<span class="normal"><a href="#__codelineno-17-9"> 9</a></span>
<span class="normal"><a href="#__codelineno-17-10">10</a></span>
<span class="normal"><a href="#__codelineno-17-11">11</a></span>
<span class="normal"><a href="#__codelineno-17-12">12</a></span>
<span class="normal"><a href="#__codelineno-17-13">13</a></span></pre></div></td><td class="code"><div><pre><span></span><code><span id="__span-17-1"><a id="__codelineno-17-1" name="__codelineno-17-1"></a><span class="gd">- from kpops.components import KubernetesApp</span>
</span><span id="__span-17-2"><a id="__codelineno-17-2" name="__codelineno-17-2"></a><span class="gi">+ from kpops.components import HelmApp</span>
</span><span id="__span-17-3"><a id="__codelineno-17-3" name="__codelineno-17-3"></a><span class="gi">+ from kpops.components.base_components.models.resource import Resource</span>
</span><span id="__span-17-4"><a id="__codelineno-17-4" name="__codelineno-17-4"></a>
</span><span id="__span-17-5"><a id="__codelineno-17-5" name="__codelineno-17-5"></a><span class="gd">- class CustomHelmApp(KubernetesApp):</span>
</span><span id="__span-17-6"><a id="__codelineno-17-6" name="__codelineno-17-6"></a><span class="gi">+ class CustomHelmApp(HelmApp):</span>
</span><span id="__span-17-7"><a id="__codelineno-17-7" name="__codelineno-17-7"></a>
</span><span id="__span-17-8"><a id="__codelineno-17-8" name="__codelineno-17-8"></a><span class="w"> </span> @override
</span><span id="__span-17-9"><a id="__codelineno-17-9" name="__codelineno-17-9"></a><span class="gd">- def template(self) -&gt; None:</span>
</span><span id="__span-17-10"><a id="__codelineno-17-10" name="__codelineno-17-10"></a><span class="gi">+ def manifest(self) -&gt; Resource:</span>
</span><span id="__span-17-11"><a id="__codelineno-17-11" name="__codelineno-17-11"></a><span class="w"> </span> &quot;&quot;&quot;Render final component resources, e.g. Kubernetes manifests.&quot;&quot;&quot;
</span><span id="__span-17-12"><a id="__codelineno-17-12" name="__codelineno-17-12"></a><span class="w"> </span> return [] # list of manifests
</span><span id="__span-17-13"><a id="__codelineno-17-13" name="__codelineno-17-13"></a><span class="w"> </span> ...
</span></code></pre></div></td></tr></table></div>
</details>
<details class="example">
<summary>github_ci_workflow.yaml</summary>
<div class="language-diff highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-18-1">1</a></span>
<span class="normal"><a href="#__codelineno-18-2">2</a></span>
<span class="normal"><a href="#__codelineno-18-3">3</a></span>
<span class="normal"><a href="#__codelineno-18-4">4</a></span>
<span class="normal"><a href="#__codelineno-18-5">5</a></span></pre></div></td><td class="code"><div><pre><span></span><code><span id="__span-18-1"><a id="__codelineno-18-1" name="__codelineno-18-1"></a><span class="w"> </span> steps:
</span><span id="__span-18-2"><a id="__codelineno-18-2" name="__codelineno-18-2"></a><span class="w"> </span> - name: ...
</span><span id="__span-18-3"><a id="__codelineno-18-3" name="__codelineno-18-3"></a><span class="gd">- uses: bakdata/kpops/actions/kpops-runner@main</span>
</span><span id="__span-18-4"><a id="__codelineno-18-4" name="__codelineno-18-4"></a><span class="gi">+ uses: bakdata/kpops@main</span>
</span><span id="__span-18-5"><a id="__codelineno-18-5" name="__codelineno-18-5"></a><span class="w"> </span> ...
</span></code></pre></div></td></tr></table></div>
</details>
<!-- dprint-ignore-end -->



Expand Down

0 comments on commit ed250e5

Please sign in to comment.