Skip to content

Commit

Permalink
Merge pull request #57548 from gabriel-rh/SRVKE-1366-namespaced-kafka…
Browse files Browse the repository at this point in the history
…-broker

SRVKE-1366 Namespaced Kafka broker
  • Loading branch information
gabriel-rh authored Apr 12, 2023
2 parents cc10a26 + 050d3cf commit cb26a3c
Show file tree
Hide file tree
Showing 3 changed files with 97 additions and 1 deletion.
70 changes: 70 additions & 0 deletions modules/serverless-create-kafka-namespaced-broker.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
// Module included in the following assemblies:
//
// * serverless/eventing/brokers/kafka-broker.adoc

:_content-type: PROCEDURE
[id="serverless-create-kafka-namespaced-broker-{context}"]
= Creating a Knative broker for Apache Kafka that uses an isolated data plane

:FeatureName: The Knative Broker implementation for Apache Kafka with isolated data plane
include::snippets/technology-preview.adoc[leveloffset=+2]

To create a `KafkaNamespaced` broker, you must set the `eventing.knative.dev/broker.class` annotation to `KafkaNamespaced`.

.Prerequisites

* The {ServerlessOperatorName}, Knative Eventing, and the `KnativeKafka` custom resource are installed on your {product-title} cluster.
* You have access to an Apache Kafka instance, such as link:https://access.redhat.com/documentation/en-us/red_hat_amq/7.6/html/amq_streams_on_openshift_overview/kafka-concepts_str#kafka-concepts-key_str[Red Hat AMQ Streams], and have created a Kafka topic.
* You have created a project, or have access to a project, with the appropriate roles and permissions to create applications and other workloads in {product-title}.
* You have installed the OpenShift CLI (`oc`).
.Procedure

. Create an Apache Kafka-based broker by using a YAML file:
+
[source,yaml]
----
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
annotations:
eventing.knative.dev/broker.class: KafkaNamespaced <1>
name: default
namespace: my-namespace <2>
spec:
config:
apiVersion: v1
kind: ConfigMap
name: my-config <2>
...
----
<1> To use the Apache Kafka broker with isolated data planes, the broker class value must be `KafkaNamespaced`.
<2> The referenced `ConfigMap` object `my-config` must be in the same namespace as the `Broker` object, in this case `my-namespace`.

. Apply the Apache Kafka-based broker YAML file:
+
[source,terminal]
----
$ oc apply -f <filename>
----

[IMPORTANT]
====
The `ConfigMap` object in `spec.config` must be in the same namespace as the `Broker` object:
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config
namespace: my-namespace
data:
...
----
====

After the creation of the first `Broker` object with the `KafkaNamespaced` class, the `kafka-broker-receiver` and `kafka-broker-dispatcher` deployments are created in the namespace. Subsequently, all brokers with the `KafkaNamespaced` class in the same namespace will use the same data plane. If no brokers with the `KafkaNamespaced` class exist in the namespace, the data plane in the namespace is deleted.
25 changes: 25 additions & 0 deletions modules/serverless-kafka-broker-with-isolated-dataplane.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
// Module included in the following assemblies:
//
// * serverless/eventing/brokers/kafka-broker.adoc

:_content-type: CONCEPT
[id="serverless-kafka-broker-with-isolated-dataplane_{context}"]
= Knative Broker implementation for Apache Kafka with isolated data plane

:FeatureName: The Knative Broker implementation for Apache Kafka with isolated data plane
include::snippets/technology-preview.adoc[leveloffset=+2]

The Knative Broker implementation for Apache Kafka has 2 planes:

Control plane:: Consists of controllers that talk to the Kubernetes API, watch for custom objects, and manage the data plane.

Data plane:: The collection of components that listen for incoming events, talk to Apache Kafka, and send events to the event sinks. The Knative Broker implementation for Apache Kafka data plane is where events flow. The implementation consists of `kafka-broker-receiver` and `kafka-broker-dispatcher` deployments.

When you configure a Broker class of `Kafka`, the Knative Broker implementation for Apache Kafka uses a shared data plane. This means that the `kafka-broker-receiver` and `kafka-broker-dispatcher` deployments in the `knative-eventing` namespace are used for all Apache Kafka Brokers in the cluster.

However, when you configure a Broker class of `KafkaNamespaced`, the Apache Kafka broker controller creates a new data plane for each namespace where a broker exists. This data plane is used by all `KafkaNamespaced` brokers in that namespace. This provides isolation between the data planes, so that the `kafka-broker-receiver` and `kafka-broker-dispatcher` deployments in the user namespace are only used for the broker in that namespace.

[IMPORTANT]
====
As a consequence of having separate data planes, this security feature creates more deployments and uses more resources. Unless you have such isolation requirements, use a *regular* Broker with a class of `Kafka`.
====
3 changes: 2 additions & 1 deletion serverless/eventing/brokers/kafka-broker.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ If your {ServerlessProductName} deployment is not configured to use Kafka broker

include::modules/serverless-kafka-broker.adoc[leveloffset=+2]
include::modules/serverless-kafka-broker-with-kafka-topic.adoc[leveloffset=+2]

include::modules/serverless-kafka-broker-with-isolated-dataplane.adoc[leveloffset=+2]
include::modules/serverless-create-kafka-namespaced-broker.adoc[leveloffset=+2]

// kafka broker general configmap
include::modules/serverless-kafka-broker-configmap.adoc[leveloffset=+1]
Expand Down

0 comments on commit cb26a3c

Please sign in to comment.