You can host an Apache Kafka cluster on AWS, or on any other cloud provider of your choice. Lambda supports Kafka as an event source regardless of where it is hosted, as long as Lambda can access the cluster.
This page describes the AWS and non-AWS hosting options available for using your Kafka cluster as an for your Lambda function.
To host your Apache Kafka cluster and topics, you can use any non-AWS cloud provider, such as CloudKarafka.
When you create a Kafka cluster using a non-AWS provider, you receive the connection information for your cluster. This information includes the Kafka cluster hostname, topic name, Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM) user name and password, and bootstrap server host-port pairs.
For more information about using a non-AWS hosted Apache Kafka cluster, see Using Lambda with self-managed Apache Kafka.
To host your Apache Kafka cluster and topics, you can use Amazon Managed Streaming for Apache Kafka (Amazon MSK).
When you create an Amazon MSK cluster, you receive the required hosting and connection information of the cluster. This information includes the Kafka cluster hostname, topic name, SASL/SCRAM user name and password, and bootstrap server host-port pairs.
To support your Kafka cluster on Amazon MSK, you might need to create Amazon Virtual Private Cloud (Amazon VPC) networking components. For more information, see Using Amazon MSK as an event source for AWS Lambda on the AWS Compute Blog.
For more information about using an MSK cluster, see Using Lambda with Amazon MSK.
You can also use other AWS hosting options for your Apache Kafka cluster and topics. For more information, see Best Practices for Running Apache Kafka on AWS on the AWS Big Data Blog.
For more information about using an AWS hosted Apache Kafka cluster on another AWS service, such as Amazon Elastic Compute Cloud (Amazon EC2), see Using Lambda with self-managed Apache Kafka.