Release v3.0.0 (#1392)

awslabs · Nov 6, 2024 · fbfd74d · fbfd74d
1 parent a159fa3
commit fbfd74d
Show file tree

Hide file tree

Showing 211 changed files with 22,029 additions and 3,671 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
diff --git a/README.md b/README.md
@@ -1,41 +1,38 @@
 # Amazon Kinesis Client Library for Java
 [![Build Status](https://travis-ci.org/awslabs/amazon-kinesis-client.svg?branch=master)](https://travis-ci.org/awslabs/amazon-kinesis-client)
 
-The **Amazon Kinesis Client Library for Java** (Amazon KCL) enables Java developers to easily consume and process data from [Amazon Kinesis][kinesis].
+The **Amazon Kinesis Client Library (KCL) for Java** enables Java developers to easily consume and process data from [Amazon Kinesis Data Streams][kinesis].
 
-* [Kinesis Product Page][kinesis]
-* [Forum][kinesis-forum]
+* [Kinesis Data Streams Product Page][kinesis]
+* [Amazon re:Post Forum: Kinesis][kinesis-forum]
 * [Javadoc][kcl-javadoc]
 * [FAQ](docs/FAQ.md)
-* [KCL Documentation](docs/) (folder)
+* [Developer Guide - Kinesis Client Library][kcl-aws-doc]
+* [KCL GitHub documentation](docs/) (folder)
 * [Issues][kinesis-client-library-issues]
-
-### Recommended Upgrade for All Users of the 1.x Amazon Kinesis Client
-:warning: We recommend customers to migrate to 1.14.1 or newer to avoid [known bugs](https://github.com/awslabs/amazon-kinesis-client/issues/778) in 1.14.0 version
-
-### Recommended Upgrade for All Users of the 2.x Amazon Kinesis Client
-**:warning: It's highly recommended for users of version 2.0 of the Amazon Kinesis Client to upgrade to version 2.0.3 or later. A [bug has been](https://github.com/awslabs/amazon-kinesis-client/issues/391) identified in versions prior to 2.0.3 that could cause records to be delivered to the wrong record processor.**  
-
-**:information_source: Amazon Kinesis Client versions 1.x are not impacted.**  
-
-Please open an issue if you have any questions.
+* [Giving Feedback][giving-feedback]
 
 ## Features
 
-* Provides an easy-to-use programming model for processing data using Amazon Kinesis
-* Helps with scale-out and fault-tolerant processing
+* **Scalability:** KCL enables applications to scale dynamically by distributing the processing load across multiple workers. You can scale your application in or out, manually or with auto-scaling, without worrying about load redistribution.
+* **Load balancing:** KCL automatically balances the processing load across available workers, resulting in an even distribution of work across workers.
+* **Checkpointing:** KCL manages checkpointing of processed records, enabling applications to resume processing from their last sucessfully processed position.
+* **Fault tolerance:** KCL provides built-in fault tolerance mechanisms, making sure that data processing continues even if individual workers fail. KCL also provides at-least-once delivery.
+* **Handling stream-level changes:** KCL adapts to shard splits and merges that might occur due to changes in data volume. It maintains ordering by making sure that child shards are processed only after their parent shard is completed and checkpointed.
+* **Monitoring:** KCL integrates with Amazon CloudWatch for consumer-level monitoring.
+* **Multi-language support:** KCL natively supports Java and enables multiple non-Java programming languages through MultiLangDaemon.
 
 ## Getting Started
 
 1. **Sign up for AWS** &mdash; Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see [AWS Account and Credentials][docs-signup] in the AWS SDK for Java Developer Guide.
-1. **Sign up for Amazon Kinesis** &mdash; Go to the Amazon Kinesis console to sign up for the service and create an Amazon Kinesis stream. For more information, see [Create an Amazon Kinesis Stream][kinesis-guide-create] in the Amazon Kinesis Developer Guide.
-1. **Minimum requirements** &mdash; To use the Amazon Kinesis Client Library, you'll need **Java 1.8+**. For more information about Amazon Kinesis Client Library requirements, see [Before You Begin][kinesis-guide-begin] in the Amazon Kinesis Developer Guide.
-1. **Using the Amazon Kinesis Client Library** &mdash; The best way to get familiar with the Amazon Kinesis Client Library is to read [Developing Record Consumer Applications][kinesis-guide-applications] in the Amazon Kinesis Developer Guide.
+2. **Sign up for Amazon Kinesis** &mdash; Go to the Amazon Kinesis console to sign up for the service and create an Amazon Kinesis stream. For more information, see [Create an Amazon Kinesis Stream][kinesis-guide-create] in the Amazon Kinesis Developer Guide.
+3. **Minimum requirements** &mdash; To use the Amazon Kinesis Client Library, you will need **Java 1.8+**. For more information about Amazon Kinesis Client Library requirements, see [Before You Begin][kinesis-guide-begin] in the Amazon Kinesis Developer Guide.
+4. **Using the Amazon Kinesis Client Library** &mdash; The best way to get familiar with the Amazon Kinesis Client Library is to read [Use Kinesis Client Library][kinesis-guide-applications] in the Amazon Kinesis Data Streams Developer Guide. For more information on core KCL concepts, please refer to the [KCL Concepts][kinesis-client-library-concepts] page.
 
 ## Building from Source
 
-After you've downloaded the code from GitHub, you can build it using Maven. To disable GPG signing in the build, use
-this command: `mvn clean install -Dgpg.skip=true`. 
+After you have downloaded the code from GitHub, you can build it using Maven. To disable GPG signing in the build, use
+this command: `mvn clean install -Dgpg.skip=true`.
 Note: This command does not run integration tests.
 
 To disable running unit tests in the build, add the property `-Dskip.ut=true`.
@@ -58,7 +55,17 @@ To make it easier for developers to write record processors in other languages,
 ## Using the KCL
 The recommended way to use the KCL for Java is to consume it from Maven.
 
+### Version 3.x
+  ``` xml
+  <dependency>
+      <groupId>software.amazon.kinesis</groupId>
+      <artifactId>amazon-kinesis-client</artifactId>
+      <version>3.0.0</version>
+  </dependency>
+  ```
+
 ### Version 2.x
+[Version 2.x tracking branch](https://github.com/awslabs/amazon-kinesis-client/tree/v2.x)
   ``` xml
   <dependency>
       <groupId>software.amazon.kinesis</groupId>
@@ -77,13 +84,36 @@ The recommended way to use the KCL for Java is to consume it from Maven.
   </dependency>
   ```
 
+> **IMPORTANT**
+> We recommend using the latest KCL version for improved performance and support.
+
 ## Release Notes
 
 | KCL Version | Changelog |
 | --- | --- |
-| 2.x | [master/CHANGELOG.md](CHANGELOG.md) |
+| 3.x | [master/CHANGELOG.md](CHANGELOG.md) |
+| 2.x | [v2.x/CHANGELOG.md](https://github.com/awslabs/amazon-kinesis-client/blob/v2.x/CHANGELOG.md) |
 | 1.x | [v1.x/CHANGELOG.md](https://github.com/awslabs/amazon-kinesis-client/blob/v1.x/CHANGELOG.md) |
 
+## Notices
+
+### Recommended Upgrade for All Users of the 1.x Amazon Kinesis Client
+We recommend customers to migrate to 1.14.1 or newer to avoid [known bugs](https://github.com/awslabs/amazon-kinesis-client/issues/778) in 1.14.0 version
+
+### Recommended Upgrade for All Users of the 2.x Amazon Kinesis Client
+It's highly recommended for users of version 2.0 of the Amazon Kinesis Client to upgrade to version 2.0.3 or later. A [bug has been](https://github.com/awslabs/amazon-kinesis-client/issues/391) identified in versions prior to 2.0.3 that could cause records to be delivered to the wrong record processor.**
+
+## Giving Feedback
+
+Help Us Improve the Kinesis Client Library! Your involvement is crucial to enhancing the Kinesis Client Library. We invite you to join our community and contribute in the following ways:
+
+* [Issue](https://github.com/awslabs/amazon-kinesis-client/issues) Reporting: This is our preferred method of communication. Use this channel to report bugs, suggest improvements, or ask questions.
+* Feature Requests: Share your ideas for new features or vote for existing proposals on our [Issues](https://github.com/awslabs/amazon-kinesis-client/issues) page. This helps us prioritize development efforts.
+* Participate in Discussions: Engage with other users and our team in our discussion forums.
+* Submit [Pull Requests](https://github.com/awslabs/amazon-kinesis-client/pulls): If you have developed a fix or improvement, we welcome your code contributions.
+
+By participating through these channels, you play a vital role in shaping the future of the Kinesis Client Library. We value your input and look forward to collaborating with you!
+
 [docs-signup]: http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-setup.html
 [kcl-javadoc]: https://javadoc.io/doc/software.amazon.kinesis/amazon-kinesis-client/
 [kinesis]: http://aws.amazon.com/kinesis
@@ -98,4 +128,7 @@ The recommended way to use the KCL for Java is to consume it from Maven.
 [kinesis-guide-consumer-deaggregation]: http://docs.aws.amazon.com//kinesis/latest/dev/kinesis-kpl-consumer-deaggregation.html
 [kclpy]: https://github.com/awslabs/amazon-kinesis-client-python
 [multi-lang-protocol]: /amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/multilang/package-info.java
-[migration-guide]: https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html
+[migration-guide]: https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-previous-versions
+[kcl-sample]: https://docs.aws.amazon.com/streams/latest/dev/kcl-example-code
+[kcl-aws-doc]: https://docs.aws.amazon.com/streams/latest/dev/kcl.html
+[giving-feedback]: https://github.com/awslabs/amazon-kinesis-client?tab=readme-ov-file#giving-feedback
diff --git a/amazon-kinesis-client-multilang/pom.xml b/amazon-kinesis-client-multilang/pom.xml
@@ -21,16 +21,12 @@
   <parent>
     <artifactId>amazon-kinesis-client-pom</artifactId>
     <groupId>software.amazon.kinesis</groupId>
-    <version>2.6.1-SNAPSHOT</version>
+    <version>3.0.0</version>
   </parent>
   <modelVersion>4.0.0</modelVersion>
 
   <artifactId>amazon-kinesis-client-multilang</artifactId>
 
-  <properties>
-    <aws-java-sdk.version>1.12.668</aws-java-sdk.version>
-  </properties>
-
   <dependencies>
     <dependency>
       <groupId>software.amazon.kinesis</groupId>
@@ -43,36 +39,10 @@
       <version>${awssdk.version}</version>
     </dependency>
 
-    <dependency>
-      <groupId>com.amazonaws</groupId>
-      <artifactId>aws-java-sdk-core</artifactId>
-      <version>${aws-java-sdk.version}</version>
-      <exclusions>
-        <exclusion>
-          <groupId>com.fasterxml.jackson.core</groupId>
-          <artifactId>jackson-databind</artifactId>
-        </exclusion>
-        <exclusion>
-          <groupId>com.fasterxml.jackson.dataformat</groupId>
-          <artifactId>jackson-dataformat-cbor</artifactId>
-        </exclusion>
-        <exclusion>
-          <groupId>org.apache.httpcomponents</groupId>
-          <artifactId>httpclient</artifactId>
-        </exclusion>
-      </exclusions>
-    </dependency>
-
-    <dependency>
-        <groupId>com.amazonaws</groupId>
-        <artifactId>aws-java-sdk-sts</artifactId>
-        <version>${aws-java-sdk.version}</version>
-    </dependency>
-
     <dependency>
       <groupId>org.projectlombok</groupId>
       <artifactId>lombok</artifactId>
-      <version>1.18.24</version>
+      <version>1.18.28</version>
       <scope>provided</scope>
     </dependency>
     <dependency>
@@ -104,6 +74,12 @@
     </dependency>
 
     <!-- Test -->
+    <dependency>
+      <groupId>org.junit.jupiter</groupId>
+      <artifactId>junit-jupiter-api</artifactId>
+      <version>5.11.3</version>
+      <scope>test</scope>
+    </dependency>
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>
@@ -122,6 +98,13 @@
       <version>1.3</version>
       <scope>test</scope>
     </dependency>
+    <!-- Using older version to be compatible with Java 8   -->
+    <dependency>
+      <groupId>org.mockito</groupId>
+      <artifactId>mockito-junit-jupiter</artifactId>
+      <version>3.12.4</version>
+      <scope>test</scope>
+    </dependency>
   </dependencies>
 
   <build>

diff --git a/...sis-client-multilang/src/main/java/software/amazon/kinesis/multilang/MultiLangDaemon.java b/...sis-client-multilang/src/main/java/software/amazon/kinesis/multilang/MultiLangDaemon.java
@@ -61,10 +61,10 @@
  * applicationName = PythonKCLSample
  *
  * # Users can change the credentials provider the KCL will use to retrieve credentials.
- * # The DefaultAWSCredentialsProviderChain checks several other providers, which is
+ * # The DefaultCredentialsProvider checks several other providers, which is
  * # described here:
- * # http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
- * AWSCredentialsProvider = DefaultAWSCredentialsProviderChain
+ * # https://sdk.amazonaws.com/java/api/2.0.0-preview-11/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html
+ * AwsCredentialsProvider = DefaultCredentialsProvider
  * </pre>
  */
 @Slf4j
@@ -141,7 +141,7 @@ void configureLogging(
         }
     }
 
-    String propertiesFile(final MultiLangDaemonArguments arguments) {
+    String validateAndGetPropertiesFileName(final MultiLangDaemonArguments arguments) {
         String propertiesFile = "";
 
         if (CollectionUtils.isNotEmpty(arguments.parameters)) {
@@ -216,9 +216,9 @@ public static void main(final String[] args) {
         MultiLangDaemonArguments arguments = new MultiLangDaemonArguments();
         JCommander jCommander = daemon.buildJCommanderAndParseArgs(arguments, args);
         try {
-            String propertiesFile = daemon.propertiesFile(arguments);
+            String propertiesFileName = daemon.validateAndGetPropertiesFileName(arguments);
             daemon.configureLogging(arguments.logConfiguration);
-            MultiLangDaemonConfig config = daemon.buildMultiLangDaemonConfig(propertiesFile);
+            MultiLangDaemonConfig config = daemon.buildMultiLangDaemonConfig(propertiesFileName);
 
             Scheduler scheduler = daemon.buildScheduler(config);
             MultiLangRunner runner = new MultiLangRunner(scheduler);

diff --git a/...s-client-multilang/src/main/java/software/amazon/kinesis/multilang/NestedPropertyKey.java b/...s-client-multilang/src/main/java/software/amazon/kinesis/multilang/NestedPropertyKey.java
@@ -15,13 +15,14 @@
 package software.amazon.kinesis.multilang;
 
 import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
 
-import com.amazonaws.regions.Regions;
 import com.google.common.base.CaseFormat;
 import lombok.AccessLevel;
 import lombok.Getter;
 import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.regions.Region;
 
 /**
  * Key-Value pairs which may be nested in, and extracted from, a property value
@@ -73,8 +74,13 @@ void visit(final NestedPropertyProcessor processor, final String endpoint) {
      * @see <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions">Available Regions</a>
      */
     ENDPOINT_REGION {
-        void visit(final NestedPropertyProcessor processor, final String region) {
-            processor.acceptEndpointRegion(Regions.fromName(region));
+        void visit(final NestedPropertyProcessor processor, final String regionName) {
+            List<Region> validRegions = Region.regions();
+            Region region = Region.of(regionName);
+            if (!validRegions.contains(region)) {
+                throw new IllegalArgumentException("Invalid region name: " + regionName);
+            }
+            processor.acceptEndpointRegion(region);
         }
     },
 

diff --git a/...nt-multilang/src/main/java/software/amazon/kinesis/multilang/NestedPropertyProcessor.java b/...nt-multilang/src/main/java/software/amazon/kinesis/multilang/NestedPropertyProcessor.java
@@ -14,7 +14,7 @@
  */
 package software.amazon.kinesis.multilang;
 
-import com.amazonaws.regions.Regions;
+import software.amazon.awssdk.regions.Region;
 
 /**
  * Defines methods to process {@link NestedPropertyKey}s.
@@ -26,11 +26,11 @@ public interface NestedPropertyProcessor {
      *
      * @param serviceEndpoint the service endpoint either with or without the protocol
      *      (e.g., https://sns.us-west-1.amazonaws.com, sns.us-west-1.amazonaws.com)
-     * @param signingRegion the region to use for SigV4 signing of requests (e.g. us-west-1)
+     * @param signingRegion the region to use for the client (e.g. us-west-1)
      *
-     * @see #acceptEndpointRegion(Regions)
-     * @see <a href="https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/client/builder/AwsClientBuilder.EndpointConfiguration.html">
-     *     AwsClientBuilder.EndpointConfiguration</a>
+     * @see #acceptEndpointRegion(Region)
+     * @see <a href="https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/client/builder/SdkClientBuilder.html#endpointOverride(java.net.URI)">
+     *     AwsClientBuilder.endpointOverride</a>
      */
     void acceptEndpoint(String serviceEndpoint, String signingRegion);
 
@@ -42,7 +42,7 @@ public interface NestedPropertyProcessor {
      *
      * @see #acceptEndpoint(String, String)
      */
-    void acceptEndpointRegion(Regions region);
+    void acceptEndpointRegion(Region region);
 
     /**
      * Set the external id, an optional field to designate who can assume an IAM role.

diff --git a/...va/software/amazon/kinesis/multilang/auth/KclSTSAssumeRoleSessionCredentialsProvider.java b/...va/software/amazon/kinesis/multilang/auth/KclSTSAssumeRoleSessionCredentialsProvider.java