feat: Add EMR runtime for flink operator blueprint #485

mithun008 · 2024-04-04T06:14:30Z

What does this PR do?

🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

Added blue print to support EMR Flink operator on EKS.

Motivation

EMR Flink operator enhances the open source Flink operator by addition AWS specific capabilities for scaling, logging and monitoring.

More

[ x] Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
[ x] Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
[ x] Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
[ x] Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

E2E Test successfully complete before merge?

Additional Notes

mengfwanaws · 2024-04-11T13:57:33Z

streaming/emr-eks-flink/README.md

+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.72 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |


Suggest to use Helm >= v3.8.0 where OCI support is enabled by default. https://helm.sh/docs/topics/registries/

This is the Terraform provider version and the latest version is 2.13.0 https://registry.terraform.io/providers/hashicorp/helm/latest/docs/guides/v2-upgrade-guide and it uses latest Helm version 3.13.2 https://github.com/hashicorp/terraform-provider-helm/blob/8d8f0503c39f9fb2d718805e5a619ef56728271b/go.mod#L13

@mithun008 Upgrade this to 2.13 in versions.tf file and rerun it

mengfwanaws · 2024-04-11T13:59:41Z

streaming/emr-eks-flink/addons.tf

+  enable_aws_load_balancer_controller = true
+
+  #---------------------------------------
+  # AWS for FluentBit - DaemonSet


We don't use FluentBit and DaemonSet. Instead, you may set up monitoring configurations to archive application logs to S3/CW.

You are absolutely right @mengfwanaws! We use FluentBit to capture the logs from other k8s add-ons and apps running on the EKS cluster.

mengfwanaws · 2024-04-11T14:00:48Z

streaming/emr-eks-flink/examples/basic-example-app-cluster.yaml

+
+
+
+  flinkVersion: v1_17


emr-7.0.0 should use v1_18.

mengfwanaws · 2024-04-11T14:01:30Z

streaming/emr-eks-flink/examples/basic-example-app-cluster.yaml

+  # add service account
+  # serviceAccount: flink-team-a-sa
+
+  executionRoleArn: arn:aws:iam::681921237057:role/emr-eks-flink-flink-team-a-20240406012025932700000008


Who's this 681921237057? Shall we dynamically set it up for customer acct?

removed it and added comments as well as docs.

mengfwanaws · 2024-04-11T14:03:08Z

streaming/emr-eks-flink/examples/cluster-autoscaler/flink-sample-job.yaml

+    taskmanager.numberOfTaskSlots: "2"
+    state.savepoints.dir: file:///flink/data/checkpoint/savepoints
+    state.checkpoints.dir: file:///flink/data/checkpoint/checkpoints
+    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory


Deprecated config. Use high-availability.type: kubernetes instead.

mengfwanaws · 2024-04-11T14:08:15Z

streaming/emr-eks-flink/helm-values/aws-for-fluentbit-values.yaml

We don't use fluentbit. Please refer to monitoring configuration instead.

mengfwanaws · 2024-04-11T14:09:50Z

website/docs/blueprints/streaming-platforms/emr-eks-flink.md

+3. Automatically tunes Autoscaler configurations based on historical trends of observed metrics.
+4. Faster Flink Job Restart during scaling or Failure Recovery
+5. IRSA (IAM Roles for Service Accounts) Native Integration
+6. Apache Airflow Integration


I don't see an example in this PR.

vara-bonthu

Thanks for the PR, @mithun008! I've suggested some changes.

Overall, I would like you to align the blueprint more closely with emr-eks-karpenter. Additionally, we should include more examples, which can be added in subsequent PRs.

vara-bonthu · 2024-04-16T16:19:28Z

streaming/emr-eks-flink/addons.tf

+  enable_aws_load_balancer_controller = true
+
+  #---------------------------------------
+  # AWS for FluentBit - DaemonSet


You are absolutely right @mengfwanaws! We use FluentBit to capture the logs from other k8s add-ons and apps running on the EKS cluster.

README.md

vara-bonthu · 2024-04-16T16:28:25Z

streaming/emr-eks-flink/README.md

+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.72 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |


This is the Terraform provider version and the latest version is 2.13.0 https://registry.terraform.io/providers/hashicorp/helm/latest/docs/guides/v2-upgrade-guide and it uses latest Helm version 3.13.2 https://github.com/hashicorp/terraform-provider-helm/blob/8d8f0503c39f9fb2d718805e5a619ef56728271b/go.mod#L13

@mithun008 Upgrade this to 2.13 in versions.tf file and rerun it

vara-bonthu · 2024-04-16T16:29:44Z

streaming/emr-eks-flink/addons.tf

+  # For example only - please evaluate for your environment
+  force_destroy = true
+
+  attach_deny_insecure_transport_policy = true


these are defaults i think now

streaming/emr-eks-flink/addons.tf

vara-bonthu · 2024-04-16T16:42:45Z

streaming/emr-eks-flink/main.tf

+        "karpenter.sh/discovery" = local.name
+      }
+    }
+


remove extra lines

vara-bonthu · 2024-04-16T16:43:01Z

streaming/emr-eks-flink/main.tf

+}
+
+#import module vpc
+module "vpc" {


move this to vpc.tf file

follow the existing emr-eks-karpenter blueprint as guidance https://github.com/awslabs/data-on-eks/blob/main/analytics/terraform/emr-eks-karpenter/vpc.tf

vara-bonthu · 2024-04-16T16:44:48Z

streaming/emr-eks-flink/main.tf

+
+
+# deploy a helm chart for flink-kubernetes-operator
+resource "helm_release" "flink_kubernetes_operator" {


remove this and use this add-on from EKS Data addons https://github.com/aws-ia/terraform-aws-eks-data-addons/blob/main/emr-flink-operator.tf. Update the addon in this repo if required

vara-bonthu · 2024-04-16T16:45:22Z

streaming/emr-eks-flink/outputs.tf

@@ -0,0 +1,9 @@
+# create output for flink operator role arn


add this

################################################################################ # EKS Managed Node Group ################################################################################ output "configure_kubectl" { description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig" value = "aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name}" }

vara-bonthu · 2024-04-16T16:46:09Z

streaming/emr-eks-flink/providers.tf

@@ -0,0 +1,36 @@
+


remove this file and move these providers to main.tf

mengfwanaws · 2024-04-22T21:01:09Z

streaming/emr-eks-flink/addons.tf

@@ -143,11 +139,27 @@ module "eks_blueprints_addons" {
 # Data on EKS Kubernetes Addons
 #---------------------------------------------------------------
 module "eks_data_addons" {
+
+  depends_on = [module.flink_irsa_jobs, module.flink_irsa_operator]
+  #source  = "[email protected]:mithun008/terraform-aws-eks-data-addons/"


Please remove the comment.

mengfwanaws · 2024-04-22T21:02:33Z

streaming/emr-eks-flink/examples/karpenter/basic-example-app-cluster.yaml

-
-  executionRoleArn: arn:aws:iam::xxxxxxxxx:role/emr-eks-flink-flink-team-a-20240406012025932700000008
+  # Replace this execution role ARN with your own
+  executionRoleArn: arn:aws:iam::681921237057:role/emr-eks-flink-flink-team-a-20240406012025932700000008


We should substitute this with variables, or provide instructions on how to generate the role arn (including the permissions).

I have the instructions in the website docs to substitute these values.

mengfwanaws · 2024-04-22T21:03:39Z

streaming/emr-eks-flink/examples/karpenter/graviton-example-app-cluster.yaml

+    state.savepoints.dir: s3://emr-flink-data/savepoints
+
+  # Replace this execution role ARN with your own
+  executionRoleArn: arn:aws:iam::681921237057:role/emr-eks-flink-flink-team-a-20240406012025932700000008


mithun008 · 2024-04-23T03:37:58Z

@vara-bonthu I have addressed your comments. Thank you!

vara-bonthu · 2024-04-24T00:27:50Z

streaming/emr-eks-flink/outputs.tf

@@ -0,0 +1,27 @@
+# create output for flink operator role arn
+output "flink_jobs_role_arn" {


flink_job_execution_role_arn

vara-bonthu · 2024-04-24T00:28:11Z

streaming/emr-eks-flink/outputs.tf

+output "flink_checkpoint_path" {
+  value       = "s3://${module.s3_bucket.s3_bucket_id}/checkpoints"
+  description = "S3 path for checkpoint data"
+}
+output "flink_savepoint_path" {
+  value       = "s3://${module.s3_bucket.s3_bucket_id}/savepoints"
+  description = "S3 path for savepoint data"
+}
+output "flink_jobmanager_path" {
+  value       = "s3://${module.s3_bucket.s3_bucket_id}/jobmanager"
+  description = "S3 path for jobmanager data"
+}
+
+output "flink_logs_path" {
+  value       = "s3://${module.s3_bucket.s3_bucket_id}/logs"
+  description = "S3 path for logs"
+}


remove all of these and add only s3bucet id

vara-bonthu · 2024-04-24T00:28:37Z

streaming/emr-eks-flink/examples/karpenter/basic-example-app-cluster.yaml

+
+    jobmanager.scheduler: adaptive
+    # Replace with s3 bucket in your own account
+    state.checkpoints.dir: s3://emr-flink-data/checkpoints


<<REPLACE_WITH_S3_BUCKET_ID>>

vara-bonthu · 2024-04-24T00:29:45Z

streaming/emr-eks-flink/versions.tf

+  # backend "s3" {
+  #   bucket = "doeks-github-actions-e2e-test-state"
+  #   region = "us-west-2"
+  #   key    = "e2e/flink/terraform.tfstate"


emr-eks-flink

vara-bonthu

left few minor comments otherwise its good

vara-bonthu · 2024-04-24T18:49:16Z

streaming/emr-eks-flink/addons.tf

+  # EMR Flink operator
+  #---------------------------------------------------------------
+  enable_emr_flink_operator = true
+  emr_flink_operator_helm_config = {


replace this bloc with

emr_flink_operator_helm_config = { repository = "oci://public.ecr.aws/emr-on-eks" operatorExecutionRoleArn = module.flink_irsa_operator.iam_role_arn }

You can remove the repository and operatorExecutionRoleArn once you update these in Data addons repo

vara-bonthu · 2024-04-24T18:50:14Z

streaming/emr-eks-flink/examples/cluster-autoscaler/flink-sample-job.yaml

@@ -0,0 +1,64 @@
+# NOTE: Make sure you you replce <ENTER_S3_BUCKET> with your flink_operator_bucket output before running this job


check spelling and repeated words

vara-bonthu · 2024-04-24T18:50:32Z

streaming/emr-eks-flink/examples/karpenter/basic-example-app-cluster.yaml

@@ -0,0 +1,74 @@
+# NOTE: Make sure you you replce <ENTER_S3_BUCKET> and with your S3 Bucket before running this job.


check spelling and repeated words

vara-bonthu · 2024-04-24T18:50:44Z

streaming/emr-eks-flink/examples/karpenter/graviton-example-app-cluster.yaml

@@ -0,0 +1,75 @@
+# NOTE: Make sure you you replce <ENTER_S3_BUCKET> with your flink_operator_bucket output before running this job


check spelling and repeated words

vara-bonthu

/lgtm

Co-authored-by: Mithun Mallick <[email protected]>

Mithun Mallick added 11 commits March 15, 2024 13:01

EMR EKS Flnk blue rpint

461b27a

fixes based on pre-commit

4681e36

fixes based on pre-commit

51eb6ba

added readme

883ef3d

fixed readme alignment

b6987da

applied changes to have auto sa creation

0fc6f27

using irsa for operator role

3096d80

karpenter related changes

1ca710b

doc added

6a1cf6c

ran pre-commit

44a100b

pre-commit changes

ddf17e2

mithun008 temporarily deployed to DoEKS Test April 4, 2024 06:14 — with GitHub Actions Inactive

karpenter config fixes

3cc5bad

mithun008 temporarily deployed to DoEKS Test April 6, 2024 02:09 — with GitHub Actions Inactive

added spot instace

880c182

mithun008 temporarily deployed to DoEKS Test April 6, 2024 18:15 — with GitHub Actions Inactive

updated README

a1ee994

mithun008 temporarily deployed to DoEKS Test April 6, 2024 18:23 — with GitHub Actions Inactive

mengfwanaws reviewed Apr 11, 2024

View reviewed changes

PR comments

978659a

mithun008 temporarily deployed to DoEKS Test April 14, 2024 20:45 — with GitHub Actions Inactive

vara-bonthu reviewed Apr 16, 2024

View reviewed changes

addressed PR review comments

3879966

mithun008 had a problem deploying to DoEKS Test April 22, 2024 20:57 — with GitHub Actions Error

mengfwanaws reviewed Apr 22, 2024

View reviewed changes

removed invalid source line

527668b

mithun008 had a problem deploying to DoEKS Test April 22, 2024 21:09 — with GitHub Actions Error

removed account id

6e91fea

mithun008 had a problem deploying to DoEKS Test April 22, 2024 21:15 — with GitHub Actions Error

updated web site doc

7560e0c

mithun008 had a problem deploying to DoEKS Test April 22, 2024 21:20 — with GitHub Actions Error

removed lines

5d0d834

mithun008 had a problem deploying to DoEKS Test April 23, 2024 23:54 — with GitHub Actions Error

vara-bonthu reviewed Apr 24, 2024

View reviewed changes

changes based on PR review

cd38905

mithun008 had a problem deploying to DoEKS Test April 24, 2024 18:02 — with GitHub Actions Error

updated website docs

bfb8b37

mithun008 had a problem deploying to DoEKS Test April 24, 2024 18:13 — with GitHub Actions Error

vara-bonthu reviewed Apr 24, 2024

View reviewed changes

fixed typos and helm config

b3538e1

mithun008 had a problem deploying to DoEKS Test April 24, 2024 19:44 — with GitHub Actions Error

fixed case of the output

dd43a6a

mithun008 had a problem deploying to DoEKS Test April 24, 2024 19:48 — with GitHub Actions Error

fixed typo

24585f2

mithun008 had a problem deploying to DoEKS Test April 25, 2024 16:17 — with GitHub Actions Failure

vara-bonthu approved these changes Apr 25, 2024

View reviewed changes

vara-bonthu merged commit ad20de8 into awslabs:main Apr 25, 2024
36 of 39 checks passed

ovaleanu pushed a commit to ovaleanu/data-on-eks that referenced this pull request Aug 10, 2024

feat: Add EMR runtime for flink operator blueprint (awslabs#485)

f1e5d10

Co-authored-by: Mithun Mallick <[email protected]>



		# deploy a helm chart for flink-kubernetes-operator
		resource "helm_release" "flink_kubernetes_operator" {

		@@ -0,0 +1,27 @@
		# create output for flink operator role arn
		output "flink_jobs_role_arn" {

		@@ -0,0 +1,64 @@
		# NOTE: Make sure you you replce <ENTER_S3_BUCKET> with your flink_operator_bucket output before running this job

		@@ -0,0 +1,74 @@
		# NOTE: Make sure you you replce <ENTER_S3_BUCKET> and with your S3 Bucket before running this job.

		@@ -0,0 +1,75 @@
		# NOTE: Make sure you you replce <ENTER_S3_BUCKET> with your flink_operator_bucket output before running this job

feat: Add EMR runtime for flink operator blueprint #485

feat: Add EMR runtime for flink operator blueprint #485

Conversation

mithun008 commented Apr 4, 2024

What does this PR do?

Motivation

More

For Moderators

Additional Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vara-bonthu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mithun008 commented Apr 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vara-bonthu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vara-bonthu left a comment

Choose a reason for hiding this comment