External access to our EKS Kubernetes cluster is provided via AWS elastic load balancers. The AWS Load Balancer Controller provisions and manages the following resources:
-
An AWS Application Load Balancer (ALB) when you create a Kubernetes Ingress.
-
An AWS Network Load Balancer (NLB) when you create a Kubernetes Service of type LoadBalancer.
You can provide external access to Kubernetes apps by creating an Kubernetes Ingress resource.
The Kubernetes cluster contains a default IngressClass with the name aws-alb.
By default, if no ingressClassName is specified, all Ingress resources are managed by the deployed AWS Load Balancer Controller.
The default IngressClass defines a group with the name default, which ensures, that all Ingresses are using the same AWS ALB.
If you need to create an Ingress which is located on another AWS ALB you need to create an additional IngressClass.
AWS ALB specific configuration is defined with additional Annotations which need to be added to the Ingress resource.
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/healthcheck-path: /The annotation alb.ingress.kubernetes.io/target-type: ip specifies, that the traffic is routed directly to the individual pods. This is the prevered approach for AWS EKS, because every Pod has a reachable IP address.
The AWS Load Balancer Controller automatically adjusts the AWS target group, if a Pod is added or removed.
You find a complete overview about all allowed annotations on the AWS Load Balancer Controller Guide.
You can expose via HTTP if you add {"HTTP": 80} to the annotation alb.ingress.kubernetes.io/listen-ports.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/target-type: ip
name: nginx-hello-http
spec:
rules:
- host: nginx-hello.ada.letuscode.dev
http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: nginx-hello
port:
number: 80The host rule nginx-hello.ada.letuscode.dev is important, because by default multiple Ingresses share the same AWS ALB. If all requests independent of the path should be forwarded to a specific target, it is also important to specify '/*' as path.
The target of an Ingress is defined as a Kubernetes service. This service must exist and be of type ClusterIp and point to an group of pods. If target-type is ip, the load balancer will directly forward the requests to the individual ports, without using the service. In this case the service is only required by the AWS Load Balancer Controller to detect the actual Pods.
For a complete example see ingress-http.yaml and deployment.yaml
The example creates an deployment with three pods using nginxdemos/hello:plain-text as image.
kubectl -n examples-$(whoami) apply -f deployment.yamlThe Ingress as created for host nginx-hello.ada.letuscode.dev and path /*.
kubectl -n example-$(whoami) apply -f ingress-http.yamlAfter the Ingress has been created, it takes some time until the AWS Load Balancer Controller created the required AWS resources.
kubectl -n examples-$(whoami) get ingress nginx-hello-http -o yaml
export LB_DNS_NAME=$(kubectl -n examples-$(whoami) get ingress nginx-hello-http -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')The attribute status.loadBalancer.ingress.hostname shows the created DNS name of the loadbalancer.
curl -H "Host: nginx-hello-http.ada.letuscode.dev" http://${LB_DNS_NAME}In order to make a request to nginx-hello-http.ada.letuscode.dev you need to create a corresponding Route53 entry.
aws elbv2 describe-load-balancers | jq ".LoadBalancers[] | select(.DNSName | contains(\"${LB_DNS_NAME}\"))"
export TOOLS_LB_ARN=$(aws elbv2 describe-load-balancers | jq -r ".LoadBalancers[] | select(.DNSName | contains(\"${LB_DNS_NAME}\")) | .LoadBalancerArn")A load balancer can have multiple listeners. This configuration created a listener for port 80.
aws elbv2 describe-listeners --load-balancer-arn ${TOOLS_LB_ARN}
export LB_LISTENER_80_ARN=$(aws elbv2 describe-listeners --load-balancer-arn ${TOOLS_LB_ARN} | jq -r ".Listeners[] | select(.Port==80) | .ListenerArn")A listener can have multiple rules. This configuration created a rule for host nginx-hello-http.ada.letuscode.dev and path /*.
aws elbv2 describe-rules --listener-arn ${LB_LISTENER_80_ARN}
export NGINX_HELLO_HTTP_TG_ARN=$(aws elbv2 describe-rules --listener-arn ${LB_LISTENER_80_ARN} | jq -r ".Rules[] | select(.Conditions[].Values[] | contains(\"nginx-hello-http\")) | .Actions[0].ForwardConfig.TargetGroups[0].TargetGroupArn")If the rule matches, requests are forwared to a node defined by the assigned target group. The AWS Load Balancer Controller automatically adds the IPs of the Pods of the specified service to this target group.
aws elbv2 describe-target-health --target-group-arn ${NGINX_HELLO_HTTP_TG_ARN}You can expose via HTTP if you add {"HTTPS": 443} to the annotation alb.ingress.kubernetes.io/listen-ports.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/target-type: ip
name: nginx-hello-https
spec:
rules:
- host: nginx-hello-https.ada.letuscode.dev
http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: nginx-hello
port:
name: httpBy default, the AWS Load Balancer Controller automaticaly discovers a suitable certificate based on the defined host.
If you need to explicitly specify a Certificate, you can add the annotation alb.ingress.kubernetes.io/certificate-arn and set the ARN for the required server certificate.
alb.ingress.kubernetes.io/certificate-arn: ${PUBLIC_HOSTED_ZONE_CERT_ARN}For a complete example see ingress-https.yaml and deployment.yaml
The example creates an deployment with three pods using nginxdemos/hello:plain-text as image.
kubectl -n examples-$(whoami) apply -f deployment.yamlThe Ingress as created for host nginx-hello-https.ada.letuscode.dev and path /*.
kubectl -n example-$(whoami) apply -f ingress-https.yamlAfter the Ingress has been created, it takes some time until the AWS Load Balancer Controller created the required AWS resources.
kubectl -n examples-$(whoami) get ingress nginx-hello-https -o yaml
export LB_DNS_NAME=$(kubectl -n examples-$(whoami) get ingress nginx-hello-https -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')The attribute status.loadBalancer.ingress.hostname shows the created DNS name of the loadbalancer.
This sould be the same load balancer which is used for the HTTP example, because the Ingress group is the same.
dig ${LB_DNS_NAME}
export TOOLS_LB_IP=$(dig +short ${LB_DNS_NAME} | tail -n1)Now you can explicitly map the load balancer ip to the expected host name of the service via --resolve command line parameter of curl and send the request. This is required, because also the SNI is checked, which is always set to the actuall host of the Url.
curl --resolve nginx-hello-https.ada.letuscode.dev:443:${TOOLS_LB_IP} https://nginx-hello-https.ada.letuscode.devIn order to make a request to nginx-hello-https.ada.letuscode.dev you need to create a corresponding Route53 entry.
For Kubernetes Services and Ingresses you cann add the external-dns.alpha.kubernetes.io/hostname annotation to the service or ingress: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/faq.md#how-do-i-specify-a-dns-name-for-my-kubernetes-objects
Example:
apiVersion: v1
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/hostname: nginx-hello-https.ada.letuscode.devFor Kubernetes Ingress, also the Host which is specified in the ingress is considered.
Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
ingressClassName: alb
rules:
- host: nginx-hello-https.ada.letuscode.dev