Network Policy Not Enforced on Initial Creation #271

kervrosales · 2024-05-17T17:39:16Z

I am experiencing an issue with network policies not being enforced upon their initial creation in my Kubernetes cluster using the AWS Network Policy Agent. The policies only take effect after being deleted and re-created. Below are the details of my configuration.

Kubernetes YAML resources

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-ingress
  namespace: demo
spec:
  podSelector: {}
  policyTypes:
  - Ingress
Kubernetes Resources:
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app
  namespace: demo
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: demo-app
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-app-index
  namespace: demo
data:
  index.html: |
    <!DOCTYPE html>
    <html>
      <head>
        <title>Welcome to Amazon EKS!</title>
        <style>
            html {color-scheme: light dark;}
            body {width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif;}
        </style>
      </head>
      <body>
        <h1>Welcome to Amazon EKS!</h1>
        <p>If you see this page, you are able successfully access the web application as the network policy allows.</p>
        <p>For online documentation and installation instructions please refer to
          <a href="https://docs.aws.amazon.com/eks/latest/userguide/eks-networking.html">Amazon EKS Networking</a>.<br/><br/>
          The migration guides are available at
          <a href="https://docs.aws.amazon.com/eks/latest/userguide/eks-networking.html">Amazon EKS Network Policy Migration</a>.
        </p>
        <p><em>Thank you for using Amazon EKS.</em></p>
    </body>
    </html>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: demo
spec:
  selector:
    matchLabels:
      app: demo-app
  replicas: 1
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
      - name: demo
        image: public.ecr.aws/docker/library/nginx:stable
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: nginx-index-volume
      volumes:
      - name: nginx-index-volume
        configMap:
          name: demo-app-index
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: client-one
  name: client-one
  namespace: demo
spec:
  containers:
  - image: curlimages/curl:latest
    name: client-one
    command:
    - sleep
    - "360000"
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: client-two
  name: client-two
  namespace: demo
spec:
  containers:
  - image: curlimages/curl:latest
    name: client-two
    command:
    - sleep
    - "360000"

What happened:
When the network policy is initially created, it does not enforce the ingress rules as expected. I am still able to access the demo-app service from the client-one pod. However, after deleting and re-creating the network policy, the policy is enforced correctly, and access is denied as expected.

Attach logs
Initial Network Policy Creation logs from Network Policy agent:
not-working-infra.json

Delete of Network Policy logs:
delete-log-infra.json

Recreation of Network policy logs:
working-infra.json

What you expected to happen:
The network policy should enforce the ingress rules upon initial creation without requiring deletion and re-creation.

How to reproduce it (as minimally and precisely as possible):

Create the namespace and apply the network policy:

kubectl create namespace demo
kubectl apply -f network-policy.yaml

Deploy the Kubernetes resources:

kubectl apply -f kubernetes-resources.yaml

Test connectivity:

kubectl exec -it client-one -- curl demo-app # This returns HTML

Delete and re-create the network policy:

kubectl delete -f network-policy.yaml
kubectl apply -f network-policy.yaml

Test connectivity again:

kubectl exec -it client-one -- curl demo-app # This times out

Anything else we need to know?:
I have verified the CNI plugin configuration and ensured that it supports network policies. This issue seems to be related to the timing or synchronization of the policy application.

Environment:
EKS Version: 1.27
CNI Plugin: v1.18.1-eksbuild.1

The text was updated successfully, but these errors were encountered:

achevuru · 2024-06-03T20:51:31Z

@kervrosales Are you saying deleting and recreating the NP solved it? or are you trying to restart pods as well? Pods will be in default allow mode until the network policies are reconciled against them and this can take up to 2-3s based on the cluster load. Please check if this situation applies to you and if it does please try with Strict mode and let us know if it helps..

kervrosales · 2024-06-10T21:36:45Z

@achevuru Thanks for your reply!

That is correct; deleting and recreating the NetworkPolicy resolved the issue. I had to do this for each namespace that requires a NetworkPolicy. However, restarting the pods did not resolve the issue, which leads me to believe that the problem I am experiencing might be different from what was previously mentioned.

Can you please confirm that enabling "strict" mode on AWS CNI is achieved by updating the DaemonSet in the kube-system namespace? I have done this, but for some reason, it does not seem to work. None of my new pods get the default-deny policy by default.

I confirm that the container has the correct environment variables:

  containers:
    - env:
      - name: NETWORK_POLICY_ENFORCING_MODE
        value: strict
      - name: ADDITIONAL_ENI_TAGS
        value: '{}'
      - name: ANNOTATE_POD_IP
        value: "false"
      - name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
        value: "false"
      - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
        value: "true"
      - name: AWS_VPC_ENI_MTU

Thanks!

samox73 · 2024-06-20T12:47:28Z

This is expected, albeit I do not know why the AWS implementation

[...] The Amazon VPC CNI plugin for Kubernetes configures network policies for pods in parallel with the pod provisioning. Until all of the policies are configured for the new pod, containers in the new pod will start with a default allow policy. [...] (https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html#cni-network-policy-considerations)

deviates from the official K8s spec for network policies:

All newly created pods affected by a given NetworkPolicy will be isolated before they are started. Implementations of NetworkPolicy must ensure that filtering is effective throughout the Pod lifecycle, even from the very first instant that any container in that Pod is started. (https://kubernetes.io/docs/concepts/services-networking/network-policies/#pod-lifecycle)

This choice results in severe security implications, mainly that pods might not be isolated for the first few seconds they are started, which would give a malicious actor the opportunity to perform network actions in this short time frame.

@achevuru, imo this should be addressed immediately, as a breach in network allow rules during pod startup poses a high security risk, especially for workloads that cannot be scanned for security issues.

achevuru · 2024-06-20T16:25:47Z

@samox73 Above doc calls out a Strict mode option under the same section, that starts with either default deny or the current set of policies configured against the pod.

https://github.com/aws/amazon-vpc-cni-k8s/tree/master?tab=readme-ov-file#network_policy_enforcing_mode-v1171

Also, in the Standard mode - any network connections opened prior to Network Policy reconciliation (in the first 1-2 seconds) will be terminated post the enforcement.

samox73 · 2024-06-21T08:20:09Z

@achevuru This is not exactly the desired behavior either though. This blocks all traffic by default. Thus, one would have to write policies for all workloads that run in the cluster and need either ingress or egress connections, which can pose a time consuming task that takes multiple days for big clusters.

It would be very cool, if pods that are associated with a network policy have traffic blocked from the start and pods that match no network policy to have default allow.

achevuru · 2024-06-21T18:57:30Z

@samox73 So, I'll summarize the options available right now,

Standard Mode --> Pod creation and Network Policy reconciliation are parallel workflows with Network Policy reconciliation kicking in after the Pod is assigned an IP address. Reconciliation can't start until an IP is assigned to a Pod and the pod Interface is UP and running. Downside of this is that it can (potentially) leave the pod's interface in Default Allow mode for the first few seconds (usually 1-3 seconds) of the pod's lifecycle. However, as I called out above any unexpected (ongoing) connection will be blocked once NP comes in to effect.
Strict Mode --> Pod Creation is blocked until either a Default Deny (or) the current active set of policies are configured against the pod. Usually the initial pod of a deployment/daemonset etc., will come up with default deny and subsequent replicas of the same deployment will come up with current set of active firewall rules against the pods. Will ensure no unauthorized network access is allowed from any pod on the cluster until the reconciliation is complete against the newly launched pod. As you called out, this mode requires an user to have a Network Policy defined for all the pods in the cluster and we landed at this requirement mainly from the customer feedback we received. I understand your concern, that it takes some work to create an NP for every pod in the cluster but you can just create a simple allow all Network Policy per namespace instead and pods in a particular namespace with no specific Network policy will inherit this allow all policy. So, one per namespace would suffice. Downside of this approach is that some pods can (potentially) stay in default deny mode until NP reconciliation is complete against the newly launched pod (usually 1-3 seconds). We're considering providing an option to user, where they can exclude some namespaces from Strict mode enforcement.

Also, If you want to reduce the initial time period (1-3 seconds based on the load) during which a pod is either in default allow (or) default deny mode(based on the NP enforcement mode selected), you can use the ANNOTATE_POD_IP feature of VPC CNI. We introduced this feature precisely for this scenario but for Calico Network policy solution and we extended this to VPC CNI's Network Policy implementation as well. With this feature, it should reduce the initial 1-3s period down to less than a second (or even lower in most cases).

paolomainardi · 2024-07-25T09:39:26Z

ANNOTATE_POD_IP seems to fix the issue on my side too.

abatilo · 2024-08-26T01:05:58Z

Using ANNOTATE_POD_IP doesn't seem to bring the time down for me.

it should reduce the initial 1-3s period down to less than a second

It appears that setting this only drops the initial period down by maybe a second? And so it still takes a while for either the network policy to be enforced, or if I set the enforcement mode to strict, it still takes a grand total of 7-9 seconds for very small (~30Mb) containers to be addressable.

I have confirmed that the annotation includes the pod IP, etc.

Is there something that I can do to help speed up the pod start up even further?

jan-g · 2024-09-09T09:14:22Z

@achevuru the comment @samox73 made is being somewhat underestimated, I think.

With enforcing mode standard, there's a window during pod start where pods have allow-all access. This is a serious security issue - by the time the policy is enforced, I can already exfiltrate IMDS tokens (for example); it doesn't matter if connections are closed post hoc if the damage is already done.

With enforcing mode strict, there are a number of comon services - external-dns and autoscaler seem to be affected - which attempt to create a client once on pod start; that client creation can fail with errors such as

F0904 10:17:07.344111       1 aws_cloud_provider.go:455] Failed to create AWS Manager: MissingRegion: could not find region configuration

We really need an implementation that follows the k8s spec for network policies.

samox73 · 2024-10-15T08:31:12Z

Hi @achevuru @jayanthvn @jaydeokar,

I wanted to follow up on the deviation of the AWS VPC CNI from the Kubernetes specification. Could you clarify whether this is an intentional divergence or if there are plans to align it with the official spec in the future? If there are ongoing efforts in this direction, any insights on the timeline would be greatly appreciated. Maybe you could also guide a new contributor on where to start to fix this? I would be open to work on this.

Thanks!

jan-g · 2024-10-16T18:40:36Z

I have sone crow to eat here I think.

Looking closely at the netpol behaviour spec, I think what it says "pods are brought up in isolation" describes the behaviour of the vpc-cni in strict mode.

The non-normative text explicitly calls out that pods should be able to operate if their required networking appears after they have started.

We have in fact seen issues with coredns where that didn't happen, but I think that's just a bug.

I am led to conclude that the misbehaviour of things like external-dns (which tries on startup to create an aws client and fails because it can't retrieve a region, for instance) is actually a bug in that software; I think pods should retry client creation until it's successful in those circumstances (and report via readiness when they are running coreectly).

kervrosales added the bug Something isn't working label May 17, 2024

creinheimer mentioned this issue Jul 19, 2024

Network denies despite allow-all policy (strict mode) #288

Open

Monska85 mentioned this issue Aug 2, 2024

Network policy blocks established connections to RDS #236

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network Policy Not Enforced on Initial Creation #271

Network Policy Not Enforced on Initial Creation #271

kervrosales commented May 17, 2024

achevuru commented Jun 3, 2024

kervrosales commented Jun 10, 2024 •

edited

Loading

samox73 commented Jun 20, 2024

achevuru commented Jun 20, 2024 •

edited

Loading

samox73 commented Jun 21, 2024

achevuru commented Jun 21, 2024

paolomainardi commented Jul 25, 2024

abatilo commented Aug 26, 2024

jan-g commented Sep 9, 2024

samox73 commented Oct 15, 2024

jan-g commented Oct 16, 2024

Network Policy Not Enforced on Initial Creation #271

Network Policy Not Enforced on Initial Creation #271

Comments

kervrosales commented May 17, 2024

achevuru commented Jun 3, 2024

kervrosales commented Jun 10, 2024 • edited Loading

samox73 commented Jun 20, 2024

achevuru commented Jun 20, 2024 • edited Loading

samox73 commented Jun 21, 2024

achevuru commented Jun 21, 2024

paolomainardi commented Jul 25, 2024

abatilo commented Aug 26, 2024

jan-g commented Sep 9, 2024

samox73 commented Oct 15, 2024

jan-g commented Oct 16, 2024

kervrosales commented Jun 10, 2024 •

edited

Loading

achevuru commented Jun 20, 2024 •

edited

Loading