Create NPEP-127 for external ingress endpoints #128
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
✅ Deploy Preview for kubernetes-sigs-network-policy-api ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
k8s-ci-robot
added
the
size/M
There was a problem hiding this comment.
I think we still have some thinking to do on this one. It seems like we're going after a pretty broad set of use cases, some of which may not be as important.
The default-deny user story resonates with me, however for the others (concerning pods as the "selected entities" not Nodes) I generally feel like having a pod exposed to the "outside world" is a rare case. i.e even with all the mentioned ingress APIs there's usually a cluster -entity which routes/load balances or directs external traffic do destination pods, which we can already block traffic from today.
Lastly we need to think carefully about having a NPEP which references multi-cluster scenarios...as I think the scope could quickly explode.
npep/npep-127.md
Outdated
| - zero trust policy: As a cluster administrator I want to set up default deny all ingress policy as a BANP. | ||
|
|
There was a problem hiding this comment.
| - zero trust policy: As a cluster administrator I want to set up default deny all ingress policy as a BANP. | |
| - zero trust policy: As a cluster administrator I want to set up default deny all ingress guardrails with a BANP. | |
There was a problem hiding this comment.
I am trying to distinguish ANP vs BANP policies with the word "default", because BANP is just the default overridable policy, and I am not sure is guardrails word makes this semantics any more obvious, wdyt?
There was a problem hiding this comment.
yeah, I don't think we actually ever used the word "guardrails" in a consistent way
npep/npep-127.md
Outdated
| get to the cluster workloads. This should apply not only to the connections coming from the other cluster workloads, but to | ||
| all incoming connections. | ||
|
|
||
| - block well-known ports: As a cluster administrator I want to block all ingress traffic on specific ports. |
There was a problem hiding this comment.
Will/should this apply to well-known ports on Pods AND nodes?
There was a problem hiding this comment.
If you are talking about subjects, then only pods, since (B)ANP only applies to non-host network pods (what we call cluster workload)?
from the peer side, this should apply to any ingress traffic, from pods, nodes, external endpoints, and anything else that can reach pods
npep/npep-127.md
Outdated
| compromised, it will not be able to affect other worker nodes. To do so, I want to create a cluster-wide policy to | ||
| deny ingress connections from some of the cluster nodes. I may also need to explicitly allow access from the control plane nodes. | ||
|
|
||
| - inter-cluster communication: As a cluster manager for multiple clusters I want to make sure ingress connections |
There was a problem hiding this comment.
Hrm so on this one I think it's not really important to isolate this user story to "ingress connections are only allowed from a pre-defined set of related clusters". More specifically, if we're planning on identifying foreign clusters by networkCIDR there's no way to distinguish between a "foreign cluster" and just a plain old "external client" right?
There was a problem hiding this comment.
yeah right, I think inter-cluster part should go to the example
npep/npep-127.md
Outdated
| have limited access to the management cluster workloads by selecting management cluster workloads I want to protect | ||
| and a set of allowed subnets for the corresponding hosted clusters. | ||
|
|
||
| - external services: As a cluster manager I want to make sure ingress connections from a pre-defined set of external endpoints |
There was a problem hiding this comment.
This is broad "pre-defined set of external endpoints" what's defined as an external endpoint? can it be ANY IP?
Also this is the first time I see "can't be denied by the namespace owners" can you add a note somewhere that all the other user stories are explicitly non-overridable by namespace owners?
There was a problem hiding this comment.
not sure what is the best place to clarify this, but I refer to external endpoints in this NPEP as anything outside the cluster. So I would say any IP, unless you see any problem with that?
|
|
||
| * Implement ingress traffic control from external destinations (outside the cluster) | ||
|
|
||
| ## Non-Goals |
There was a problem hiding this comment.
Determining the non-goals here is going to be really important... I think we need to narrow this NPEP down
There was a problem hiding this comment.
that would be nice, but I just don't have any ideas on what to put here for now. I hoped that something will be outlined in the discussion here, if you have any ideas for non-goals, please lmk :)
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: npinaeva The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
npep/npep-127.md
Outdated
| get to the cluster workloads. This should apply not only to the connections coming from the other cluster workloads, but to | ||
| all incoming connections. | ||
|
|
||
| - block well-known ports: As a cluster administrator I want to block all ingress traffic on specific ports. |
There was a problem hiding this comment.
If you are talking about subjects, then only pods, since (B)ANP only applies to non-host network pods (what we call cluster workload)?
from the peer side, this should apply to any ingress traffic, from pods, nodes, external endpoints, and anything else that can reach pods
npep/npep-127.md
Outdated
| compromised, it will not be able to affect other worker nodes. To do so, I want to create a cluster-wide policy to | ||
| deny ingress connections from some of the cluster nodes. I may also need to explicitly allow access from the control plane nodes. | ||
|
|
||
| - inter-cluster communication: As a cluster manager for multiple clusters I want to make sure ingress connections |
There was a problem hiding this comment.
yeah right, I think inter-cluster part should go to the example
npep/npep-127.md
Outdated
| have limited access to the management cluster workloads by selecting management cluster workloads I want to protect | ||
| and a set of allowed subnets for the corresponding hosted clusters. | ||
|
|
||
| - external services: As a cluster manager I want to make sure ingress connections from a pre-defined set of external endpoints |
There was a problem hiding this comment.
not sure what is the best place to clarify this, but I refer to external endpoints in this NPEP as anything outside the cluster. So I would say any IP, unless you see any problem with that?
npep/npep-127.md
Outdated
| - compromised node protection: As a cluster administrator I want to make sure a compromised node can't send traffic to the | ||
| cluster workloads scheduled on the other nodes. | ||
|
|
||
| Example: my cluster has a set of nodes with very sensitive workloads, I want to make sure that if a worker node A is | ||
| compromised, it will not be able to affect other worker nodes. To do so, I want to create a cluster-wide policy to | ||
| deny ingress connections from some of the cluster nodes. I may also need to explicitly allow access from the control plane nodes. |
There was a problem hiding this comment.
Hm... so are there pods on the compromised node? Can't the attacker forge connections from pod IPs rather than the node IP? Or is it assumed that that will be blocked as well?
There was a problem hiding this comment.
this user story considers that all pods from the same namespace are on the same node, and that inter-namespace communication is also denied. I omitted it since we already have the means to implement this part with namespace selectors (and potentially tenancy is you label namespace with nodes), but maybe it makes sense to add these details
| it is applied to the cluster workloads, and should provide the required level of security regardless of how/if a cluster workload | ||
| is exposed to the outer world. | ||
|
|
||
| ### Use cases |
There was a problem hiding this comment.
So all of these are pretty vague about what kind of cluster ingress they're talking about, but there are a lot of different kinds, and some are harder to deal with than others, so I think it's important that our user stories explain what sorts of ingress activity it is that the admin cares about. (Even if it turns out that we declare some of these ingress types to be Non-Goals, it's good to be able to see clearly what sort of user stories we decided we can't solve right now.)
eg:
- direct delivery to hostNetwork pod
- pod HostPort
- direct delivery to pod-network pod IP (for network plugins that support that)
- service NodePort (with Cluster or Local externalTrafficPolicy)
- service ExternalIP (with Cluster or Local externalTrafficPolicy)
- service LoadBalancer IP (with Cluster or Local externalTrafficPolicy)
- direct delivery to service ClusterIP (for network plugins that support that)
- Ingress/Gateway (which generally resolves to one of the above, but exactly which one is implementation-dependent).
There was a problem hiding this comment.
I considered the way traffic reaches the pod to be not important in this context (meaning it should work in all situations), but now I think it may be useful to differentiate, e.g. in case some plugins can't implement CIDR-based filtering for some options. Or how do you think we should use this list in this context?
There was a problem hiding this comment.
IMHO let's just start really really simple. Just keep simple specific use cases that are not implementation specific, for me I want to be able to create policies for ingress (southbound traffic) for when pods are accessed using external traffic policy = local load balancer by an external client. Here the srcIP is that of the clientIP and I want to say whether this connection should be served by my pod
BUT
as soon as I re-read what I wrote above ^ :P that defies my definition of "service" because ofc that pod is supposed to serve all clients right? :) because its exposed as a service to the outside world?
AGAIN
I might wanna only allow a subset of clients to talk to this service so adding policies in a way to protect my backend pods..
There was a problem hiding this comment.
direct delivery to hostNetwork pod
pod HostPort
^ these seem very rare on cloud env's as use cases, can't think of any BUT on BMs can be useful but then again mostly the bastion or jump host or whatever you put on the same subnet and put accessible routes to is an entity you are consciously choosing to allow.. why would we want to block that...
Even if it turns out that we declare some of these ingress types to be Non-Goals, it's good to be able to see clearly what sort of user stories we decided we can't solve right now.)
right ^ good idea so that when we get asked why the current cidr and node peers are only egress.. we can just look at the non-goals section of this NPEP.
There was a problem hiding this comment.
I considered the way traffic reaches the pod to be not important in this context (meaning it should work in all situations)
I mean, ideally, yes, certainly. But in practice that is impossible, because the network policy implementation is normally completely separate from cloud LBs and Gateway mechanisms, and has no power over them or even any insight into how they work. Cluster ingress, as currently defined, is a very leaky abstraction.
We talked about this a little at KubeCon Paris and Shane mentioned https://github.com/kubernetes-sigs/gateway-api/blob/main/geps/gep-735/index.md which is a (Declined) GEP about doing policy-ish stuff in Gateway Routes. It is likely that for some user stories, providing a non-ANP policy API inside Gateway API is likely to work better than trying to implement ANP policy against arbitrary Gateway mechanisms. There was talk about doing a "GEPNPEP"...
(This is somewhat related to the problem of pod-to-cloud-LB-to-whatever traffic discussed in #203, and one thing that could potentially help to solve both would be to come up with a "bigger" model of "Kubernetes Networking" that makes the cloud network and gateways more legible to the network policy implementation. In theory, this could be part of KNI, though it is not something currently being discussed.)
Signed-off-by: Nadia Pinaeva <[email protected]>
|
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
/remove-lifecycle rotten |
k8s-ci-robot
removed
the
lifecycle/rotten
|
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
|
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/reopen |
|
@npinaeva: Failed to re-open PR: state cannot be changed. The npep-ingress branch was force-pushed or recreated. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
An attempt to collect use cases for external ingress endpoints.
Reviews and more examples are welcome.
Tracking issue #127