Pods shall define their own thresholds

[JHK]

Our workloads occasionally encounter an edge case where specific input data causes individual Pods to gradually increase their memory usage. These Pods typically take some time to reach the globally configured memory threshold for the cluster. However, during this time, multiple other Pods can enter a similar state, compounding the issue.

Since this pattern is predictable, we want to evict such Pods early to mitigate the impact. A useful feature would be the ability for these Pods to define their own memory threshold.

One implementation idea is an annotation, such as soft-pod-memory-evicter/eviction-threshold, where the value is specified as a percentage.

[maxlaverse]

Sounds good with such an annotation.

[maxlaverse]

If you're not in a hurry, I can probably implement this in the next weeks.