Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add probes for cache statefulsets #9771

Closed
Sabian-A opened this issue Oct 30, 2024 · 5 comments
Closed

Add probes for cache statefulsets #9771

Sabian-A opened this issue Oct 30, 2024 · 5 comments
Labels
enhancement New feature or request helm

Comments

@Sabian-A
Copy link

Is your feature request related to a problem? Please describe.

I've encountered an operational challenge in monitoring and ensuring the health of the caches (result, chunks, index,metadata )StatefulSets efficiently. Currently, there's no straightforward method to integrate Kubernetes readiness and liveness probes directly through the Helm chart configurations. This makes it difficult to automatically manage the healthy state and readiness of our deployed services, impacting our ability to reliably scale and maintain our infrastructure.

Describe the solution you'd like

I would like the Helm chart for the cache StatefulSets to support configuration options that enable the easy integration of Kubernetes probes.

Describe alternatives you've considered

An alternative solution was manually modifying the deployment templates post-Helm generation to include these probes, but this approach is not maintainable or scalable as it bypasses the advantages of using Helm for deployment configurations. Using pre-hooks in Helm to modify deployments post-deployment was also considered but dismissed for the same reasons.
@56quarters 56quarters added enhancement New feature or request helm labels Oct 30, 2024
@56quarters
Copy link
Contributor

Which Kubernetes checks specifically do you want the caches to have and why?

In my experience they're not useful for Memcached:

  • Readiness check: Memcached starts in a fraction of a second.
  • Startup check: Again not useful because Memcached starts in a fraction of a second.
  • Liveness check: I don't see how this would be useful for Memcached and liveness checks in general seem like huge liability with the potential to make minor issues worse.

@Sabian-A
Copy link
Author

@56quarters Thank you for getting back to me! I specifically need the Readiness check because, in our GKE setup, the Pod Disruption Budget (PDB) relies on the readiness probe to determine if a pod is healthy and can handle traffic. Without the readiness check, the PDB has limited visibility into the pod’s true health status, only knowing that it’s up and running but not if it’s fully ready to serve requests.

@56quarters
Copy link
Contributor

@56quarters Thank you for getting back to me! I specifically need the Readiness check because, in our GKE setup, the Pod Disruption Budget (PDB) relies on the readiness probe to determine if a pod is healthy and can handle traffic. Without the readiness check, the PDB has limited visibility into the pod’s true health status, only knowing that it’s up and running but not if it’s fully ready to serve requests.

OK, that sounds reasonable if you'd like to open a PR To add a TCP readiness probe to Memcached instances in the helm chart. Thanks!

@Sabian-A
Copy link
Author

@56quarters #9990

@Sabian-A Sabian-A mentioned this issue Nov 22, 2024
Closed
4 tasks
@56quarters
Copy link
Contributor

Closing this issue for the reasons detailed in this PR, thanks. #9990 (review)

@56quarters 56quarters closed this as not planned Won't fix, can't repro, duplicate, stale Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request helm
Projects
None yet
Milestone
No milestone
2 participants
@56quarters @Sabian-A