Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark ephemeral pods created as part of kanister functions with JobID label #2778

Merged
merged 22 commits into from
Apr 5, 2024

Conversation

mabhi
Copy link
Contributor

@mabhi mabhi commented Mar 21, 2024

Change Overview

Due to an ongoing work regarding identifying ephemeral pods those get spawned during various operations as part of some of the kanister functions, we need to mark these pod with some labels. One such pre-decided label for these pods is kanister.io/JobID. This label will help:

  • Developers to verfiy whether required pod-specs were applied to these pods.
  • Identify pods, due to erroroneous cases stay lingering in memory without ever completing.
  • If multiple such type of pods are instantiated and we want to target / debug one out of these.

Right now the flow of this label information is, with the instantiation of actionset against a blueprint that services various kanister functions is triggered from an external application, and let's say if this label populated with a JobID, then this label [key: value] pair is added in entirerity to the temporary pod, so we can track it.

This PR handle cases when the kube-task labelled pods invoke kanister function that has a context.Context instance passed that contains a key-value pair corresponding to the pre-decided label, then it will make its way to the pods label selector map in addition to others if present already.

Pull request type

Please check the type of change your PR introduces:

  • 🚧 Work in Progress
  • 🌈 Refactoring (no functional changes, no api changes)
  • 🐹 Trivial/Minor
  • 🐛 Bugfix
  • 🌻 Feature
  • 🗺️ Documentation
  • 🤖 Test

Issues

  • fixes #issue-number

Test Plan

  • 💪 Manual
  • ⚡ Unit test
  • 💚 E2E
  1. Create Deployment -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: time-logger
  namespace: test
  labels:
    app: time-logger-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: time-logger
  template:
    metadata:
      labels:
        app: time-logger
    spec:
      containers:
      - name: test-container
        image: containerlabs/aws-sdk
        command: ["sh", "-c"]
        args: ["while true; do for x in $(seq 1200); do date >> /var/log/time.log; sleep 1; done; truncate /var/log/time.log --size 0; done"]
EOF
  1. Create Blueprint -
cat <<EOF | kubectl create -f -
apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: time-log-bp
  namespace: kanister
actions:
  backup:
    phases:
    - func: KubeTask
      name: backupToS3
      args:
        namespace: "{{ .Deployment.Namespace }}"
        image: busybox
        podOverride:
          containers:
          - name: container
            imagePullPolicy: IfNotPresent
        command: ["sh", "-c", "counter=1; while [ $counter -lt 20 ]; do echo $counter; ((counter++)); sleep 1; done"]
EOF
  1. Create Actionset -
cat <<EOF | kubectl create -f -
apiVersion: cr.kanister.io/v1alpha1
kind: ActionSet
metadata:
  generateName: s3backup-
  namespace: kanister
  labels:
    kanister.io/JobID: hello-world

spec:
  actions:
  - name: backup
    blueprint: time-log-bp
    object:
      kind: Deployment
      name: time-logger
      namespace: test
EOF
  1. Watch ephemeral pod and check its label -
NAME                     READY   STATUS    RESTARTS   AGE   LABELS
kanister-job-j298f   1/1     Running   0          0s    createdBy=kanister,kanister.io/JobID=hello-world

pkg/function/kube_task.go Outdated Show resolved Hide resolved
pkg/function/kube_task.go Outdated Show resolved Hide resolved
@viveksinghggits
Copy link
Contributor

@mabhi are we just going to handle KubeTask in this PR?

@mabhi mabhi marked this pull request as draft March 22, 2024 13:37
@mabhi mabhi changed the title Mark ephemeral pods created as part of KubeTask with JobID label Mark ephemeral pods created as part of kanister functions with JobID label Mar 22, 2024
@mabhi
Copy link
Contributor Author

mabhi commented Mar 25, 2024

@mabhi are we just going to handle KubeTask in this PR?

Yes. Accounted additional pods as suggested in a separate PR.

@mabhi mabhi marked this pull request as ready for review March 26, 2024 12:11
@mabhi mabhi force-pushed the mark-ephemeral-kanister-job-pod branch from 0f7a1da to 9037165 Compare March 28, 2024 03:47
mabhi added 8 commits March 28, 2024 09:19
Signed-off-by: Abhijit Mukherjee <[email protected]>
Signed-off-by: Abhijit Mukherjee <[email protected]>
Signed-off-by: Abhijit Mukherjee <[email protected]>
Signed-off-by: Abhijit Mukherjee <[email protected]>
@mabhi mabhi force-pushed the mark-ephemeral-kanister-job-pod branch from 9037165 to 65f92ea Compare March 28, 2024 03:49
pkg/kube/utils.go Outdated Show resolved Hide resolved
pkg/kube/utils.go Outdated Show resolved Hide resolved
@mabhi mabhi requested a review from PrasadG193 April 2, 2024 07:15
pkg/function/kube_task.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
mabhi added 2 commits April 2, 2024 18:05
Signed-off-by: Abhijit Mukherjee <[email protected]>
Signed-off-by: Abhijit Mukherjee <[email protected]>
@mabhi mabhi requested a review from PrasadG193 April 2, 2024 12:59
@mabhi mabhi requested a review from PrasadG193 April 3, 2024 12:33
pkg/kube/utils.go Outdated Show resolved Hide resolved
Signed-off-by: Abhijit Mukherjee <[email protected]>
@mabhi mabhi requested a review from PrasadG193 April 3, 2024 14:36
pkg/kube/utils.go Outdated Show resolved Hide resolved
pkg/kube/utils.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
Signed-off-by: Abhijit Mukherjee <[email protected]>
@mabhi mabhi requested review from PrasadG193 April 4, 2024 07:09
Copy link
Contributor

@PrasadG193 PrasadG193 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, minor suggestions about formatting and optimization in the tests.

pkg/function/kube_task.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@PrasadG193 PrasadG193 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added suggestions about additional validation we can add in the test.

pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
pkg/kube/pod_runner_test.go Outdated Show resolved Hide resolved
@mabhi mabhi added the kueue label Apr 5, 2024
@mergify mergify bot merged commit c214c35 into master Apr 5, 2024
15 checks passed
@mergify mergify bot deleted the mark-ephemeral-kanister-job-pod branch April 5, 2024 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants