Adding labels to Pods in ScaledJob

[chucksspencer]

Hi there. I'm trying to enforce one-pod-per-node using a podAntiAffinity rule. I'd like the rule to work based on a label on the pod, however I'm unable to apply custom labels. I found this resolved issue: #1627 and attempted to upgrade my Keda installation, hoping it would fix my issue - but I'm still seeing no labels on my pods when the jobs spin up (and the antiAffinity rule isn't working as needed).

Simple/n00b question first - is the #1627 fix deployed? How can I be sure that my Keda installation has this fix? Here's the output for helm list -n keda:
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION keda keda 1 2021-03-24 14:47:53.354651 -0500 CDT deployed keda-2.2.0 2.2.0

Here is the relevant yaml:

metadata:
  name: my-scaled-job
  namespace: my-namespace
spec:
  pollingInterval: 30
  maxReplicaCount: 24
  jobTargetRef:
    parallelism: 1
    completions: 1
    activeDeadlineSeconds: 3600 # How long a pod can run before it gets forced to shut down
    backoffLimit: 6
    template:
      metadata:
        labels:
          myjobidentifier: job1
      spec:
        nodeSelector:
          "beta.kubernetes.io/os": linux
          agentpool: mynodepool # The name of the node pool this job's pods are allowed to run on
        affinity:
          # This anti-affinity rule makes sure a new pod won't start on a node where another one is already running
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    myjobidentifier: job1
        containers:
           ...

[tomkerkhove]

I would need to dig into it but it looks like the labels from the ScaledJob are added, but you cannot apply your own in the specification. Is that correct @TsuyoshiUshio?

[zroubalik]

Please check that you have the correct version of ScaledJob CRD installed on your cluster, it needs to contain this setting: https://github.com/kedacore/charts/blob/b3ad4daa4990c1c0f78d6cacb90af7809ca8fc73/keda/templates/03-crd-scaledjobs.keda.sh.yaml#L160

[chucksspencer]

I think this may be the issue. How can I update the CRD? I tried the uninstall directions in the documentation, including the delete commands, on one of our clusters and ended up in a quagmire of locked resources I couldn't delete that were preventing reinstall. Hoping to avoid that this time.

[chucksspencer]

Thanks @zroubalik - you were correct - when I upgraded my Keda version I didn't get those scripts updated. I did a fresh install of the newest version on a new cluster and it is working as hoped.

Can I suggest that a section on upgrading be added to the documentation along with deploying and uninstalling?

Thanks!

[chucksspencer]

Please check that you have the correct version of ScaledJob CRD installed on your cluster, it needs to contain this setting: https://github.com/kedacore/charts/blob/b3ad4daa4990c1c0f78d6cacb90af7809ca8fc73/keda/templates/03-crd-scaledjobs.keda.sh.yaml#L160

This sub-reply from @zroubalik was the solution. When updating Keda it is necessary to update the CRDs as well. Quoting so I can mark the correct answer.