API Permissions needed for /api/v1/workflows/{namespace}/submit

[wesleyhaws]

I worked through this doc: https://argoproj.github.io/argo-workflows/access-token/

and got an access token. I'm able to:

curl -X GET https://{endpoint}/api/v1/workflow-templates/{namespace}/{template} -H "Authorization: Bearer $ARGO_TOKEN"

just fine. However, if I run:

curl -X POST \
  --url https://workflows-dev-main.shippodev.com/api/v1/workflows/argo-workflows/submit \
  --header 'Authorization: Bearer $ARGO_TOKEN' \
  --data '{ "resourceKind": "WorkflowTemplate", "namespace": "{namespace}", "resourceName": "{template}", "submitOptions": { "parameters": [ "{param1}={param1_value}", "{param2}={param2_value}" ], "submitOptions": { "labels": "workflows.argoproj.io/workflow-template={template}", "name": "{arbitrary name}" } } }'

I get the following error:

{"code":16,"message":"Unauthorized"}

for the life of me I cannot figure out what permissions I might need to do this. I can't seem to find any discussion or docs anywhere on the matter. What might I be doing wrong?

[wesleyhaws]

NOTE: I have tried the following permissions on the role that the service account i'm using is using:

      {
        api_groups = ["argoproj.io"]
        resources  = ["*"]
        verbs      = ["list", "get", "watch"]
      },
      {
        api_groups = ["argoproj.io"]
        resources  = ["workflows", "workflowtaskresults"]
        verbs      = ["create", "patch"]
      }

I even tried it with resources *, and it didn't do anything. So there must be something else somewhere that I'm missing.

[wesleyhaws]

How do I regenerate my token to have a valid one then, as it seems like this token is auto generated by something else?

[agilgur5]

It's just the ServiceAccount's Secret, per the docs.
Bearer is part of it the token, so when you used BearerToken that is correctly invalid. You didn't get that error when using Bearer <secret>, meaning that your initial header was correct.

[wesleyhaws]

I think I'm getting closer here:

curl -X 'POST' \
  'https://{endpoint}/api/v1/workflows/{namespace}/submit' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "resourceKind": "WorkflowTemplate",
    "namespace": "{namespace}",
    "resourceName": "scripts-testing",
    "submitOptions": {
        "entryPoint": "failure",
        "parameters": [
            "id=0",
            "name=sample"
        ],
        "labels": "workflows.argoproj.io/workflow-template=scripts-testing",
        "name": "scripts-testing-callback"
    }
}'

Will now spit out:

{"code":2,"message":"The POST operation against Workflow.argoproj.io could not be completed at this time, please try again."}

Just to make sure it's not a role issue, I made sure to try this out with verbs of * on workflows and workflowtaskresults. No matter how many times I try to issue the POST it says the same thing. The logs on the server say the same thing with no additional helpful info sadly.

Role permissions:

PolicyRule:
  Resources                                Non-Resource URLs  Resource Names  Verbs
  ---------                                -----------------  --------------  -----
  workflows.argoproj.io                    []                 []              [*]
  workflowtaskresults.argoproj.io          []                 []              [*]
  *.argoproj.io                            []                 []              [list watch get]
  *.dataflow.argopro.io                    []                 []              [list watch get]

[wesleyhaws]

Okay I found this helpful link: #7580

Sure enough when I changed the name it ran (at least a dryrun, nothing is displaying in the UI yet). I guess I need to scour the API paths to guess how I can submit the same workflow name with different parameters (or re-sumbit)

[agilgur5]

Ah different issue. generateName is commonly used to provide different names

[wesleyhaws]

So people don't have to read through the chain, because I know more people are going to run across this, and save yourself days worth of debugging. Here is how to get this to work.

1. Deploy the workflow template
2. Deploy your SA, Rolebinding, and secret
3. You will need these permissions (maybe not all of these but this is what I tried it with and it worked)

PolicyRule:
  Resources                                Non-Resource URLs  Resource Names  Verbs
  ---------                                -----------------  --------------  -----
  workflows.argoproj.io                    []                 []              [create patch]
  workflowtaskresults.argoproj.io          []                 []              [create patch]
  *.argoproj.io                            []                 []              [list watch get]
  *.dataflow.argopro.io                    []                 []              [list watch get]

4. You can get your api token by doing:

export ARGO_TOKEN=$(kg secret <secret_name> -o=jsonpath='{.data.token}' -n <namespace> | base64 --decode)

NOTE: Still unsure if this is a short lived token or not.

5. You MUST use Authorization: Bearer <token> in your curl headers, even though the API shows Authorization: <token> or even Authorization: BearerToken <token>.

6. Issue the submit curl:

curl -X 'POST' \
  'https://{endpoint}/api/v1/workflows/{namespace}/submit' \
  -H 'accept: application/json' \
  -H "Authorization: Bearer ${ARGO_TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
    "resourceKind": "WorkflowTemplate",
    "namespace": "{namespace}",
    "resourceName": "{workflow_template_name}",
    "submitOptions": {
        "dryrun": true,
        "entryPoint": "{template_name}",
        "parameters": [
            ...
        ],
        "labels": "workflows.argoproj.io/workflow-template={workflow_template_name}",
        "name": "{Whatever_you_want_Here}"
    }
}'

NOTE: The above entryPoint is NOT the spec.entrypoint, but the spec.templates[?].name.
NOTE2: You can use submitOptions.generateName to name the workflow yourself otherwise it inherits the name of the workflow-{random_string}

Some caveats. If the workflow has already been run with the provided name it will throw this error:

The POST operation against Workflow.argoproj.io could not be completed at this time, please try again.

To fix it, just change the name of your run. Alternatively you can just "rerun" the workflow by resubmitting it with the previous runs name:

curl -X 'PUT' \
  'https://{endpoint}/api/v1/workflows/{namespace}/{prev_run_name}/resubmit' \
  -H 'accept: application/json' \
  -H "Authorization: Bearer ${ARGO_TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
        "dryrun": true,
        "parameters": [
            ...
        ],
        "name": "{prev_run_name}"
}'

[agilgur5]

5. You MUST use Authorization: Bearer <token> in your curl headers, even though the API shows Authorization: <token> or even Authorization: BearerToken <token>.

In the docs, $ARGO_TOKEN includes Bearer . You are referring to the ServiceAccount Secret, which is the second half of $ARGO_TOKEN (the part after Bearer ).

Where does the API say say "BearerToken "? That should definitely be corrected. Feel free to submit a PR as well.

[agilgur5]

The main reference I could find for "BearerToken" was in the generated SDKs. Those are generated via OpenAPI tooling, so not directly written by Argo. I also did not find a reference that specifically said Authorization: BearerToken <token>.

[agilgur5]

Made some small clarifications to two places in the docs that were a bit inconsistent in #11714

[sahil-sharma]

We have a slack bot (say bot) that we plan to use to trigger some WorkflowTemplates. For this we decided to use HTTP API calls to Argo Workflows. We are facing a challenge in the implementation when it comes to token.
This is what we are doing. We are aware we need below stuff to make this work:

Role and RoleBinding
SA
Secret

We store our secrets in AWS Secret Manager and plan to store the token value there. Get it via External Secret and mount this value.
We created the Role and RoleBindings, SA as follows:
bot-sa-yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-bot
  namespace: argo-workflows
  annotations:
    "eks.amazonaws.com/role-arn": "arn:aws:iam::123456789:role/external-secrets-bot"
automountServiceAccountToken: true

bot-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: bot-role
rules:
- apiGroups:
    - argoproj.io
  resources:
    - workflows
    - workflowtemplates
    - cronworkflows
  verbs:
    - create
    - get
    - list
    - watch

bot-rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bot-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: bot-role
subjects:
  - kind: ServiceAccount
    name: external-secrets-bot
    namespace: argo-workflows

First question is: How we can pass (or generate) a custom JWT token (stored and managed by us in AWS SecretManager)? That further can be used by SA for HTTP API calls.
I did create a secret via plain YAML file:

apiVersion: v1
kind: Secret
metadata:
  name: bot
  namespace: argo-workflows
  annotations:
    kubernetes.io/service-account.name: external-secrets-bot
type: kubernetes.io/service-account-token

It does create the secret with name as bot and then we can get the token as:
export ARGO_WF_TOKEN="Bearer $(kubectl get secret bot -o=jsonpath='{.data.token}' | base64 --decode)"
Then we fire a cURL request and it works:
curl -I -XGET https://$ARGO_WF_URL/api/v1/workflows/argo -H "Authorization: $ARGO_WF_TOKEN"

Now if we try to create an ExternalSecret, SecretStore that would try to get the secret value from AWS it fails.
This is how we are doing it.
bot-secret-store.yaml

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: bot
  namespace: argo-workflows
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-bot

bot-es.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: bot-external-secret
  namespace: argo-workflows
spec:
  secretStoreRef:
    name: bot
    kind: SecretStore

  refreshInterval: "1h"

  target:
    name: bot
    creationPolicy: "Owner"
    deletionPolicy: "Delete"

    template:
      engineVersion: v2
      type: Opaque
      metadata:
        labels:
          app.kubernetes.io/part-of: argo-workflows
  data:
    - secretKey: token
      remoteRef:
        key: "arn:aws:secretsmanager:eu-west-1:123456789:secret:argo-workflows/bot-http-token-123ABC"

IAM roles and permissions are in-place.
Above would create a SecretStore and an ExternalSecret that would further create a K8s secret object (with bot name in argo-workflows) to fetch the token value from AWS SM object (argo-workflows/bot-http-token-123ABC) that has a value in plain-text.
As of now, we did a hack like create a secret via K8s YAML file and copy the value in AWS SM and use it. This way it works.
But if we have to pass a user-managed token then it does not work. I did try to understand the JWT token payload from the actual (aka working) token generated by K8s secret YAML file (https://jwt.io) and the header and the payload were as:

{
  "alg": "RS256",
  "kid": "61iSk-G-4NkR7smu7CN0bXK_VFQnVKr1CoTDBD5yCiM"
}
---
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "argo-workflows",
  "kubernetes.io/serviceaccount/secret.name": "bot",
  "kubernetes.io/serviceaccount/service-account.name": "external-secrets-bot",
  "kubernetes.io/serviceaccount/service-account.uid": "78bf5b20-e65a-4f09-9d34-e9c73c9e2fcf",
  "sub": "system:serviceaccount:argo-workflows:external-secrets-bot"
}

When we try to generate a custom JWT token with a python script we used Algo as HS256. Script is:

import jwt
import datetime

# Define your secret key
secret_key = "<our-secret-key>"

# Load your RSA private key
#with open('/jwt-key', 'r') as f:
#   private_key = f.read()

# Define the payload
payload = {
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "argo-workflows",
  "kubernetes.io/serviceaccount/secret.name": "bot",
  "kubernetes.io/serviceaccount/service-account.name": "external-secrets-bot",
  "kubernetes.io/serviceaccount/service-account.uid": "78bf5b20-e65a-4f09-9d34-e9c73c9e2fcf",
  "sub": "system:serviceaccount:argo-workflows:external-secrets-max"
}

# Generate the JWT token
token = jwt.encode(payload, secret_key, algorithm="HS256")

print(f"Generated JWT token: {token}")

As the SA has already been created I copied the UUID from it and passed it in my code. If we generate a JWT token this way it does not work. I am aware that we need to pass this secret_key to ARGO server so that it can validate the token sent in API request. How to do this I am not aware.
If my approach is incorrect then please educate me on how a custom token can be passed for HTTP API calls to Argo Workflow.
Thanks in advance.

[agilgur5]

There's a lot going on here and not in the original question that you should really make a separate discussion.
But, in short, spoofing the token would not be a correct approach and overcomplicates the problem. The token must be created by Kubernetes. You could then insert it into AWS Secrets Manager for other tools to use, but you shouldn't recreate it in the same cluster nor generate it yourself.

Note that if you have the cluster's private key, you might be able to generate it as such, but again, I would not recommend that as an anti-pattern.