Job worker cannot register in self-managed environment with multi-tenancy enabled

[J0Men]

Describe the bug

To Reproduce

Create a spring boot application with spring-boot-starter-camunda. Deploy a process and try to register job workers. Use a camunda environment with multi-tenancy enabled.

We use following application.properties:

zeebe.client.enabled=true
zeebe.client.broker.gateway-address=camunda-zeebe-url
zeebe.client.security.plaintext=false
zeebe.client.security.cert-path=/some/path
zeebe.client.connection-mode=ADDRESS
zeebe.client.default-tenant-id=<default>
zeebe.client.id=zeebe
zeebe.client.secret=secret
zeebe.authorization.server.url=keycloak-url
zeebe.token.audience=zeebe-api
zeebe.client.broker.gatewayAddress=camunda-zeebe-url

common.enabled=true
common.client-id=zeebe
common.client-secret=secret
common.keycloak.url=keycloak-url
common.keycloak.realm=camunda-platform
common.keycloak.token-url=keycloak-url/auth/realms/camunda-platform/protocol/openid-connect/tokencamunda.operate.client.base-url=camunda-operate-url

camunda.operate.client.client-id=zeebe
camunda.operate.client.client-secret=secret
camunda.operate.client.enabled=true
camunda.operate.client.url=camunda-operate-url
camunda.operate.client.keycloak-token-url=keycloak-url/auth/realms/camunda-platform/protocol/openid-connect/token
camunda.operate.client.auth-url=keycloak-url

Deploying and starting the process works as expected and the first task in the bpmn was executed but in the following task we get a "No value present" error in camunda operate.

Expected behavior

The job worker can register and execute successfully.

Log/Stacktrace

Full Stacktrace

2024-01-09T09:05:50.040+01:00  WARN 1820 --- [ault-executor-3] io.camunda.zeebe.client.job.poller       : Failed to activate jobs for worker *worker* and job type *jobType*

io.grpc.StatusRuntimeException: UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
	at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.60.0.jar:1.60.0]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.60.0.jar:1.60.0]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574) ~[grpc-core-1.60.0.jar:1.60.0]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72) ~[grpc-core-1.60.0.jar:1.60.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742) ~[grpc-core-1.60.0.jar:1.60.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.60.0.jar:1.60.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.60.0.jar:1.60.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.60.0.jar:1.60.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Environment:

• OS: Windows/Kubernetes
• Deployment: Self Managed (8.3.3)
• Spring-Zeebe Version: 8.3.4.5
• SpringBoot Version: 3.1.4

I suspect that the error message comes from here

[stephanpelikan]

On setting common.zeebe.enabled=false and providing a ZeebeClient using this code

@Bean
public ZeebeClient zeebeClient() {
    OAuthCredentialsProvider cp =
              new OAuthCredentialsProviderBuilder()
                       .authorizationS
                       .audience("zeebe")
                       .clientId("zeebe")
                       .clientSecret("secret")
                       .build();
    return ZeebeClient.newClientBuilder()
             .gatewayAddress("my-gateway-address")
             .credentialsProvider(cp)
             .build();
}

the error message is gone. We also checked the JobWorkerBuilderImpl in the debugger which looks fine: The tenant set using this code

var worker = client
        .newWorker()
        .jobType("myAwesomeTask")
        .handler(taskHandler)
        .name(workerId)
        .tenantId("iris");
worker.open(); 

can be found:

So, the ZeebeClient built by spring-zeebe's autoconfiguration is somehow different and does not pass the given tenantId properly to the Zeebe-Cluster.