Can't connect to API using fresh docker on GCE

[chreds]

Topic

api docker

Revelant information

VERSION=0.50.24

I've followed the instructions at https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine/ and can't get the API to work. I can connect to the UI, create connections, sources, destinations, so obviously the internal API is working, but I can't get any results out of the API.

I've tried with debian 10, 11 and ubuntu 22.04 LTS underlying dockers same scenario.

If I run a GET http://localhost:8006/v1/workspaces, it responds after ~20 seconds with a 500 error. I've tried curl from the airbyte-api-server docker console too and get the same problem. It looks like maybe connectivity error from the airbyte-api-server to the airbyte-server dockers?

Example with new API (doesn't work - returns 500):
curl -u airbyte:password --request GET http://localhost:8006/v1/workspaces --header 'accept: application/json' {"type":"about:blank","status":500}

Example with config API (works):
curl -u airbyte:password --request POST http://localhost:8000/api/v1/workspaces/list --header 'accept: application/json' {"workspaces":[{"workspaceId":"6446bfdb-f5f7-420c-9757-5ab5c02ed02d","customerId":"67759121-5171-43c8-806f-fee1c8c5956c","name":"6446bfdb-f5f7-420c-9757-5ab5c02ed02d","slug":"6446bfdb-f5f7-420c-9757-5ab5c02ed02d","initialSetupComplete":false,"displaySetupWizard":true,"notifications":[],"notificationSettings":{},"defaultGeography":"auto","organizationId":"00000000-0000-0000-0000-000000000000"}]}

In the logs I see:

airbyte-api-server                | 2023-09-01 18:14:41 ERROR i.m.r.i.RecoveryInterceptor(resolveFallback):244 - Type [io.airbyte.api.server.forwardingClient.ConfigApiClient$Intercepted] executed with error: Read Timeout
airbyte-api-server                | io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
airbyte-api-server                |     at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26) ~[micronaut-http-client-core-3.10.1.jar:3.10.1]
airbyte-api-server                |     at io.micronaut.http.client.netty.DefaultHttpClient.lambda$exchangeImpl$33(DefaultHttpClient.java:1097) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onError(ReactorSubscriber.java:64) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
airbyte-api-server                |     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
airbyte-api-server                |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
airbyte-api-server                |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
airbyte-api-server                |     at java.lang.Thread.run(Thread.java:1589) ~[?:?]
airbyte-api-server                | 2023-09-01 18:14:41 WARN i.a.a.TrackingClientSingleton(get):39 - Attempting to fetch an initialized track client. Initializing a default one.
airbyte-api-server                | 2023-09-01 18:14:41 INFO i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/workspaces, operation=GET, status_code=500, api_version=0.50.24}
airbyte-api-server                | 2023-09-01 18:14:41 ERROR i.m.h.s.RouteExecutor(logException):444 - Unexpected error occurred: Read Timeout
airbyte-api-server                | io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
airbyte-api-server                |     at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26) ~[micronaut-http-client-core-3.10.1.jar:3.10.1]
airbyte-api-server                |     at io.micronaut.http.client.netty.DefaultHttpClient.lambda$exchangeImpl$33(DefaultHttpClient.java:1097) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onError(ReactorSubscriber.java:64) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
airbyte-api-server                |     at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.5.5.jar:3.5.5]
airbyte-api-server                |     at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
airbyte-api-server                |     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
airbyte-api-server                |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
airbyte-api-server                |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
airbyte-api-server                |     at java.lang.Thread.run(Thread.java:1589) ~[?:?]
airbyte-proxy                     | 172.20.0.1 - airbyte [01/Sep/2023:18:14:42 +0000] "GET /v1/workspaces HTTP/1.1" 500 35 "-" "PostmanRuntime/7.32.3"
airbyte-api-server                | 2023-09-01 18:14:42 INFO i.m.h.s.n.h.a.e.AccessLog(log):125 - airbyte-proxy.airbyte_airbyte_internal - - [01/Sep/2023:18:14:16 +0000] "GET /v1/workspaces HTTP/1.0" 500 35

I'm following the instructions without any modifications to the docker-compose scripts. Only difference is I needed to run the first command as sudo (sudo bash run-ab-platform.sh)

The exact same docker scripts work fine when running on Docker Desktop on Windows. Maybe something to do with differences between networking on docker windows and docker linux?

Earlier up I see some errors in the airbyte-api-server logs, but I saw these on Windows too and it still worked there so not sure if it's relevant.

airbyte-api-server                | 2023-09-01 18:17:49,152 main ERROR Error processing element Log4j2Appender ([Route: null]): CLASS_NOT_FOUND
airbyte-api-server                | 2023-09-01 18:17:49,162 main ERROR Error processing element Log4j2Appender ([Route: null]): CLASS_NOT_FOUND
airbyte-api-server                | 2023-09-01 18:17:53,552 main ERROR A route must specify an appender reference or an appender definition
airbyte-api-server                | 2023-09-01 18:17:53,562 main ERROR Null object returned for Route in Routes.
airbyte-api-server                | 2023-09-01 18:17:53,625 main ERROR A route must specify an appender reference or an appender definition
airbyte-api-server                | 2023-09-01 18:17:53,632 main ERROR Null object returned for Route in Routes.
airbyte-api-server                | 2023-09-01 18:17:53,660 main ERROR A route must specify an appender reference or an appender definition
airbyte-api-server                | 2023-09-01 18:17:53,698 main ERROR Null object returned for Route in Routes.

Any ideas?

[franviera92]

I have this issue with the api-server :(

[chreds]

I've tried in 50.0.27 now with same issue except it takes 150s to timeout instead of the 20s.

I turned debug on and went through logs on the docker in windows (WSL2) version and the ubuntu docker. Both are running docker 5.0.24 underneath. The API (:8006/v1/workspaces) is working under windows, but not under ubuntu. I've tried debian 10 as well in line with the instructions for GCE/Docker on the site.

Here's what I've found so far.

On the working version I see all the API routes getting added. I don't see this on the linux non-working version.

	Line 11103: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/create -> WorkspaceApiController#createWorkspace (application/json)
	Line 11104: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/delete -> WorkspaceApiController#deleteWorkspace (application/json)
	Line 11105: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/get -> WorkspaceApiController#getWorkspace (application/json)
	Line 11106: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/get_by_slug -> WorkspaceApiController#getWorkspaceBySlug (application/json)
	Line 11107: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/list -> WorkspaceApiController#listWorkspaces (application/json)
	Line 11108: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/list_all_paginated -> WorkspaceApiController#listAllWorkspacesPaginated (application/json)
	Line 11109: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/list_paginated -> WorkspaceApiController#listWorkspacesPaginated (application/json)
	Line 11110: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/update -> WorkspaceApiController#updateWorkspace (application/json)
	Line 11111: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/tag_feedback_status_as_done -> WorkspaceApiController#updateWorkspaceFeedback (application/json)
	Line 11112: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/update_name -> WorkspaceApiController#updateWorkspaceName (application/json)
	Line 11113: airbyte-server                    | 2023-09-07 15:51:16 �[36mDEBUG�[m i.m.w.r.AnnotatedMethodRouteBuilder(lambda$new$1):112 - Created Route: POST /api/v1/workspaces/get_by_connection_id -> WorkspaceApiController#getWorkspaceByConnectionId (application/json)

[chreds]

Here are the logs after a curl -x GET on /v1/workspaces

The working version calls the list_paginated endpoint after the first response.

Windows (works):

airbyte-server                    | 2023-09-07 15:54:37 �[36mDEBUG�[m i.m.h.s.n.RoutingInBoundHandler(syncWriteAndFlushNettyResponse):1323 - Response 200 - POST /api/v1/workspaces/list
airbyte-server                    | 2023-09-07 15:54:37 �[32mINFO�[m i.m.h.s.n.h.a.e.AccessLog(log):125 - airbyte-api-server.airbyte_internal - - [07/Sep/2023:15:54:36 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 200 397
airbyte-server                    | 2023-09-07 15:54:37 �[36mDEBUG�[m i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=POST /api/v1/workspaces/list]
airbyte-server                    | 2023-09-07 15:54:38 �[36mDEBUG�[m i.a.c.s.s.AuthorizationServerHandler(updateHeaders):57 - Checking HTTP request '{"workspaceIds":["60d7998d-f946-444c-9a61-335d658691d6"],"includeDeleted":false,"pagination":{"pageSize":20,"rowOffset":0}}' for field 'authUserId'...
...
airbyte-server                    | 2023-09-07 15:54:38 �[36mDEBUG�[m i.m.h.s.n.RoutingInBoundHandler(channelRead0):296 - Request POST /api/v1/workspaces/list_paginated
airbyte-server                    | 2023-09-07 15:54:38 �[36mDEBUG�[m i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestReceivedEvent[source=POST /api/v1/workspaces/list_paginated]
airbyte-server                    | 2023-09-07 15:54:38 �[36mDEBUG�[m i.m.c.b.DefaultBeanIntrospector(findIntrospection):99 - No BeanIntrospection found for bean type: class io.airbyte.api.model.generated.ListResourcesForWorkspacesRequestBody
airbyte-server                    | 2023-09-07 15:54:38 �[36mDEBUG�[m i.m.h.s.n.RoutingInBoundHandler(syncWriteAndFlushNettyResponse):1323 - Response 200 - POST /api/v1/workspaces/list_paginated
airbyte-server                    | 2023-09-07 15:54:38 �[32mINFO�[m i.m.h.s.n.h.a.e.AccessLog(log):125 - airbyte-api-server.airbyte_internal - - [07/Sep/2023:15:54:38 +0000] "POST /api/v1/workspaces/list_paginated HTTP/1.1" 200 397
airbyte-server                    | 2023-09-07 15:54:38 �[36mDEBUG�[m i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=POST /api/v1/workspaces/list_paginated]
airbyte-api-server                | 2023-09-07 15:54:38 �[32mINFO�[m i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/workspaces, operation=GET, status_code=200}
airbyte-api-server                | 2023-09-07 15:54:38 �[32mINFO�[m i.m.h.s.n.h.a.e.AccessLog(log):125 - 172.18.0.1 - - [07/Sep/2023:15:54:35 +0000] "GET /v1/workspaces HTTP/1.1" 200 134

Linux (not working):

airbyte-server                    | 2023-09-07 15:42:47 �[36mDEBUG�[m i.m.h.s.n.RoutingInBoundHandler(channelRead0):296 - Request POST /api/v1/workspaces/list
airbyte-server                    | 2023-09-07 15:42:47 �[36mDEBUG�[m i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestReceivedEvent[source=POST /api/v1/workspaces/list]
airbyte-server                    | 2023-09-07 15:42:47 �[32mINFO�[m i.m.h.s.n.h.a.e.AccessLog(log):125 - airbyte-api-server.airbyte_internal - - [07/Sep/2023:15:42:47 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 200 469
airbyte-server                    | 2023-09-07 15:42:47 �[36mDEBUG�[m i.m.h.s.n.RoutingInBoundHandler(syncWriteAndFlushNettyResponse):1323 - Response 200 - POST /api/v1/workspaces/list
airbyte-server                    | 2023-09-07 15:42:47 �[36mDEBUG�[m i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=POST /api/v1/workspaces/list]
airbyte-api-server                | 2023-09-07 15:45:18 �[32mINFO�[m i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/workspaces, operation=GET, status_code=500}
airbyte-api-server                | 2023-09-07 15:45:18 �[32mINFO�[m i.m.h.s.n.h.a.e.AccessLog(log):125 - airbyte - - [07/Sep/2023:15:42:47 +0000] "GET /v1/workspaces HTTP/1.1" 500 35
airbyte-api-server                | 2023-09-07 15:45:22 �[32mINFO�[m i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/workspaces, operation=GET, status_code=500}
airbyte-api-server                | 2023-09-07 15:45:22 �[32mINFO�[m i.m.h.s.n.h.a.e.AccessLog(log):125 - airbyte - - [07/Sep/2023:15:42:51 +0000] "GET /v1/workspaces HTTP/1.1" 500 35

[chreds]

This is fixed now in latest release

[chreds]

For anyone following this, some configs need updating due to the last release. I've documented them here: #29506