Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

405 Method Not Allowed during a new source creation #43

Closed
PavelSusloparov opened this issue Oct 20, 2023 · 5 comments
Closed

405 Method Not Allowed during a new source creation #43

PavelSusloparov opened this issue Oct 20, 2023 · 5 comments

Comments

@PavelSusloparov
Copy link

PavelSusloparov commented Oct 20, 2023 •
edited
Loading

Provider version: 0.3.4
Airbyte OS version: 0.50.31

I'm running into the following exception when attempting to create a new source connection using the Terraform provider.

airbyte_source_faker.my_source_faker: Creating...
╷
│ Error: unexpected response from API. Got an unexpected response code 405
│
│   with airbyte_source_faker.my_source_faker,
│   on source_snowflake.tf line 22, in resource "airbyte_source_faker" "my_source_faker":
│   22: resource "airbyte_source_faker" "my_source_faker" {
│
│ **Request**:
│ POST /sources HTTP/1.1
│ Host: <redacted>
│ Accept: application/json
│ Authorization: Basic YWlyYnl0ZTpwYXNzd29yZA==
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/terraform 0.3.4 2.108.3 1.0.0
│
│ {"configuration":{"always_updated":false,"count":3,"parallelism":9,"records_per_slice":5,"seed":6,"sourceType":"faker"},"name":"Faker Source","workspaceId":"<redacted>"}
│
│ **Response**:
│ HTTP/2.0 405 Method Not Allowed
│ Connection: close
│ Content-Type: text/html
│ Date: Fri, 20 Oct 2023 19:26:20 GMT
│ Server: nginx/1.25.2
│
│ <html>
│ <head><title>405 Not Allowed</title><script data-script="config-overwrites">
│                 window.TRACKING_STRATEGY = "segment";
│                 </script></head>
│ <body>
│ <center><h1>405 Not Allowed</h1></center>
│ <hr><center>nginx/1.25.2</center>
│ </body>
│ </html>

This is the resource I'm using to test the provider.

resource "airbyte_source_faker" "my_source_faker" {
  configuration = {
    always_updated    = false
    count             = 3
    parallelism       = 9
    records_per_slice = 5
    seed              = 6
    source_type       = "faker"
  }
  name         = "Faker Source"
  workspace_id = var.airbyte.connectors.workspace_id
}

Provider configuration.

provider "airbyte" {
  server_url = var.airbyte.connectors.server_url

  password = "password"
  username = "airbyte"
}

There is no user I can create in the Airbyte OSS, so I set username/password to default.

When I open developer tools and analyze the network traffic, I create a new source endpoint, which is POST /sources/create and not POST /sources.
@PavelSusloparov PavelSusloparov changed the title 405 Method Not Allowed 405 Method Not Allowed during a new source creation Oct 20, 2023
@erisnar
Copy link

erisnar commented Nov 3, 2023

@PavelSusloparov we are also experiencing this issue. Were you able to find a solution?

@erisnar
Copy link

erisnar commented Nov 4, 2023

See comment #8 (comment). Make sure the api-server pod is running.

❯ k get services -n airbyte
NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
airbyte-airbyte-api-server-svc                 ClusterIP   10.96.217.49    <none>        80/TCP         46m
airbyte-airbyte-connector-builder-server-svc   NodePort    10.96.227.230   <none>        80:30337/TCP   46m
airbyte-airbyte-server-svc                     ClusterIP   10.96.50.73     <none>        8001/TCP       46m
airbyte-airbyte-webapp-svc                     ClusterIP   10.96.164.223   <none>        80/TCP         46m
airbyte-db-svc                                 ClusterIP   10.96.160.67    <none>        5432/TCP       47m
airbyte-minio-svc                              ClusterIP   10.96.83.12     <none>        9000/TCP       47m
airbyte-temporal                               ClusterIP   10.96.64.180    <none>        7233/TCP       46m

Port forward to reach the api-server endpoint k port-forward -n airbyte services/airbyte-airbyte-api-server-svc 8006:80.

provider "airbyte" {
  server_url = "http://localhost:8006/v1"
}

@PavelSusloparov
Copy link
Author

PavelSusloparov commented Nov 27, 2023 •
edited
Loading

Thank you for the follow up @erisnar !

I confirmed that api-server is running

> kubectl get services -n airbyte
NAME                                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
airbyte-airbyte-api-server-svc                 ClusterIP   172.20.189.64    <none>        80/TCP         37d
airbyte-airbyte-connector-builder-server-svc   NodePort    172.20.185.17    <none>        80:30831/TCP   167d
airbyte-airbyte-server-svc                     ClusterIP   172.20.246.216   <none>        8001/TCP       167d
airbyte-airbyte-webapp-svc                     NodePort    172.20.201.223   <none>        80:30592/TCP   167d
airbyte-db-svc                                 ClusterIP   172.20.169.106   <none>        5432/TCP       37d
airbyte-minio-svc                              ClusterIP   172.20.205.168   <none>        9000/TCP       37d
airbyte-temporal                               ClusterIP   172.20.73.232    <none>        7233/TCP       167d

I updated the terraform provider config and ran

kubectl port-forward -n airbyte services/airbyte-airbyte-api-server-svc 8006:80
Forwarding from 127.0.0.1:8006 -> 8006
Forwarding from [::1]:8006 -> 8006

Terraform provider can reach the server now. However the API call fails with 500.

The api-server logs.

│ 2023-11-27 14:36:28 INFO i.a.a.s.n.LoggingNettyChannelHandler(channelRead):32 - [POST] /v1/sources                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     │
│ 2023-11-27 14:36:28 ERROR i.m.r.i.RecoveryInterceptor(resolveFallback):240 - Type [io.airbyte.api.server.forwardingClient.ConfigApiClient$Intercepted] attempting to resolve fallback for unavailable service [airbyte-airbyte-server-svc:8001]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        │
│ 2023-11-27 14:36:28 INFO i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/sources, operation=POST, status_code=500}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         │
│ 2023-11-27 14:36:28 ERROR i.m.h.s.RouteExecutor(logException):444 - Unexpected error occurred: No available services for ID: airbyte-airbyte-server-svc:8001                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           │
│ io.micronaut.discovery.exceptions.NoAvailableServiceException: No available services for ID: airbyte-airbyte-server-svc:8001                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           │
│     at io.micronaut.http.client.loadbalance.AbstractRoundRobinLoadBalancer.getNextAvailable(AbstractRoundRobinLoadBalancer.java:50) ~[micronaut-http-client-core-3.10.1.jar:3.10.1]

api-server can not recognize the server, however, you can see that the server is up and running.

@PavelSusloparov
Copy link
Author

Solved the connectivity problem between api-server and server by adding the config env var to the configmap

INTERNAL_API_HOST=http://airbyte-airbyte-server-svc:8001

based on this post - airbytehq/airbyte#29506 (comment)

@hongbo-miao hongbo-miao mentioned this issue Jun 19, 2024
Closed
@hongbo-miao
Copy link

hongbo-miao commented Jul 2, 2024 •
edited
Loading

Thanks @erisnar @PavelSusloparov !

For future people who come here, I want to point out today is actually just simply switching Terraform provider to server_url = "${var.airbyte_url}/api/public/v1". Here airbyte_url points to airbyte-webapp-svc.airbyte-namespace.

It helps

  • Terraform works well with new endpoint.
  • This new endpoint is latest. airbyte-api-server is actually out-of-date today. The service underneath this new endpoint fixed bug like this one.
  • If you use Helm chart to deploy, you can disable airbyte-api-server which is one less server! 
    airbyte-api-server:
  enabled: false
  • Ingress only needs to expose one endpoint now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hongbo-miao @PavelSusloparov @erisnar