docs(openapi): Proposal for better OpenAPI definitions

[jvallesm]

Because

• OpenAPI docs are poorly documented and hard to read.

This PR

• Hides all unclear endpoints by setting their visibility to INTERNAL.
  • Previous definition is kept temporarily under openapiv2/internal.
• Improves comments and adds options such as tags, responses, etc., to the vdp.pipeline.v1beta.PipelinePrivateService proto definition as a proposal for an OpenAPI document template.
• Adds github actions to generate and lint the OpenAPI definitions as part of the CI/CD.

Notes and questions

• ❓ From which point of view are we documenting these endpoints? From the gateway, public endpoints are prefixed by /vdp, /internal, /core, etc., so I doubt these definitions can be direclty transformed to cURL request (or that working code can be generated from them).
  • If we want different hosts (or base paths), we'll need one OpenAPI document per service. This can be easily done with buf generate --path {}, but it will change how we render the definitions. Something to keep in mind for the future, probably for beta this is enough.
• docs(openapi): improve OpenAPI documentation #147 was checked along the documentation process, though only some of its changes made it to this PR.
  • There's a script there modifying some path parameters (e.g. pipelinePermalink becomes pipelines/permalink). I think this approach makes the proto -> openAPI translation unclear(er), hiding behind some undocumented rules. Therefore, I only used what googleapis and gprc-gateway provide.
• I'm not the most knowledgeable about Instill's domain, so any feedback regarding documentation is welcome!
• Not an expert on GHA either, feel free to suggest improvements.

Old vs new documentation

Generated with redocly preview-docs:

Old 👺

New 👶

[linear]

INS-3014 Improve comments in `protobuf` repo

[donch1989]

The PR looks good to me.

❓ From which point of view are we documenting these endpoints? From the gateway, public endpoints are prefixed by /vdp, /internal, /core, etc., so I doubt these definitions can be direclty transformed to cURL request (or that working code can be generated from them).
If we want different hosts (or base paths), we'll need one OpenAPI document per service. This can be easily done with buf generate --path {}, but it will change how we render the definitions. Something to keep in mind for the future, probably for beta this is enough.

The API endpoints should be exposed from the gateway perspective. So in my understanding, there are two approaches

1. One OpenAPI document, but we need to find a way to fix the prefix problem
2. Three OpenAPI documents, we can easily generate the correct path.

I think three OpenAI document is fine for beta. Not sure what does @pinglin and @xiaofei-du think

BTW, we are planning to make our OpenAPI document hosted by https://readme.com/ .

[jvallesm]

The API endpoints should be exposed from the gateway perspective.

I do agree, but then all the Private services would need their own document as they aren't exposed through the gateway (if I understood the configuration correctly) and probably they have their own authentication.

I'll have a quick look at how to modify the public endpoints' base path. If there's no straightforward solution, I agree this is good enough for beta 🤝 .