Filters - Custom Filter Dialect/Custom Protocol

[zZHorizonZz]

We're working on implementing the subscription API, but as we are going trough implementing this on our own as there aren't any language-specific implementations yet, we're a bit hesitant on some things. We're thinking that if language-specific implementations do appear in the future, there might be a problem if someone wants to use their own dialect.

Here's our reasoning: if someone generates their own server based on the OpenAPI specs, it'll generate the models for filters. But if they want to use a custom dialect, like CEL (Common Expression Language) in our case, they'd have to create a custom version of the subscription API specs and potentially provide custom SDKs instead of relying on official SDKs which we can imagine will contain the client or at least client abstraction for communication with subscription API.

We had similar thoughts about sink protocols. There's no way to provide a custom protocol right now. For example, if we wanted to use gRPC (with its own HTTP/2 protocol), we'd have to propose it and wait for it to be added to the official specification.

While we understand that the API specs aim to unify things, we think that implementors/providers should be able to decide on and provide their own filters and protocols.

I think for optional dialects there could be something like:

{ 
  "custom": {
    "type": "sql (Or any other dialect)"
    "expression": "source LIKE '%cloudevents%'"
  }
}

[duglin]

@zZHorizonZz I'm not following. The spec has this:

  "filters": [ ?
    { "[dialect name]": [dialect specific object] } +
  ],

and the [dialect specific object] allows for anything. So while a core SDK may not know about a new/custom dialect, if it allowed for extension dialog-specific modules to be used, then that new module would know how to interpret the dialect language. But i don't think it would require an entirely new SDK per se.

Or am I missing something?

btw - the spec is still very much a work-in-progress.

[zZHorizonZz]

The specifications themselves are ok, in my opinion. I primarily think there's an issue with schemas that might arise from this. Currently, only OpenAPI is available, but I can imagine that a Proto schema could be added in the future. For OpenAPI, there are two options for creating models: generating them based on provided schemas or creating and fine-tuning them manually.

When using client SDK and server stubs based on OpenAPI specifications, models can be inherently defined (As they are in these specs). The Filter type is specified as an object which, through generator modification or manual model recreation, can be made inheritable (e.g., in Java, AllFilter extends Filter). While filters can be inherited, adding a custom expression requires providing the model for that custom Filter to both the client SDK and server (If you are not using runtime interpreted language where you can do whatever you want). This could be problematic if there's an official subscription-client-sdk in the future, as users/companies would need to maintain custom variants of these SDKs just for one custom string filter.

For advanced filters, this might be unavoidable. However, for simple custom expression filters, I think this would just annoy people. So we think that this could be solved by turning the SQL expression into an Expression Filter where you could use any desired expression. While there are specifications for Cloud Event SQL, some companies would prefer not to add another system to manage, instead opting to use CEL or OData filters. Obviously, it could be made more robust if you could specify the type of language (dialect) you want to use e.g SQL,CEL,OData,etc.

So possible solutions to this would be:

{ "expression": "source LIKE '%cloudevents%'" }

Or

{ "expression": { "dialect": "SQL", "value": "source LIKE '%cloudevents%'" } }