Federation of (predefined) processes

[soxofaan]

The aggregator combines multiple back-ends, each of which support different sets of predefined processes (or processes with slightly differing process definitions), as exposed by their respective GET /processes listings.

The aggregator's /processes currently returns the intersection of the process sets, because this is the safest choice with lowest surprise factor.
An alternative is taking the union, which could cause problems that violate the openEO API: aggregator claims to support process X (from backend A), but execution (on backend B) fails with "unsupported process X" .

Note that even if the /processes just list the intersection of processes, a savvy user can still submit process graphs that use back-end specific processes if they know that the execution will happen on that back-end (e.g. because of which collections are used, or some other mechanism).

This ticket is for tracking questions and concerns about the strategy of process federation.

[m-mohr]

I also wonder whether the intersection of the processes at the back-end could get an issue when a user realizes that the processes for one of the use cases we advertize are not actually listed in the aggregator. That could be confusing. So wondering whether it would make sense to go the same route as with processes? (union + "supported by EODC/VITO" in the summary?)

Also, for example, the Web Editor highlights missing processes in red. So if you load up a use case with processes only available at VITO or EODC, you would get confused for sure. And you can't even build them from scratch...

But ultimately, I also don't know what the better approach would be. Just giving some thoughts.

[soxofaan]

Taking the union is indeed a more pragmatic strategy, like @jdries proposed to do for file formats in #1:

Backends will then throw an error if a certain format or parameter is not supported, which will point to further alignment needs.

[m-mohr]

And the aggregator can make it more obvious to users by simply appending a "only supported by X" thing to the summary or so. As long as it's transparent, I don't see a larger issue with the union (as long as processes parameters etc don't diverge too much, which we need to document).

[soxofaan]

ok makes sense

FYI: the problem of parameter differences is an "intersection problem", expanding to union does not change anything.
Related to that: I listed a couple of parameter differences of EODC with the official spec at Open-EO/openeo-eodc-driver#131

[m-mohr]

FYI: the problem of parameter differences is an "intersection problem", expanding to union does not change anything.

I don't buy that. There are still issues with the "union" if you have different implementations. See openEOPlatform/documentation#12 (comment) for examples of when it could be a problem. (Or maybe I misunderstood what you meant to say here.)

[soxofaan]

I was just referring to union on process_id level. For example: if backend 1 has processes A and B, and backend 2 has A, and C, you only risk parameter differences in process A (the intersection), the extra processes from the union (B and C) only exist on one backend, so there is no problem with differences

[m-mohr]

The more I'm working with Platform, the more the intersection of the processes is stopping me from succeeding with the Platform as the clients tell me that x or y is not available (although it is available at one back-end), we also already got a RID based on something is not shown on https://docs.openeo.cloud/processes.
I think we should strongly think about using the union here.

[jdries]

Yes, I'm starting to feel the same. Even if we get all backends to support a core set of processes, it's still annoying for users that they have to switch.
Would it be an option that the aggregator can use an additional field to indicate which backends support a given process? Than maybe the web editor can apply some intuitive color coding, or small indication to help the user get a quick overview?

[soxofaan]

aggregator now returns union (initial implementation)

still to do / to discuss:

• how to handle differences in parameters/return (now VITO version has higher priority)
• annotate processes on which back-ends they are supported

[JeroenVerstraelen]

I have listed all discrepancies in process definitions among openeo backends here:
https://github.com/openEOPlatform/architecture-docs/issues/265

[soxofaan]

I have listed all discrepancies in process definitions among openeo backends here: https://github.com/openEOPlatform/architecture-docs/issues/265

next iteration of this report (done around Nov 10, 2022): https://github.com/openEOPlatform/architecture-docs/issues/282

[soxofaan]

TODO: create new iteration of this report and fix low hanging fruit

[jdries]

Federation works, script to report discrepancies is available.
Upcoming compliancy test suite can/should also check process metadata.