Aggregator Evolution (Small Activity Proposal) Planning

[soxofaan]

This is central ticket for the planning of the "Aggregator Evolution" Small Activity Proposal.

Starting from the proposal doc, I extracted tasks, action points and ideas, and grouped them thematically a bit.
We also had a brainstorm session to further discuss them.

• Large area processing: iterate on current POC:

  • current POC was "Large area processing", but the system should not be limited to that use case (e.g. cross-back-end jobs, or progressive jobs with partial results)
  • more advanced splitting approaches
    • current POC now just provides static tile based spatial splitting (simple: no dependencies)
    • feasibility study (see further): Collection/process availability based splitting (harder: adds dependencies and load_result interactions) Feasibility study: cross-back-end process graph handling #53
    • Aggregate_spatial with large vector cube: split based on proximity/geospatial index/.... instead of based on tiles Partitioned jobs with aggregate_spatial and large "Vector cube" #54
    • split based on user-provided vector file (with extra fields for priority, title, ....) Partitioned jobs based on user provided tiling #55
    • filter_spatial + sample_by_feature based implementation Partitioned jobs based on user provided tiling #55
  • authentication challenges Auth management for long running jobs #29
  • job result:
    • metadata of final result: properly merge metadata of subjobs (e.g. STAC collection) Advanced merging and publishing of partitioned job results #56
    • "publication" of results: e.g. VRT generation of (partial) result, or upload to a STAC catalog Advanced merging and publishing of partitioned job results #56 Partitioned job progress tracking, reporting, visualization #58
    • support for partial result downloading Partitioned jobs: partial result support #40
  • scheduling
    • Implement queueing system for subtasks (already available in current POC)
    • Implement retry mechanism for failed subtasks Large area processing: Dynamically/adaptively start sub-jobs? #37
    • dependency handling (between splits): part of feasibility study Feasibility study: cross-back-end process graph handling #53
    • more scheduling logic on level of aggregator (e.g. based on user priorities, and back-end load instead of just dumping all the work to upstream back-ends in one go Large area processing: Dynamically/adaptively start sub-jobs? #37 Partitioned jobs tracking: limit/throttle upstream requests #38
  • user "interface"
    • how to submit split tuning params, progress tracking, ... API/UI of partitioned job feature in aggregator #57
    • standardize through federation extension? Or document in another way? API/UI of partitioned job feature in aggregator #57
    • Mechanism to display subtask progress through CSV or GeoJSON file Partitioned job progress tracking, reporting, visualization #58
    • [Optional] support user defined tile list? API/UI of partitioned job feature in aggregator #57
    • [Optional] support subtask ordering schemes -> more scheduling logic on level of aggregator API/UI of partitioned job feature in aggregator #57 Partitioned jobs based on user provided tiling #55
    • admin UI?

• Feasibility study of cross-back-end processing: Support process graphs with collections and processes from different backends Feasibility study: cross-back-end process graph handling #53

  • Workflow that combines datasets and or processes from separate back-ends
  • Module to find maximal subgraphs supported by each backend
  • Run each subgraph on respective backend
  • Module to create final result on the backend that supports all processes: not necessary to consider this a special task, will be handled automatically as side-effect of previous tasks

• Define, implement and finetune openEO "federation" extension, for better harmonization between federation members (metadata merging, error handling, monitoring, performance monitoring, usage reporting, ...)

  • Implement current state of federation extension (both in aggregator and participating back-ends) Federation extension #59
  • Iterate on extension spec: e.g. load_result support is a dependency for a couple of federation features Federation extension #59
  • Improved merging of collection (metadata) Federation of collections #5
  • Improved merging of process (metadata) Federation of (predefined) processes #4 Hide or mark proprietary processes #23

• General performance and operations

  • Improve aggregator resilience wrt failures/performance issues of some back-ends
  • state management: cache/store more info from upstream back-ends? Currently: ZooKeeper, consider alternative (ES, redis, ...)? At least provide enough abstraction to allow swapping later. State management #60
  • caching improvements: shared cache between workers, invalidation logic/tools (e.g. flush without restart) Caching improvements? #2 State management #60
  • portability of ML models across back-ends? Out of scope of this proposal
  • implications of specific hardware such as GPU's -> part of federation extension to express back-end capabilities? Federation extension #59
  • Consider supporting a high available (HA) setup: is relatively straightforward with current implementation: just spin up additional instances, and make sure state is stored in robust/HA storage solution High availability #61
  • Standalone tracking system that does not require user polls to get status from upstream back-ends (authentication?) Auth management for long running jobs #29 Background tracker #62
  • Integration tests Integration tests #63
  • Improve concurrency by switching from synchronous HTTP requests to async requests to upstream back-ends? Parallelize back-end requests (e.g. for /jobs) #28 Add parallelization for performance #33

• Sample notebooks that demonstrate

  • harmonized collections, report and resolve encountered issues.
  • Large area processing
  • Most of these notebooks will be side-effect of the development anyway