Order Engine Design

This document contains a high-level design of an Order Engine, which would ease the integration of Brokers with Booking Systems when using the Open Booking API.

It has a dependency of a yet-to-be-built Feed Normaliser, and sits within an overall architecture of components that are outlined below. The Architecture and Feed Normaliser high-level designs are included here for context.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Architecture

The services are separated by their function, and after a thorough analysis there is little useful overlap between the three feed consumption services - such that a clean separation of concerns exists, as shown in the diagram below.

Production

Order Engine

Scope

Goals:

• Microservice
• Easy webhook configuration
• Orchestrate Broker side Order logic, via hooks for Broker events in Booking flow
• Handles rollbacks
• Order persistence
• Order feed processing and notifications
• Passes through C1, C2, B
• Provides webhooks for Payment Capture, Payment Auth, Payment Refund and Order updates (Customer notice, Opportunity attendance updates, and cancellations)
• Uses Feed Normaliser for Change of Logistics notifications only
• Persists client credentials and authentication tokens for each Seller, granted by the Booking System
• Trigger and handle the associated OAuth flows
• Generate basic HTML invoices

Non-goals:

• Opportunity persistence
• Aggregating opportunity data (though the Feed Normaliser can feed an Search service directly)
• Use by Integration Tests

Interfaces

Configured with:

• Feed Normaliser URL
• OAuth Client credentials for each Booking System

Input:

• Dereferenced Data Catalogue from Feed Normaliser
• Change of Logistics RPDE Feeds
• Open Booking API endpoints
• Get Opportunity by ID endpoint

Output

• C1, C2 and B passthrough
• Payment Auth Webhook
• Payment Auth Cancellation Webhook
• Payment Capture Webhook
• Payment Refund Webhook, and Cancellation Processed Webhook
• Order Update Webhook
• Change of Logistics Webhook
• Customer Notification Webhook
• Customer Notice Webhook
• Opportunity attendance updates Webhook
• Credentials management admin page

Logistics Hash

A Change of Logistics feed endpoint should be available from the Feed Normaliser, which only includes opportunities where the contents of their logistics properties (e.g. due to change of address, startDate, etc) have changed. This can be achieved by computing a simple hash of the logistics properties, and comparing the old hash to the new hash on each RPDE item update. Where the hashes are different, a field is set against the item that includes it in the change of logistics feed at the time such a hash difference was noticed (this would likely be a “logistics_updated” field in Opportunity table, to ensure the change of logistics RPDE feed is ordered by logistics updates only. Null values of “logistics_updated” are simply not included in the feed). The logic to compute this hash should be included in the Harvester Utilities Library.

Functional Specification

A microservice that is configured through environment variables (including webhook URLs).

The microservice acts as pass-through for C1, C2, and B calls, using the Get Opportunity by ID endpoint of the Feed Normaliser to check the validity of the opportunities requested, and that their logistics match those provided.

The microservice persists each Order that is created through B, along with the orderCreationStatus defined in the Open Booking API specification. This allows it to effectively execute the rollback procedure. It also computes a logistics hash of each referenced Opportunity before storing it. The microservice relies on a Postgres database, to allow it to be easily run within a number of hosting environments.

The microservice processes the Change of Logistics feed from the Feed Normaliser, for each opportunity retrieved it computes the logistics hash against all stored Orders that reference the opportunity. It then runs a query to return all Orders that have a matching opportunity ID with a differing logistics hash, and triggers a Change of Logistics Webhook call for each, which highlights the properties that have changed - including the old and new content for each.

The microservice calls the Payment Auth and Payment Capture webhooks as part of processing B.

The microservice also processes the Orders feed, and triggers a webhook for any Order updates: Customer notice, Cancellation and Opportunity attendance updates. In the case of cancellation, it triggers a Payment Refund webhook.

The microservice will also handle the broker-side steps required to process B, including retrying the Customer Notification webhook. It will also handle the booking flow rollback, by calling the Payment Auth Cancellation, issuing an Order Deletion (DELETE) request (it does not attempt to retry payment).

The service generates basic HTML invoices associated with each Order (in accordance with the specification described here), and makes them available at a URL which is included in the Customer Notification, Order Update, and Cancellation Processed Webhooks.

The service handles persisting the OAuth Client Credentials for each Booking System, and OAuth Access Tokens for each Seller within each Booking System. It also handles triggering the OAuth flows described here.

A simple HTML page is available that allows the Broker to manage their client credentials for each booking system.

Separation of Concerns

It is possible to split this microservice into an Orders Engine library and wrapping microservice, to provide more flexibility in implementation options. However given its dependency on a database for persistence and booking system authentication credentials, this document suggests that abstracting a library from the microservice is an opportunity for future iterations rather than for the initial build.

Suggested Task Breakdown

• Create simple pass-through microservice for C1, C2 and B that stores Orders from B in a Postgres database. The jsonb data type should be used to allow the Orders to be queried.
• Improve the B call to use the Get Opportunity by ID endpoint of the Feed Normaliser to check the validity of the opportunities requested, and that their logistics match those provided, and compute a logistics hash to store alongside each opportunity (in a new custom property) within the jsonb column. It should call the Customer Notification webhook when complete.
• Add RPDE polling for the Orders feed from the booking system, and the Change of Logistics feed from the feed normaliser, using a similarly robust approach to that taken within the Feed Normaliser
• Implement Change of Logistics logic in response to new Change of Logistics feed items, as specified, including Change of Logistics webhook.
• Implement webhooks for the B happy path: Payment Auth, Payment Capture and Customer Notification
• Implement cancellation logic in response to Order feed update: Payment Refund webhook followed by Cancellation Processed Webhook.
• Implement other webhooks in response to Order feed update: Customer Notice Webhook, Opportunity attendance updates Webhook and generic Order Update Webhook - with the webhook triggered depending on the contents of the Order that has changed compared with the stored version.
• Implement booking flow rollback as specified.
• Generate HTML invoices as specified.
• Persist a table of booking systems (identified by their dataset sites), for each they should have OAuth Client Credentials and OAuth Access Tokens for each Seller within them. The pass-through call to C1, C2 and B should pick up the correct tokens from this table and use them to make the call to the correct booking system.
• Create a simple HTML page that lists the booking system’s configured, the client credentials for each booking system, and how many Seller Access Tokens exist for that booking system. It allows client credentials for Booking Systems to be added to the database - the booking systems themselves are auto-populated to the page directly from the Dereferenced Single Data Catalog, but entries in the booking system table are made when Client Credentials are set against the booking system.
• Create endpoint that triggers Seller OAuth flow

Feed Normaliser

Consumes multiple feeds from each publisher’s dataset and combines them into one

Scope

Goals:

• Data-driven configuration, so it will work “out of the box”
• Read multiple dataset sites automatically from a data catalog
• Combine all feeds into a normalised feed
• Provide a change of logistics feed
• Production-ready

Non-goals:

• Support for in-memory or test suite
• Support for anything related to the Open Booking API

Interfaces

Configured with:

• URL of JSON file containing data catalogue URLs

Input:

• Data Catalogues (which contain Dataset Site URLs)
• Dataset Sites
• Opportunity RPDE Feeds

Output

• Single normalised feeds for Sessions, Facilities, Courses, etc
• Single normalised feed for only Sessions, Facilities, Courses, etc that have had a Change in Logistics since the service started
• Dereferenced single data catalog endpoint, that includes Dataset Site content for each of the feeds contained in the single normalised feed, and polling statistics/status for each
• Get Opportunity by ID endpoint for all “Bookable” opportunity types

Functional Specification

The service should read from the Data Catalogs specified by the Data Catalog list configured in an environment variable, and spider the referenced Dataset Sites, automatically registering feeds to consume. The list of Dataset Sites being consumed should be available via the Dereferenced single data catalog endpoint, with polling statistics/status for each.

This service normalises ingested feeds from these into a standard data model (ScheduledSession, SessionSeries, EventSeries, etc), contained within tables “Opportunity”, “ParentOpportunity”, and “GrandparentOpportunity” (which include the item JSON), and provide a single feed as output that is created by joining these three tables. Note that when a “ParentOpportunity” entry is updated, all “Opportunity” entries must be updated to ensure they are included in the normalised feed.

The feeds within each dataset site should be automatically recognised - using the “parent” attribute of meta.json to automatically infer “Opportunity” and “OpportunityParent”.

There is much overlap of the logic used to normalise feeds in-memory within the Feed Listener, and such logic should be moved into the Harvester Utilities Library to be shared between both codebases.

Given this service is designed for production, the RPDE polling mechanism needs to be robust e.g. using https://www.npmjs.com/package/async-polling. Additionally it needs to be able to recover from a service outage, and hence the ‘next’ URL for each feed must be persisted to allow for such recovery.

There should be a configuration option to not retain or propagate events in the past (i.e they don’t get stored, and are filtered out at the point of ingestion), to minimise the size of the persistence required.

For each item, optionally (via configuration) run data-model-validator validation and store results - this validation is carried out on the composited opportunity rather than on individual components. Results are included in the normalised feed as “errors” alongside the RPDE “data” property.

A Change of Logistics feed endpoint is also available, which only includes opportunities where the contents of their logistics properties (e.g. due to change of address, startDate, etc) have changed. This can be achieved by computing a simple hash of the logistics properties, and comparing the old hash to the new hash on each RPDE item update. Where the hashes are different, a field is set against the item that includes it in the change of logistics feed at the time such a hash difference was noticed (this would likely be a “logistics_updated” field in Opportunity table, to ensure the change of logistics RPDE feed is ordered by logistics updates only. Null values of “logistics_updated” are simply not included in the feed). The logic to compute this hash should be included in the Harvester Utilities Library.

There is a mechanism, via a simple API endpoint, to reset all feeds within a specific dataset - causing its normalised content to be purged from the database, and for the feeds within the dataset to be synchronised again.

The Get Opportunity ID endpoint that returns the full composited JSON-LD for any of the “Bookable” opportunity types, including their embedded parents, as per the diagram below:

An API endpoint should be available that includes a dereferenced single data catalog, which contains Dataset Site content for each of the feeds contained in the single normalised feed, and polling statistics/status for each too. This can be achieved by augmenting the DataDownload objects in the distribution array within the Dataset with custom namespaced properties as follows:

• Number of pages processed successfully: “feedstatus:pagesProcessedSuccessfully”
• Number of feed read errors encountered: “feedstatus:pagesProcessedWithErrors”
• Last processed page timestamp: “feedstatus:lastPageReceivedDate”,
• Number of updates received successfully: “feedstatus:itemsProcessedSuccessfully”
• Number of validator errors for these updates: “feedstatus:itemsProcessedWithErrors”

Suggested Task Breakdown

• Using Feed Listener as inspiration, create a Postgres-based version of the single normalised feed, using a single dataset site in order to receive URL configuration of feeds.
• Extend the implementation to work with multiple Dataset Sites, spidering data catalogs by ingesting a Data Catalog List as specified by an environment variable, and updating this hourly.
• Implement dereferenced single data catalog endpoint, including polling statistics/status for each feed as well as the original Dataset Site JSON-LD content.
• Implement reliable RPDE polling mechanism, including rate limiting (using a global rate limit) and exponential backoff (with a limit) in the case of error. RPDE page being ingested should be persisted to ensure the service is robust resume after an outage.
• Implement configurable feature to prevent events in the past from being ingested and stored.
• Implement Get Opportunity by ID endpoint
• Run data-model-validator against each composite opportunity: this validation is carried out on the composited opportunity rather than on components from individual feeds. This can be done at the point of writing to the “Opportunity” table, where a “ParentOpportunity” exists.
• Implement dataset reset mechanism
• Implement change of logistics notification feed