java-generic-outbox (#691)

* added TODOs * started to describe the new generic outbox API * fixed typos * described the configuration of custom outboxes * started to describe outboxing of arbitrary CAP services * Docs for new `sap.common.Timezone` (#686) * Docs for new `sap.common.TimeZone` * Rename to `Timezone`. Add API links * add internal fragment * added example for outboxing a service * detail changes to outboxing arbitrary CAP services * fixed typos * Update java/outbox.md Co-authored-by: BraunMatthias <[email protected]> * Update java/outbox.md Co-authored-by: BraunMatthias <[email protected]> * Update java/outbox.md Co-authored-by: BraunMatthias <[email protected]> * Update java/outbox.md Co-authored-by: BraunMatthias <[email protected]> * Update java/outbox.md Co-authored-by: BraunMatthias <[email protected]> * Apply suggestions from code review Co-authored-by: BraunMatthias <[email protected]> * added warning * fixed merge conflicts * described how to outbox custom CAP service events * fixed typo * minor changes * added anchor * added tip for using custom outbox when using multiple channels * Apply suggestions from code review Co-authored-by: Marc Becker <[email protected]> * changes after review * removed chapter regarding outboxing of custom CAP services * Update outbox.md * Apply suggestions from code review Co-authored-by: Marc Becker <[email protected]> * changes after code review * Apply suggestions from code review Co-authored-by: BraunMatthias <[email protected]> * changes after code review * editing --------- Co-authored-by: Christian Georgi <[email protected]> Co-authored-by: Rene Jeglinsky <[email protected]> Co-authored-by: BraunMatthias <[email protected]> Co-authored-by: Marc Becker <[email protected]>
cap-js · Feb 27, 2024 · f080ba8 · f080ba8
1 parent 0447137
commit f080ba8
Show file tree

Hide file tree

Showing 2 changed files with 167 additions and 38 deletions.
diff --git a/java/messaging-foundation.md b/java/messaging-foundation.md
@@ -33,10 +33,6 @@ In contrast, the nature of synchronous communication between services can be dis
 
 In the following, we provide a basic introduction to publish-subscribe-based messaging and then explain how to use it in CAP. If you're already familiar with publish-subscribe-based messaging, feel free to skip the following introduction section.
 
-::: tip
-The described messaging features are available from version `[email protected]`.
-:::
-
 ## Pub-Sub Messaging
 
 In a publish-subscribe-based messaging scenario (pub-sub messaging), senders send a message tagged with a topic to a message broker. Receivers can create queues at the message broker and subscribe these queues to the topics they're interested in. The message broker will then copy incoming messages matching the subscribed topics to the corresponding queues. Receivers can now consume these messages from their queues. If the receiver is offline, no messages will be lost as the message broker safely stores messages in the queue until a receiver consumes the messages. After the receiver acknowledges the successful processing of a message, the message broker will delete the acknowledged message from the queue.
@@ -88,8 +84,11 @@ As shown in the example, there are two flavors of sending messages with the mess
 
 In section [CDS-Declared Events](#cds-declared-events), we show how to declare events in CDS models and by this let CAP generate EventContext interfaces especially tailored for the defined payload, that allows type safe access to the payload.
 
-::: tip
-The messages are sent once the transaction is successful. Per default, an in-memory outbox is used, but there's also support for a persistent outbox. See [Java - Outbox](./outbox) for more information.
+::: tip Using an outbox
+The messages are sent once the transaction is successful. Per default, an in-memory outbox is used, but there's also support for a [persistent outbox](./outbox#persistent). 
+
+You can configure a [custom outbox](./outbox#custom-outboxes) for a messaging service by setting the property
+`cds.messaging.services.<key>.outbox.name` to the name of the custom outbox. This specifically makes sense when [using multiple channels](../guides/messaging/#using-multiple-channels).
 :::
 
 
@@ -668,7 +667,7 @@ The way how unsuccessfully delivered messages are treated, fully depends on the
 Not all messaging brokers provide the acknowledgement support. This means, the result of the error handler has no effect for the messaging broker.
 
 | Messaging Broker                                       | Support |         Cause          |
-|--------------------------------------------------------|:-------:|:----------------------:|
+| ------------------------------------------------------ | :-----: | :--------------------: |
 | [File Base Messaging](#local-testing)                  |  <Na/>  |                        |
 | [Event Mesh](#configuring-sap-event-mesh-support)      |  <X/>   | removed from the queue |
 | [Message Queuing](#configuring-sap-event-mesh-support) |  <X/>   | removed from the queue |

diff --git a/java/outbox.md b/java/outbox.md
@@ -20,46 +20,25 @@ status: released
 <!--- Migrated: @external/java/355-Outbox/01-service.md -> @external/java/outbox/service.md -->
 ## Concepts
 
-Usually the emit of messages should be delayed until the main transaction succeeded, otherwise recipients will also receive messages in case of a rollback.
+Usually the emit of messages should be delayed until the main transaction succeeded, otherwise recipients also receive messages in case of a rollback.
 To solve this problem, a transactional outbox can be used to defer the emit of messages until the success of the current transaction.
 
-The transactional outbox is a part of the CAP technical services for [Messaging](./messaging-foundation) and [AuditLog](./auditlog).
-
-
 ## In-Memory Outbox (Default) { #in-memory}
 
 The in-memory outbox is used per default and the messages are emitted when the current transaction is successful. Until then, messages are kept in memory.
 
 
 ## Persistent Outbox { #persistent}
 
-The persistent outbox requires a persistence layer in order to persist the messages before emitting them. Here, the to-be-emitted message is stored in a database table first. The same database transaction is used as for other operations, therefore transactional consistency is guaranteed.
+The persistent outbox requires a persistence layer to persist the messages before emitting them. Here, the to-be-emitted message is stored in a database table first. The same database transaction is used as for other operations, therefore transactional consistency is guaranteed.
 
-Once the transaction succeeds, the messages are read from the database table and emitted.
+Once the transaction succeeds, the messages are read from the database table and are emitted.
 
 - If an emit was successful, the respective message is deleted from the database table.
 - If an emit wasn't successful, there will be a retry after some (exponentially growing) waiting time. After a maximum number of attempts, the message is ignored for processing and remains in the database table. Even if the app crashes the messages can be redelivered after successful application startup.
 
-To configure the persistent outbox you can use the `outbox.persistent` section in the _application.yaml_:
-
-```yaml
-cds:
-  outbox:
-    persistent:
-      enabled: true
-      maxAttempts: 10
-      storeLastError: true
-```
-
-You have the following configuration options:
-- `enabled` (default `true`): Persistent outbox enablement.
-- `maxAttempts` (default `10`): The number of unsuccessful emits until the message is ignored. It will still remain in the database table.
-- `storeLastError` (default `true`): If this flag is enabled, the last error that occurred, when trying to emit the message
-of an entry,  is stored. The error is stored in the element `lastError` of the entity `cds.outbox.Messages`.
+To enable the persistence for the outbox, you need to add the service `outbox` of kind `persistent-outbox` to the `cds.requires` section in the _package.json_ or _cdsrc.json_. Please note that the _cdsrc.json_ file represents already the `cds` section and only the `requires` section should be added to the _cdsrc.json_ file:
 
-::: warning _❗ Warning_
-In order to enable the persistence for the outbox, you need to add the service `outbox` of kind `persistent-outbox` to the `cds.requires` section in the _package.json_ or _cdsrc.json_. Please note that the _cdsrc.json_ file represents already the `cds` section and only the `requires` section should be added to the _cdsrc.json_ file:
-:::
 
 ```json
     "cds": {
@@ -75,19 +54,170 @@ In order to enable the persistence for the outbox, you need to add the service `
 Be aware that you need to migrate the database schemas of all tenants after you've enhanced your model with an outbox version from `@sap/cds`  version 6.0.0 or later.
 :::
 
-In case of MT scenario make sure that the required configuration is also done in MTX sidecar service. In any case, the base model in all tenants needs to be updated to activate the outbox.
+For a multitenancy scenario, make sure that the required configuration is also done in MTX sidecar service. Make sure that the base model in all tenants is updated to activate the outbox.
+
+::: info Option: Add outbox to your base model
+Alternatively, you can add `using from '@sap/cds/srv/outbox';` to your base model. In this case, you need to update the tenant models after deployment but you don't need to update MTX Sidecar.
+:::
+
+CAP Java by default provides two persistent outbox services:
+
+-  `DefaultOutboxOrdered` - is used by default by messaging services
+-  `DefaultOutboxUnordered` - is used by default by the AuditLog service
+
+The default configuration for both outboxes can be overridden using the `cds.outbox.services` section, for example in the _application.yaml_:
+
+```yaml
+cds:
+  outbox:
+    services:
+      DefaultOutboxOrdered:
+        maxAttempts: 10
+        storeLastError: true
+      DefaultOutboxUnordered:
+        maxAttempts: 10
+        storeLastError: true
+```
+
+You have the following configuration options:
+- `maxAttempts` (default `10`): The number of unsuccessful emits until the message is ignored. It still remains in the database table.
+- `storeLastError` (default `true`): If this flag is enabled, the last error that occurred, when trying to emit the message
+of an entry, is stored. The error is stored in the element `lastError` of the entity `cds.outbox.Messages`.
+
+> Persistent outbox is supported starting with these versions: `@sap/cds: 5.7.0`,  `@sap/cds-compiler: 2.11.0` (`@sap/cds-dk: 4.7.0`)
+
+
+### Configuring Custom Outboxes { #custom-outboxes}
+
+Custom persistent outboxes can be configured using the `cds.outbox.services` section, for example in the _application.yaml_:
+
+```yaml
+cds:
+  outbox:
+    services:
+      MyCustomOutbox:
+        maxAttempts: 5
+        storeLastError: false
+      MyOtherCustomOutbox:
+        maxAttempts: 10
+        storeLastError: true
+```
+
+Afterwards you can access the outbox instances from the service catalog:
+
+```java
+OutboxService myCustomOutbox = cdsRuntime.getServiceCatalog().getService(OutboxService.class, "MyCustomOutbox");
+OutboxService myOtherCustomOutbox = cdsRuntime.getServiceCatalog().getService(OutboxService.class, "MyOtherCustomOutbox");
+```
+
+Alternatively it's possible to inject them into a Spring component:
+
+```java
+@Component
+public class MySpringComponent {
+  private final OutboxService myCustomOutbox;
+
+  public MySpringComponent(@Qualifier("MyCustomOutbox") OutboxService myCustomOutbox) {
+    this.myCustomOutbox = myCustomOutbox;
+  }
+}
+```
+
+::: warning When removing a custom outbox ...
+... it must be ensured that there are no unprocessed entries left.
+
+Removing a custom outbox from the `cds.outbox.services` section doesn't remove the
+entries from the `cds.outbox.Messages` table. The entries remain in the `cds.outbox.Messages` table and isn't
+processed anymore.
+
+:::
+
+## Outboxing CAP Service Events
+
+Outbox services support outboxing of arbitrary CAP services. Typical use cases are remote OData
+service calls, but also supports calls to other CAP services to decouple them from the business logic flow.
+
+The API `OutboxService.outboxed(Service)` is used to wrap services with outbox handling. Events triggered
+on the wrapper are stored in the outbox first, and executed asynchronously. Relevant information from
+the `RequestContext` is stored with the event data, however the user context is downgraded to a system user context.
+
+The following example shows how to outbox a CAP Java service:
+
+```java
+OutboxService myCustomOutbox = ...;
+CqnService remoteS4 = ...;
+CqnService outboxedS4 = myCustomOutbox.outboxed(remoteS4);
+```
+
+The outboxed service can be cached; caching them is thread-safe.
+Any service that implements the interface `com.sap.cds.services.Service`
+or an inherited interface can be outboxed. Each call to the outboxed service is asynchronously
+executed, if the API method internally calls the method `com.sap.cds.services.Service.emit(EventContext)`.
 
-::: tip
-Alternatively, you can add `using from '@sap/cds/srv/outbox';` to your base model. You need to update the tenant models after deployment. You don't need to update MTX Sidecar in this case.
+::: warning Asynchronous execution
+All calls to `CqnService.run` methods return null since they're executed asynchronously.
 :::
 
-::: tip
-Persistent outbox is supported starting with these version: `@sap/cds: 5.7.0`,  `@sap/cds-compiler: 2.11.0` (`@sap/cds-dk: 4.7.0`)
+A service wrapped by an outbox can be unboxed by calling the API `OutboxService.unboxed(Service)`. Method calls to the unboxed
+service are executed synchronously without storing the event in an outbox.
+
+::: tip Custom outbox for scaling
+The default outbox services can be used for outboxing arbitrary CAP services. If you detect a scaling issue,
+you can define custom outboxes that can be used for outboxing.
 :::
 
+## Technical Outbox API
+
+Outbox services provide the technical API `OutboxService.submit(String, OutboxMessage)` that can
+be used to send custom messages via outbox.
+When submitting a custom message, an `OutboxMessage` needs to be provided that can contain parameters for the
+event. As the `OutboxMessage` instance is serialized and stored in the database, all data provided in that message
+must be serializable and deserializable to/from JSON. The following example shows the submission of a custom message to an outbox:
+
+```java
+OutboxService outboxService;
+
+OutboxMessage message = OutboxMessage.create();
+Map<String, Object> parameters = new HashMap<>();
+parameters.put("name", "John");
+parameters.put("lastname", "Doe");
+message.setParams(parameters);
+
+outboxService.submit("myEvent", message);
+```
+
+A handler for the custom message must be registered on the outbox service instance to perform the processing of the
+message as shown in the following example:
+
+```java
+OutboxService outboxService;
+
+outboxService.on("myEvent", null, (ctx) -> {
+  OutboxMessageEventContext context = ctx.as(OutboxMessageEventContext.class);
+  OutboxMessage message = context.getMessage();
+  Map<String, Object> params = message.getParams();
+  String name = (String) param.get("name");
+  String lastname = (String) param.get("lastname");
 
-#### Troubleshooting
+  // Perform task for myEvent
+
+  ctx.setCompleted();
+});
+```
+
+You must ensure that the handler is setting the context to completed before returning.
+Also the handler shall only be registered once on the outbox service.
+
+[Learn more about event handlers.](./provisioning-api){.learn-more}
+
+
+## Troubleshooting
 
 To manually delete entries in the `cds.outbox.Messages` table, you can either
 expose it in a service or programmatically modify it using the `cds.outbox.Messages`
 database entity.
+
+::: tip Use paging logic
+Avoid to read all entries of the `cds.outbox.Messages` table at once, as the size of an entry is unpredictable
+and depends on the size of the payload. Prefer paging logic instead.
+:::