docs: [FC-0074] explain how the event bus works and how to use it #426

mariajgrimaldi · 2024-11-29T13:19:25Z

Description

This PR adds sections dedicated to the Open edX Event bus technology:

An Overview of the technology:
What is the Open edX Event Bus?
Why use the Open edX Event Bus?
How Does the Open edX Event Bus Work? With an architectural diagram.
How is the Open edX Event Bus Used?
Using the Open edX Event Bus: install event-bus plugins, configure them, produce events, consume events, run the consumer
Event Bus Configurations: the available configurations taken from in-line code annotations.
Glossary terms for the event-bus: event bus, message, message broker, worker, producer, consumer, topic

These docs were written based on the available documentation about the Open edX Event Bus:
https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus
https://openedx.atlassian.net/wiki/spaces/AC/pages/3474784257/Event+Bus+Architecture+Overview

This PR partially addresses #238

openedx-webhooks · 2024-11-29T13:19:30Z

Thanks for the pull request, @mariajgrimaldi!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/hooks-extension-framework. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

mariajgrimaldi · 2024-12-02T13:47:40Z

docs/how-tos/adding-events-to-event-bus.rst

@@ -1,42 +0,0 @@
-Using the Open edX Event bus


This was moved to this file: https://github.com/openedx/openedx-events/blob/3954196516163fa6f31561885e251e63c6c6e50b/docs/how-tos/using-the-event-bus.rst

Maybe we should add a redirect here?

docs/reference/glossary.rst

sarina · 2024-12-02T17:11:26Z

Should these docs live here? Or on docs.openedx.org? (I don't know the right answer just asking)

mariajgrimaldi · 2024-12-03T20:05:53Z

Hey @sarina, not sure either. A similar question was raised here #279, what do you think about the alternatives?

mariajgrimaldi · 2024-12-03T20:09:38Z

docs/concepts/event-bus.rst

+* **Eliminate Expensive, Delayed, Batch Synchronization**: replace expensive batch processing with near real-time data updates.
+* **Reduce the need for Plugins**: reduce the computational load for plugins that don't need to run in the same process by allowing cross-service communication of lifecycle events.
+
+How Does the Open edX Event Bus Work?


Should we also include more detail on how the data is serialized/deserialized? Or is this a good high-level overview?

Either way, I'm writing some documents on adding event bus support to new events so that the serialization/deserialization details can be included there.

I think this is good for a high level overview. I think it would be nice to put something in here about the differences between the Event Bus and Celery tasks, and when to use each. It's one of the more common questions I get.

Yes! I'm going to add a comparison table or something similar about them. Thank you for the suggestion!

I compared the event bus async communication style with async tasks here: fb51a60. Let me know if this is what you had in mind. Thanks!

bmtcril

Just a couple of comments, this is looking great!

bmtcril · 2024-12-03T20:39:04Z

docs/how-tos/using-the-event-bus.rst

+Install the Open edX Event Bus Plugin
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, you need to install the Open edX Event Bus plugin in both the producing and consuming services. The plugin is a Django app that provides the necessary tools and configurations to produce and consume events. You could install the Redis plugin by running:


Maybe a note here about things that go into selecting a backend and why we're using redis here?

Thanks for the suggestion! I added a note about redis being the community supported solution here: 0d3218a

sarina · 2024-12-03T21:41:59Z

@mariajgrimaldi the more I think about it the more I think leaving the docs here is good!

@bmtcril said he'd take a look at these docs for me, since he's more familiar with the event bus.

docs/how-tos/using-the-event-bus.rst

docs/reference/glossary.rst

mariajgrimaldi · 2024-12-04T21:08:37Z

docs/how-tos/using-the-event-bus.rst

@@ -0,0 +1,84 @@
+Using the Open edX Event Bus


@bmtcril: I also created a document on how to ensure event bus compatibility for new events, what do you think about merging these two docs into one?

I think they are probably good as they are, don't want to make them too long!

bmtcril · 2024-12-05T15:09:44Z

docs/concepts/event-bus.rst

+
+- Service A sends the event as a message to the event bus and continues its execution, as we previously explained.
+- Service B polls the message broker for new messages and processes them.
+- Service B re-emits the event with the data that was transmitted.


I think re-emits is a little confusing here, I initially read it as "re-emits the event to the bus". Maybe change it to "locally emits the Django signal of the event..."?

You're right, I improved the wording here: 12e4768

bmtcril · 2024-12-05T15:13:40Z

docs/concepts/event-bus.rst

+
+This is an example of an asynchronous approach (from the producer point of view) to send messages to another services but with a blocking nature.
+
+Use the Open edX Event bus instead of asynchronous tasks when:


This is great! A thing we don't talk about often, and I'm not sure we should, is that the event bus (especially Kafka) can be used to send event data out of the Open edX ecosystem entirely. Many different databases and products can consume Kafka events for various purposes, for instance Student Information Services or Contact Relationship Managers may listen to enrollment events to update their own state. Do you think it's worth a bullet point here about that?

That's a pretty interesting use case we could include here: https://docs.openedx.org/projects/openedx-events/en/latest/reference/real-life-use-cases.html#cross-services-communication. I think it'd give community folks a more rounded view of all that can be done with the event bus, do you have more info on this use case? Although I could use the info in the comment.

For this document purpose, I added some high-level ideas of external services that can consume events sent through the event bus: ee3f8e5

bmtcril · 2024-12-05T15:18:41Z

docs/how-tos/using-the-event-bus.rst


 First, you need to install the Open edX Event Bus plugin in both the producing and consuming services. The plugin is a Django app that provides the necessary tools and configurations to produce and consume events. You could install the Redis plugin by running:

 .. code-block:: bash

   pip install edx-event-bus-redis

-Configure the Event Bus
-~~~~~~~~~~~~~~~~~~~~~~~
+.. note:: Redis is the community-supported plugin for the Open edX Event Bus and is the recommended plugin to use. You can find more information about the Redis plugin in the `event-bus-redis`_ repository. However, you can also implement your own plugin with your preferred message broker by following the :doc:`../how-tos/add-new-event-bus-concrete-implementation` documentation.


I think that both redis and kafka are community supported, but redis is just the default. We should mention Kafka as an alternative before doing another implementation.

Thanks for clarifying, I actually thought it was only Redis. I've changed the note with the suggestion: 1a4bafa

bmtcril

Looks good to me!

…orted

mariajgrimaldi · 2024-12-10T10:39:00Z

@bmtcril, thank you so much for the thorough review!

@sarina: I'm going to go ahead and merge this since Brian already approved it. Let me know if there's another suggestion you'd like me to include!

mariajgrimaldi · 2024-12-11T13:12:12Z

docs/how-tos/using-the-event-bus.rst

@@ -0,0 +1,87 @@
+Using the Open edX Event Bus


I'm going to add more details about this, like console logs, etc, in another PR. Thanks!

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Nov 29, 2024

mariajgrimaldi changed the title ~~docs: explain how the event bus works and how to use it~~ docs: [FC-0074] explain how the event bus works and how to use it Nov 29, 2024

mariajgrimaldi added the FC Relates to an Axim Funded Contribution project label Nov 29, 2024

mariajgrimaldi force-pushed the MJG/event-bus-concepts branch from 1feabc8 to 3954196 Compare November 29, 2024 14:13

mariajgrimaldi commented Dec 2, 2024

View reviewed changes

docs/reference/glossary.rst Show resolved Hide resolved

mariajgrimaldi marked this pull request as ready for review December 2, 2024 14:07

mariajgrimaldi requested a review from a team as a code owner December 2, 2024 14:07

mariajgrimaldi marked this pull request as draft December 2, 2024 14:08

mariajgrimaldi marked this pull request as ready for review December 2, 2024 16:16

mariajgrimaldi requested review from sarina and robrap December 2, 2024 16:16

mariajgrimaldi requested a review from bmtcril December 3, 2024 20:07

mariajgrimaldi commented Dec 3, 2024

View reviewed changes

bmtcril reviewed Dec 3, 2024

View reviewed changes

mariajgrimaldi commented Dec 4, 2024

View reviewed changes

bmtcril reviewed Dec 5, 2024

View reviewed changes

bmtcril approved these changes Dec 5, 2024

View reviewed changes

mariajgrimaldi added 7 commits December 9, 2024 12:01

docs: explain how the event bus works and how to use it

3ced530

refactor: move configurations to reference and improve formatting

1db044e

refactor: drop references to confluence event bus page

3619687

refactor: reference terms in glossary

be4da06

refactor: add comparison between the event bus and async tasks

2b24bcf

refactor: add note about redis being the community supported solution

6831456

refactor: make clear that service converts messages into signals

5433d78

mariajgrimaldi added 3 commits December 9, 2024 12:02

refactor: mention that external services can also consume events

531e910

refactor: make a note about redis and kafka being both community supp…

0615ccd

…orted

fix: remove trailing spaces

dc5d29e

mariajgrimaldi force-pushed the MJG/event-bus-concepts branch from 5777568 to dc5d29e Compare December 9, 2024 11:02

mariajgrimaldi added 2 commits December 9, 2024 12:12

fix: use references to the local event bus docs

7e091e9

docs: add missing connector to sentence

99ffa52

mariajgrimaldi commented Dec 11, 2024

View reviewed changes

mariajgrimaldi merged commit b681ebd into main Dec 11, 2024
8 checks passed

mariajgrimaldi deleted the MJG/event-bus-concepts branch December 11, 2024 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: [FC-0074] explain how the event bus works and how to use it #426

docs: [FC-0074] explain how the event bus works and how to use it #426

mariajgrimaldi commented Nov 29, 2024 •

edited

Loading

openedx-webhooks commented Nov 29, 2024 •

edited

Loading

mariajgrimaldi Dec 2, 2024

mariajgrimaldi Dec 2, 2024

sarina commented Dec 2, 2024

mariajgrimaldi commented Dec 3, 2024

mariajgrimaldi Dec 3, 2024

bmtcril Dec 3, 2024

mariajgrimaldi Dec 4, 2024

mariajgrimaldi Dec 4, 2024

bmtcril left a comment

bmtcril Dec 3, 2024

mariajgrimaldi Dec 4, 2024

sarina commented Dec 3, 2024

mariajgrimaldi Dec 4, 2024

bmtcril Dec 5, 2024

mariajgrimaldi Dec 5, 2024

bmtcril Dec 5, 2024

mariajgrimaldi Dec 5, 2024

bmtcril Dec 5, 2024

mariajgrimaldi Dec 5, 2024

bmtcril Dec 5, 2024

mariajgrimaldi Dec 5, 2024

bmtcril left a comment

mariajgrimaldi commented Dec 10, 2024

mariajgrimaldi Dec 11, 2024


		This is an example of an asynchronous approach (from the producer point of view) to send messages to another services but with a blocking nature.

		Use the Open edX Event bus instead of asynchronous tasks when:

docs: [FC-0074] explain how the event bus works and how to use it #426

docs: [FC-0074] explain how the event bus works and how to use it #426

Conversation

mariajgrimaldi commented Nov 29, 2024 • edited Loading

Description

openedx-webhooks commented Nov 29, 2024 • edited Loading

What's next?

🔘 Get product approval

🔘 Provide context

🔘 Get a green build

🔘 Let us know that your PR is ready for review:

Who will review my changes?

Where can I find more information?

When can I expect my changes to be merged?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarina commented Dec 2, 2024

mariajgrimaldi commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmtcril left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarina commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmtcril left a comment

Choose a reason for hiding this comment

mariajgrimaldi commented Dec 10, 2024

Choose a reason for hiding this comment

mariajgrimaldi commented Nov 29, 2024 •

edited

Loading

openedx-webhooks commented Nov 29, 2024 •

edited

Loading