Skip to content

Commit

Permalink
docs: [FC-0074] add details for illustrating how filters work (#231)
Browse files Browse the repository at this point in the history
  • Loading branch information
mariajgrimaldi authored Dec 10, 2024
1 parent 6b8354e commit a791841
Show file tree
Hide file tree
Showing 3 changed files with 48 additions and 18 deletions.
Binary file added docs/_images/openedx-filters-workflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 39 additions & 13 deletions docs/concepts/openedx-filters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,44 +13,70 @@ What are Open edX Filters?

An Open edX Filter is a pipeline mechanism that executes a series of functions when configured. Each function receives input arguments, which are data used by the process in execution, and returns the same arguments, possibly modified. Given this design, filters can modify the application flow according to the specified configuration, altering or adding new behaviors during execution time.

The pipeline mechanism is implemented by a class called `OpenEdxPublicFilter`_, which provides the necessary tools to fulfill the Open edX Filters requirements, such as ordered execution, configurability, interchangeable functions, argument definition, and cumulative behavior. This enables filters to modify the flow of the application dynamically during runtime based on predefined business logic or conditions. We refer to the pipeline mechanism as the **Pipeline Tooling** throughout this document.

How do Open edX Filters work?
-----------------------------

Open edX Filters are implemented using an accumulative pipeline mechanism, which executes a series of functions in a specific order. Each function in the pipeline receives the output of the previous function as input, allowing developers to build complex processing logic by chaining multiple functions together. The pipeline ensures that the order of execution is maintained and that the result of a previous function is available to the current one in the form of a pipeline. The filter execution process follows these steps:
Open edX Filters are implemented using an accumulative pipeline mechanism, which executes a series of functions in a specific order. Each function in the pipeline receives the output of the previous function as input, allowing developers to build complex processing logic by chaining multiple functions together. The pipeline ensures that the order of execution is maintained and that the result of a previous function is available to the current one in the form of a pipeline.

This pipeline mechanism is implemented by the `OpenEdxPublicFilter`_ class, which provides the necessary tools to fulfill the Open edX Filters requirements mentioned previously, such as ordered execution, configurability, interchangeable functions, argument definition, and cumulative behavior. This enables filters to modify the flow of the application dynamically during runtime based on predefined business logic or conditions.

Architectural Diagram
*********************

In this diagram, we illustrate the workflow of triggering an Open edX Filter:

.. image:: ../_images/openedx-filters-workflow.png
:alt: Open edX Filters Workflow
:align: center

Components
~~~~~~~~~~

#. An application component (caller) invokes the filter by calling the ``run_filter()`` method implemented by the filter definition.
#. Application (caller): The component that calls the filter during its execution, triggering the pipeline to process the input data. Developers may have added this call to a part of the application to include different behaviors. E.g., a user enrolls in a course, triggering the `CourseEnrollmentStarted filter`_.
#. OpenEdxPublicFilter: The class that implements all methods used to manage the execution of the filter.
#. PipelineStep1...N: The pipeline steps that are executed in sequence, each processing the input data and returning potentially modified data. These steps are defined by the developer to introduce additional behaviors. E.g., a pipeline step that checks user eligibility for enrollment.

#. The ``run_filter`` method calls the **Pipeline Tooling** under the hood, which manages the execution of the filter's pipeline.
Workflow
~~~~~~~~

#. The filter's tooling retrieves the configuration from ``OPEN_EDX_FILTERS_CONFIG``, which defines a list of N functions :math:`f_1, f_2, \ldots, f_{n}` that will be executed.
#. An application component (caller) invokes the filter during its execution by calling the ``run_filter`` method implemented by its :term:`filter definition<Filter Definition>`.

#. The tooling then executes each function in the pipeline sequentially, starting with :math:`f_1`, which processes the input arguments and applies the developer's operations, returning potentially modified arguments.
#. The caller passes the input data to the filter through the ``run_filter`` method, this data are in-memory platform objects that the filter will process.

#. The next function (if there are more than one) :math:`f_2` receives the potentially modified arguments and applies further operations, returning another modified set of arguments. This process continues through the list of functions.
#. The ``run_filter`` method of the filter calls the ``OpenEdxPublicFilter.run_pipeline`` method under the hood, which manages the execution of the filter's pipeline.

#. This method retrieves the configuration from ``OPEN_EDX_FILTERS_CONFIG``, which defines a list of N functions :math:`f_1, f_2, \ldots, f_{n}` that will be executed.

#. Then it executes each function in the pipeline sequentially, starting with :math:`f_1`, which processes the input arguments ``kwargs`` and applies the developer's operations, returning potentially modified arguments ``kwargs_1``.

#. The next function (if there are more than one) :math:`f_2` receives the potentially modified arguments ``kwargs_1`` and applies further operations, returning another modified set of arguments ``kwargs_2``. This process continues through the list of functions.

#. Each subsequent function receives the output from the previous function and returns its modified output until all functions have been executed.

#. Additionally, at any point in the pipeline, a developer can halt execution by raising an exception, based on conditions defined in the processing logic, to stop the application flow.
#. At any point in the pipeline, a developer can halt execution by raising an exception, based on conditions defined in the processing logic, to stop the application flow. Let's assume that :math:`f_{2}` raises an exception instead of returning the modified arguments ``kwargs_2``. In this case, the pipeline stops, and the ``OpenEdxPublicFilter.run_pipeline`` method raises the exception to the caller as the final output. From there the caller can handle the exception as needed.

#. Once the final function :math:`f_{n}` has been executed, the final modified arguments are returned to the caller, which may use them for the remaining part of its execution.
#. If no exceptions are raised, the pipeline continues executing the functions until the final function :math:`f_{n}` has been executed.

#. The final modified arguments ``kwargs_n`` are returned to the caller, which may use them for the remaining part of its execution.

Each function in the pipeline has the ability to modify the input data, add new data, or halt execution based on specific conditions, such as raising exceptions if certain criteria is not met. This pipeline structure ensures that complex business logic can be applied during runtime without directly altering the application code.

Here's an example of a filter in action:
Real-Life Example
~~~~~~~~~~~~~~~~~

Here's an example of the `CourseEnrollmentStarted filter`_ in action:

#. A user enrolls in a course, triggering the `CourseEnrollmentStarted filter`_ by calling the ``run_filter`` method with the enrollment details. This filter processes information about the user, course, and enrollment details.

#. The filter tooling executes a series of functions configured in ``OPEN_EDX_FILTERS_CONFIG``, e.g. checking user eligibility for enrollment, updating the enrollment status, and notifying the user about the enrollment.
#. The ``run_pipeline`` method executes a series of functions configured in ``OPEN_EDX_FILTERS_CONFIG``, e.g. checking user eligibility for enrollment or updating the enrollment status in a third-party system.

#. Each function can modify the input data or halt the process based on business logic, e.g. denying enrollment if the user is ineligible.

#. The final output of the pipeline, such as the updated enrollment details, is returned to the caller, or an exception is raised if the user is not eligible.

#. The process is complete once all functions in the pipeline have executed, and the enrollment process continues based on the final output.

By organizing this workflow through a pipeline, Open edX Filters allow developers to extend platform functionality in a flexible and maintainable way.
By running filters in key places of the Open edX platform, developers can extend the platform's functionality in a flexible and maintainable way.

How are Open edX Filters used?
------------------------------
Expand Down
14 changes: 9 additions & 5 deletions docs/reference/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,26 @@ A filter has multiple components that are used to define, execute and handle fil
Pipeline
A pipeline is a list of functions executed in a specific order; these functions are known as pipeline steps. Each function in the pipeline takes the output of the previous function as its input, with the final function's output serving as the overall output of the filter. The pipeline behavior was inspired by the `Python Social Auth accumulative pipeline`_, which is described in detail in the :doc:`/decisions/0003-hooks-filter-tooling-pipeline` ADR. These pipelines are configured in the filter configuration and are executed in sequence.

Filter Tooling
The filter tooling is a set of methods that manage the execution of the filter pipeline. The tooling retrieves the filter configuration, executes the pipeline steps in the specified order, and handles exceptions raised by the pipeline steps. This tooling ensures that the pipeline steps are executed in the correct order and that the output of each step is passed to the next step in the pipeline. All this is mainly done by the `OpenEdxPublicFilter`_ class, which provides the necessary definitions to fulfill the Open edX Filters requirements.

Pipeline Step
A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from the base class `PipelineStep`_ and define specific logic within their `run_filter`_ method, which is executed by the pipeline tooling when the filter is triggered.

Filter Definition
A filter definition is a class that inherits from `OpenEdxPublicFilter`_ that implements the ``run_filter`` method which defines the input and output behavior of the filter. This class executes the configured pipeline steps by calling the method `run_pipeline`_, passing down the input arguments, handling exceptions and returning the final output of the filter. Since the ``run_filter`` method is the entry point for the filter, the pipeline steps must have the same signature as the filter definition.
A filter definition is a class that inherits from `OpenEdxPublicFilter`_ that implements the ``run_filter`` method which defines the input and output behavior of the filter. This class executes the configured pipeline steps by calling the method `run_pipeline`_, passing down the input arguments, handling exceptions and returning the final output of the filter. Since the ``run_filter`` method is the entry point for the filter, the pipeline steps must have the same signature as the filter definition. E.g., the `CourseEnrollmentStarted filter`_ is a filter definition that processes information about the user, course, and enrollment details.

Filter Signature
The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, specifying the data the filter will process. The filter signature is used to ensure that all pipeline steps have the same input and output structure, enabling interchangeability between steps.
The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, specifying the data the filter will process. The filter signature is used to ensure that all pipeline steps have the same input and output structure, enabling interchangeability between steps. E.g., the `CourseEnrollmentStarted filter`_ signature might include parameters like ``user``, ``course_key``, and ``enrollment mode``.

Filter Type
The filter type is a unique identifier for the filter, following a standardized format following the :doc:`/decisions/0004-filters-naming-and-versioning`. This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter.
The filter type is a unique identifier for the filter, following a standardized format following the :doc:`/decisions/0004-filters-naming-and-versioning`. This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter. E.g., the `CourseEnrollmentStarted filter`_ has the `filter_type` ``org.openedx.learning.course.enrollment.started.v1``.

Filter Exceptions
Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, allowing the filter to control the application flow.
Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, allowing the filter to control the application flow. E.g., the `CourseEnrollmentStarted filter`_ might raise an exception if the user is ineligible for enrollment called ``PreventEnrollment``.

Filter Configuration
Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed.
Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. E.g., the configuration for the `CourseEnrollmentStarted filter`_ might include settings like ``fail_silently: False`` and ``['my_plugin.filters.StopEnrollmentIfNotValidEmail']`` as its pipeline steps. See the :doc:`/decisions/0002-hooks-filter-config-location` for more details on the configuration format.

This glossary provides a high-level overview of the key concepts and components of the Open edX Filters library. Understanding these terms will help you implement filters in your application and leverage the filter tooling to control the flow of your application based on specific conditions. For a better illustration of these concepts, refer to the :doc:`/how-tos/using-filters` guide.

Expand All @@ -33,3 +36,4 @@ This glossary provides a high-level overview of the key concepts and components
.. _run_filter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/filters.py#L60
.. _OpenEdxPublicFilter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/tooling.py#L14
.. _run_pipeline: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/tooling.py#L164
.. _CourseEnrollmentStarted filter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/learning/filters.py#L142

0 comments on commit a791841

Please sign in to comment.