Skip to content

Commit

Permalink
SIANXSVC-826: Added direct-reply result backend
Browse files Browse the repository at this point in the history
  • Loading branch information
beachmachine committed Nov 30, 2022
1 parent c617b06 commit 51a2803
Show file tree
Hide file tree
Showing 8 changed files with 359 additions and 20 deletions.
5 changes: 3 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ jobs:
fail-fast: false
matrix:
python-version:
- "3.6"
- "3.7"
- "3.8"
- "3.9"
- "3.10"
celery-version:
- "5.0"
- "5.1"
- "5.2"

steps:
- uses: actions/checkout@v2
Expand Down Expand Up @@ -51,7 +51,8 @@ jobs:
ln -s ./tests/test_project/manage.py manage.py
# run tests with coverage
coverage run --source='./celery_amqp_backend' manage.py test
coverage run --append --source='./celery_amqp_backend' manage.py test --settings=test_project.settings.backend
coverage run --append --source='./celery_amqp_backend' manage.py test --settings=test_project.settings.direct_reply_backend
coverage xml
- name: Upload coverage to Codecov
Expand Down
46 changes: 34 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,26 @@ celery-amqp-backend
[![Test Status](https://github.com/anexia/celery-amqp-backend/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/anexia/celery-amqp-backend/actions/workflows/test.yml)
[![Codecov](https://codecov.io/gh/anexia/celery-amqp-backend/branch/main/graph/badge.svg)](https://codecov.io/gh/anexia/celery-amqp-backend)

`celery-amqp-backend` is a rewrite of the Celery's original `amqp://` result backend, which was removed from Celery
with version 5.0. Celery encourages you to use the newer `rpc://` result backend, as it does not create a new
result queue for each task and thus is faster in many circumstances. However, it's not always possible to switch
to the new `rpc://` result backend, as it does have restrictions as follows:
`celery-amqp-backend` contains two result backens for Celery.

# `AMQPBackend` result backend

The `AMQPBackend` result backend is a rewrite of the Celery's original `amqp://` result backend, which was removed from
Celery with version 5.0. Celery encourages you to use the newer `rpc://` result backend, as it does not create a new
result queue for each task and thus is faster in many circumstances. However, it's not always possible to switch to the
new `rpc://` result backend, as it does have restrictions as follows:
- `rpc://` does not support chords.
- `rpc://` results may hold a wrong state.
- `rpc://` may lose results when using `gevent` or `greenlet`.

The result backend `celery_amqp_backend.AMQPBackend://` does not suffer from the same issues.

# `DirectReplyAMQPBackend` result backend

The `DirectReplyAMQPBackend` result backend makes use of RabbitMQ's direct-reply feature. It is much faster than the
traditional `AMQPBackend` result backend and should even beat Celery's built-in `rpc://` result backend. However,
contrary to the `AMQPBackend` result backend it does not support chords.

# Installation

With a [correctly configured](https://pipenv.pypa.io/en/latest/basics/#basic-usage-of-pipenv) `pipenv` toolchain:
Expand All @@ -29,7 +39,7 @@ You may also use classic `pip` to install the package:
pip install celery-amqp-backend
```

# Getting started
# Getting started with `AMQPBackend`

## Configuration options

Expand Down Expand Up @@ -57,6 +67,19 @@ Default: `'direct'`

The type of the exchange created by the backend (e.g. `'direct'`, `'topic'` etc.).

# Getting started with `DirectReplyAMQPBackend`

## Important notes

* You must set the `reply_to` property of Celery tasks to `"amq.rabbitmq.reply-to"`.
* The `DirectReplyAMQPBackend` does not support chords.

## Configuration options

### `result_backend: str`

Set to `'celery_amqp_backend.DirectReplyAMQPBackend://'` to use this result backend.

## Example configuration

```python
Expand All @@ -68,13 +91,12 @@ result_exchange_type = 'direct'

# Supported versions

| | Celery 5.0 | Celery 5.1 |
|-------------|------------|------------|
| Python 3.6 |||
| Python 3.7 |||
| Python 3.8 |||
| Python 3.9 |||
| Python 3.10 |||
| | Celery 5.0 | Celery 5.1 | Celery 5.2 |
|-------------|------------|------------|------------|
| Python 3.7 ||||
| Python 3.8 ||||
| Python 3.9 ||||
| Python 3.10 ||||

# List of developers

Expand Down
2 changes: 1 addition & 1 deletion celery_amqp_backend/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
from .exceptions import *
from .backend import *
from .exceptions import *
Loading

0 comments on commit 51a2803

Please sign in to comment.