Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Redis Sentinel #133

Merged
merged 8 commits into from
Jun 28, 2022
Merged

Add support for Redis Sentinel #133

merged 8 commits into from
Jun 28, 2022

Conversation

sagepe
Copy link
Member

@sagepe sagepe commented Apr 14, 2022

This adds support for optionally using Redis Sentinel to mediate connections to Redis. In order to test, this extends the GitHub Actions with services for both Redis and Redis Sentinel which are used by the tests when REDIS_SENTINEL_HOSTS is set. (not needed, now mocking)

This also includes some updates to get tests passing against Python 3.9 and removes Python 2.7 support from the test suite at the same time as well as Django 1.11.

Currently this also includes commits from #127 rebased against master.

Fixes #132

@codecov
Copy link

codecov bot commented Apr 14, 2022

Codecov Report

Merging #133 (4145c31) into master (73a576f) will decrease coverage by 0.41%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##           master     #133      +/-   ##
==========================================
- Coverage   86.49%   86.07%   -0.42%     
==========================================
  Files          48       48              
  Lines        2132     2162      +30     
==========================================
+ Hits         1844     1861      +17     
- Misses        288      301      +13     
Impacted Files Coverage Δ
api_keys/models.py 97.91% <ø> (+0.13%) ⬆️
mapit_mysociety_org/middleware.py 76.47% <ø> (-0.68%) ⬇️
mapit_mysociety_org/wsgi_monitor.py 0.00% <0.00%> (ø)
subscriptions/models.py 100.00% <ø> (ø)
bulk_lookup/forms.py 69.41% <66.66%> (ø)
api_keys/utils.py 95.23% <80.00%> (-4.77%) ⬇️
api_keys/tests.py 100.00% <100.00%> (ø)
api_keys/urls.py 100.00% <100.00%> (ø)
bulk_lookup/csv.py 74.07% <100.00%> (-17.18%) ⬇️
bulk_lookup/models.py 85.36% <100.00%> (+0.18%) ⬆️
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73a576f...4145c31. Read the comment docs.

@sagepe
Copy link
Member Author

sagepe commented Apr 26, 2022

Noting that I attempted to update the various pyexcel-* packages to more recent versions, but these generated additional errors. Perhaps these can be returned to later.

@sagepe sagepe requested a review from dracos April 26, 2022 16:09
@sagepe sagepe marked this pull request as ready for review April 29, 2022 09:45
@dracos dracos force-pushed the 132-redis-sentinels branch 2 times, most recently from fa4f0d3 to f7f5814 Compare May 13, 2022 17:03
dracos and others added 7 commits June 7, 2022 10:04
Add Django 3.2, drop Django 1.11 and 3.1.
Drop Python 2 support, and therefore six.

Set Django 3.2 as the 'installed' version.
xlrd needs updating for recent Python versions, but the version that
supports Python 3.9 no longer handles XLSX files.

This removes the constraint on xlrd and adds a new pyexcel-xlsx plugin
as well as setting this for .xslx files.
We can just use the venv module to create the virtualenv, rather than
needing an additional package installed.
This adds basic support for using Redis Sentinel to mediate connections
to the primary Redis server used by the API functionality.

Setting `REDIS_SENTINEL_HOSTS` to a dict of "'host': port" key/values
will override any settings for `REDIS_DB_HOST` and `REDIS_DB_PORT` with
values provided by Sentinel.

Fixes #132
@sagepe
Copy link
Member Author

sagepe commented Jun 7, 2022

This has now been rebased over move-python-packages which in turn was rebased over master, so will effectively supersede #127.

@sagepe sagepe requested a review from dracos June 7, 2022 09:16
Copy link
Member

@dracos dracos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, couple of questions just to check. Looks also like commits might need tidying, there's two that now say they drop Django 3.1, I mean, I can take a look at that when ready.

# Install GDAL, correct version, right headers
gdal_version=$(gdal-config --version)
# stretch version of GDAL doesn't have matching pypi version
if [ $gdal_version = "2.1.2" ]; then gdal_version="2.1.3"; fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think we said we could drop this line now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, it's only needed on older operating systems - we'd only need it if we ever want to deploy this PR to Debian Stretch. I'll take that out.

gdal_version=$(gdal-config --version)
# stretch version of GDAL doesn't have matching pypi version
if [ $gdal_version = "2.1.2" ]; then gdal_version="2.1.3"; fi
C_INCLUDE_PATH=/usr/include/gdal CPLUS_INCLUDE_PATH=/usr/include/gdal pip install GDAL==$gdal_version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And you said in a different PR this might not be needed like this any more?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe so; testing this on newer OSes and the package builds successfully without it. I'll take that out as well to bring it in-line with the other deployments.

Debian Stretch is very close to the end of LTS support, so we no longer
need to check for GDAL 2.1.2, the version packaged on Debian Stretch,
and instead install 2.1.3, the nearest version available in PyPI.

In addition, settting environment variables no longer appears to be
needed, so this removes these.
@sagepe sagepe merged commit 4145c31 into master Jun 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Sentinel for Redis connections
2 participants