Convert project to poetry

[guidopetri]

What type of PR is this?

• Other

Description

This PR converts us from requirements.txt-pinned dependencies to poetry, which has much more stable dependency chains and should prevent us from having updated dependencies break our project.

I added two groups (besides the main group): dev, for development dependencies, and all_ds, to keep the separation we had with the requirements files.

How is this tested?

• Unit tests (pytest, jest)
• E2E Tests (Cypress)
• Manually

Untested as of yet (so don't merge), but the plan is to:

• build the image
• pip freeze > image_packages.txt
• on a new installation, install dependencies with poetry install
• pip freeze > poetry_packages.txt
• diff image_packages.txt poetry_packages.txt

Related Tickets & Documents

#5910

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

N/A

[codecov]

Codecov Report

Merging #6423 (523d3f1) into master (b1f738f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6423   +/-   ##
=======================================
  Coverage   60.99%   60.99%           
=======================================
  Files         157      157           
  Lines       12819    12819           
  Branches     1741     1741           
=======================================
  Hits         7819     7819           
  Misses       4764     4764           
  Partials      236      236           

Files Changed	Coverage Δ
redash/authentication/ldap_auth.py	34.78% <ø> (ø)

[justinclift]

No objection here, it sounds like it'll be useful.

@getredash/maintainers @arikfr This ok with yourselves? 😄

[guidopetri]

This isn't quite ready yet :P

[snickerjp]

I thought Poetry was a good attempt.

[justinclift]

This isn't quite ready yet :P

No worries. Was more thinking we should ensure the concept itself is ok with people. 😄

[guidopetri]

#6428

[guidopetri]

grep -Ir requirements . returns only hits in ./bin/upgrade now, and I can't find any programmatic use for that script. @arikfr , do you remember what this was used for? Should we update it?

I've made one important change to the Dockerfile: I removed the skip_ds_deps, skip_dev_deps, test_all_deps build args and replaced them with an install_groups build argument. This is passed directly to poetry and can control which dependency groups are installed. By default, it will install main, all_ds and dev; I changed the Cypress yaml and CI workflows accordingly. The image builds on my local now with docker build -t redash ..

I'm now going to create a venv to compare the poetry install with the requirements.txt install and see if there are any differing packages.

[guidopetri]

Ok, here's the diff list I got:

55a56
> futures==3.0.5
74c75
< inflection==0.5.1
---
> inflection==0.3.1
115a117
> pip==23.2.1
159c161,162
< rdflib==7.0.0
---
> rdflib==6.3.2
> redash==23.9.0       /home/sid/Git/redash
175a179
> setuptools==68.2.0
210a215
> wheel==0.40.0

wheel/setuptools are from my pyenv, and redash is because I installed without --no-root. The inflection version being different is interesting, as well as rdflib. I suspect this is some incompatibility in one of the packages with the newer versions of these two packages. As long as the tests pass, I'm good merging. @getredash/maintainers , any objections to this?

[justinclift]

Just noticed that this is missing the dependency updates from the rq related PR yesterday: 41495ba

They should be added yeah?

[guidopetri]

d727602 should be the rq changes. Is it not showing up on the diff?

I forgot about the dev suffix, I think poetry is fine with it but let me add that and double check. It looks like I also missed something in the CI.

[justinclift]

Is it not showing up on the diff?

Ahhh, yeah that wasn't showing up for me. No worries. 😄

[guidopetri]

Alright, I've enabled auto-merge on this PR. @justinclift when you have a chance, can you re-review to merge?

[arikfr]

bin/upgrade is a relic from the pre-docker days. We should just delete it.

[arikfr]

Suggested change

	authors = ["Your Name <[email protected]>"]
	authors = ["Redash maintainers and contributors"]

Or does it have to be an email address?

[guidopetri]

Shoot, the auto-merge didn't quite help here did it. Sorry @arikfr ! I'll create a followup PR. The author does indeed have to include an email address; do we have one on the redash.io domain that we could use?

[arikfr]

I can set something up on redash.io - just wonder where to send it? Maybe there is some email->discord integration?

[justinclift]

Maybe there is some email->discord integration?

Interesting idea. We'd better make sure it never sends anything confidential via that channel though (password resets, etc), unless we're sure it can direct to some private for-admins-only channel. 😄