[CT-1411] [Feature] Support Python 3.11

[dbeatty10]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Support Python 3.11.x in dbt Core and adapters:

• [CT-1529] [Feature] Support Python 3.11 dbt-snowflake#331
• [CT-1530] [Feature] Support Python 3.11 dbt-bigquery#398
• [CT-1524] [Feature] Support Python 3.11 dbt-redshift#225
• ADAP-56 Support Python 3.11 dbt-spark#524

Describe alternatives you've considered

No response

Who will this benefit?

Folks using the most recent release of Python

Are you interested in contributing this feature?

Will mark as a "good first issue" and let someone else take a shot at this

Anything else?

Last time

It looks like this work was split across multiple PRs last time, but it would be nice to have a single PR this time:

1. https://github.com/dbt-labs/dbt-core/pull/4866/files
2. https://github.com/dbt-labs/dbt-core/pull/5025/files

How to implement

Add 3.11 (and 3.10 when it is missing) to these locations:

1. dbt-core/core/setup.py

   Lines 82 to 84 in 1916784

   	"Programming Language :: Python :: 3.9",
   	"Programming Language :: Python :: 3.10",
   	],

2. dbt-core/tests/adapter/setup.py

   Lines 53 to 54 in 1916784

   	"Programming Language :: Python :: 3.9",
   	],

3. dbt-core/plugins/postgres/setup.py

   Lines 84 to 86 in 1916784

   	"Programming Language :: Python :: 3.9",
   	"Programming Language :: Python :: 3.10",
   	],

4. dbt-core/tox.ini

   Line 5 in 1916784

   [testenv:{unit,py37,py38,py39,py310,py}]

5. dbt-core/tox.ini

   Line 17 in 1916784

   [testenv:{integration,py37-integration,py38-integration,py39-integration,py310-integration,py-integration}]

6. dbt-core/.github/workflows/main.yml

   Line 74 in 1916784

   python-version: ["3.7", "3.8", "3.9", "3.10"]

7. dbt-core/.github/workflows/main.yml

   Line 119 in 1916784

   python-version: ["3.7", "3.8", "3.9", "3.10"]

8. dbt-core/Dockerfile.test

   Lines 46 to 51 in 1916784

   	python3.9 \
   	python3.9-dev \
   	python3.9-venv \
   	python3.10 \
   	python3.10-dev \
   	python3.10-venv \

9. dbt-core/CONTRIBUTING.md

   Line 59 in 1916784

   - [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, and 3.10

10. dbt-core/CONTRIBUTING.md

    Line 163 in 1916784

    [`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, and Python 3.10 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`.

11. Any other relevant file I may have missed in the two pull requests from last time (when we added Python 3.10)

[jtcohen6]

https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython

CPython 3.11 is on average 25% faster than CPython 3.10 when measured with the pyperformance benchmark suite, and compiled with GCC on Ubuntu Linux. Depending on your workload, the speedup could be up to 10-60% faster.

It took a teensy bit of finagling (48e6f96), but I was able to experiment running dbt-core + dbt-postgres with Python 3.11.0.

Completely anecdotal evidence: Running on my M1 locally, a full parse is ~3x faster (just over 9 seconds, instead of just under 27 seconds) for our internal analytics project (almost 2k files, 700+ models) compared with 3.9.13.

I'm going to remove the good_first_issue label because this was a slightly heavier lift due to internal changes within py311, and because this is worth us scheduling as work that definitely gets in for v1.4.

[leahwicz]

We should open up issues for each adapter repo as well

[dbeatty10]

We should open up issues for each adapter repo as well

✅ done!

[Fleid]

@leahwicz @nathaniel-may @jtcohen6 I'm going to assign this one to the adapter team unless there's a huge outcry. We have to do the 4 adapters + Postgres anyway so...

[jtcohen6]

@Fleid Makes sense for the adapters!

@leahwicz + I were thinking of giving the dbt-core part to @dbt-labs/core-language. It sounds like #6108 is a first step on the way toward unblocking this.

[leahwicz]

@Fleid if Adapters wants to do this, that is fine with me. #6326 has already been put up as a draft PR and but still needs works obviously. I created a Slack thread in #dev-core to consolidate knowledge on who has looked at what so far with Python 3.11 and then we can have an Adapters person pick it up from there

[jelstongreen]

Folks the Dockerfile is currently pinned to 3.10 due to a bug - any plans to upgrade to 3.11?

[dbeatty10]

Hey @jelstongreen !

Just replied to you here: #9491 (comment)