chore: update Python runtime and AWS CDK (#38)

* Bump db layer to Py3.12

* Bump root devtools to Py3.12

* Bump date_generator to Py3.12

* Bump downloader to Py3.12

* Bump link_fetcher to Py3.12

* Bump mock_scihub_product_api to Py3.12

* Bump mock_scihub_search_api to Py3.12

* Bump requeuer to Py3.12

* Bump Lambda runtimes to Py3.12

* Install Py3.12 in GH Actions

* Bump and relock flake8 & isort

* Bump boto3 and psycopg2 to supported versions

* Bump moto and update decorator usage for v5 (only mock_aws now)

* Upgrade alembic_migration

* Mocked SQS queue name is unique per test

* Update alembic_migration use of moto^=5

* Try to fix missing urllib3 despite it being in lockfile

* Support CDK v2

* sigh downgrade to 3.11 because of aws/aws-cdk#30170

* Bump boto3/botocore + requests

* fix lint

* Bump tox to run on py311 for integration tests

* Remove deprecated psycopg2-layer since db layer includes psycopg2-binary

* Bump reference to Py38 in integration tests workflow

* Bump versions mentioned in README

* Zap remaining use of CDK v1 in app_integration

* Fix CDK v2 use of aws_lambda_python (alpha)

* Fix missing downgrade from 3.12->3.11

* replace black/isort/flake8 with ruff

* use package.json as source of truth for cdk install

* Deploy stack w/ permissions boundary on resources

* Specify permissions boundary via full ARN

* Ensure permissions boundary is passed via .env

* cleanup references to isort/flake8/black

.github/workflows/ci.yml

@@ -15,10 +15,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.8.18
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.8.18
+          python-version: 3.11
 
       - name: Install Pipenv
         run: curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python
@@ -159,10 +159,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.8.18
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.8.18
+          python-version: 3.11
 
       - name: Install Pipenv
         run: curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python
@@ -347,11 +347,12 @@ jobs:
           touch .env
           echo OWNER="ci-developmentseed" >> .env
           echo IDENTIFIER=${{ steps.vars.outputs.branch }}-${{ steps.vars.outputs.commit }} >> .env
+          echo PERMISSIONS_BOUNDARY_ARN=${{ vars.PERMISSIONS_BOUNDARY_ARN }} >> .env
 
-      - name: Set up Python 3.8.18
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.8.18
+          python-version: 3.11
 
       - name: Setup up Node
         uses: actions/setup-node@v4
@@ -360,7 +361,7 @@ jobs:
 
       - name: Install AWS CDK
         run: |
-          npm install -g aws-cdk
+          npm install
 
       - name: Install Pipenv
         run: curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python

.github/workflows/deploy-on-release.yml

@@ -15,10 +15,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.8.18
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.8.18
+          python-version: 3.11
 
       - name: Install Pipenv
         run: curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python
@@ -196,12 +196,13 @@ jobs:
           USE_INTHUB2="TRUE"
           REMOVAL_POLICY_DESTROY="FALSE"
           UPLOAD_BUCKET="${{ secrets.UPLOAD_BUCKET }}"
+          PERMISSIONS_BOUNDARY_ARN="${{ vars.PERMISSIONS_BOUNDARY_ARN }}"
           EOF
 
-      - name: Set up Python 3.8.18
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.8.18
+          python-version: 3.11
 
       - name: Setup up Node
         uses: actions/setup-node@v4
@@ -210,7 +211,7 @@ jobs:
 
       - name: Install AWS CDK
         run: |
-          npm install -g aws-cdk
+          npm install
 
       - name: Install Pipenv
         run: curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python

Makefile

@@ -37,9 +37,8 @@ install:
 	$(MAKE) -C layers/db install
 
 lint:
-	pipenv run flake8 cdk/ integration_tests/
-	pipenv run isort --check-only --profile black cdk/ integration_tests/
-	pipenv run black --check --diff cdk/ integration_tests/
+	pipenv run ruff format --diff cdk/ integration_tests/
+	pipenv run ruff check cdk/ integration_tests/
 	$(MAKE) -C lambdas/link_fetcher lint
 	$(MAKE) -C lambdas/date_generator lint
 	$(MAKE) -C lambdas/downloader lint

Pipfile

@@ -6,9 +6,7 @@ name = "pypi"
 [packages]
 
 [dev-packages]
-flake8 = "==6.1.0"
-isort = "==5.12.0"
-black = "==23.11.0"
+ruff = "==0.7.1"
 
 [requires]
-python_version = "3.8"
+python_version = "3.11"

README.md

@@ -21,9 +21,9 @@ This project aims to provide a serverless implementation of the current [HLS S2
 
 To develop on this project, you should install:
 
-* NVM [Node Version Manager](https://github.com/nvm-sh/nvm) / Node 12
+* NVM [Node Version Manager](https://github.com/nvm-sh/nvm) / Node 18
 * [AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) - There is a `package.json` in the repository, it's recommended to run `npm install` in the repository root and make use of `npx <command>` rather than globally installing AWS CDK
-* [pyenv](https://github.com/pyenv/pyenv) / Python 3.8.6
+* [pyenv](https://github.com/pyenv/pyenv) / Python 3.11
 * [pipenv](https://github.com/pypa/pipenv)
 * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)
 * [OpenSSL](https://github.com/openssl/openssl) (For Postgres/DB work)
@@ -101,11 +101,11 @@ A `Makefile` is available in the root of the repository to abstract away commonl
 
 **`make lint`**
 
-> This will perform a dry run of `flake8`, `isort`, and `black` and let you know what issues were found
+> This will perform a dry run of `ruff` and let you know what issues were found
 
 **`make format`**
 
-> This will peform a run of `isort` and `black`, this **will** modify files if issues were found
+> This will perform a run of `ruff`, this **will** modify files if issues were found
 
 **`make diff`**
 

alembic_migration/Makefile

@@ -7,13 +7,13 @@ install:
 	pipenv install --dev
 
 lint:
-	pipenv run flake8 .
-	pipenv run isort --check-only --profile black .
-	pipenv run black --check --diff .
+	pipenv run ruff format --diff
+	pipenv run ruff check
 
 format:
-	pipenv run isort --profile black .
-	pipenv run black .
+	pipenv run ruff check --fix
+	pipenv run ruff check --select I --fix
+	pipenv run ruff format
 
 test:
 	pipenv run pytest -s --cov=. --cov-report term-missing tests/

alembic_migration/Pipfile

@@ -4,26 +4,24 @@ verify_ssl = true
 name = "pypi"
 
 [packages]
-boto3 = "==1.16.60"
+boto3 = "==1.35.44"
 retry = "==0.9.2"
 cfnresponse = "==1.1.1"
 alembic = "==1.12.1"
 sqlalchemy = "==1.4.0"
 
 [dev-packages]
 db = {editable = true, path = "./../layers/db"}
-flake8 = "==3.8.4"
-isort = "==5.6.4"
-black = "==23.11.0"
 pytest = "==7.4.3"
-moto = "==4.1.11"
-psycopg2 = "==2.8.6"
+moto = "==5.0.17"
+psycopg2 = "==2.9.10"
 pytest-docker = "==2.0.1"
 assertpy = "==1.1"
 pytest-cov = "==4.1.0"
+ruff = "==0.7.1"
 
 [requires]
-python_version = "3.8"
+python_version = "3.11"
 
 [pipenv]
 allow_prereleases = true