[AWSX-537] Support python version 3.10 #719

ge0Aja · 2023-12-07T16:23:40Z

What does this PR do?

Support python version 3.10

Motivation

Support request

Testing Guidelines

Additional Notes

This PR adds support for python version 3.10 which requires a higher version for datadog-lambda package.
The usage of the higher version for that package adds a new field DD-API-KEY in the header section of the of the events response forwarded by the function. It also removes the API key string from the path parameter.

Types of changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog
This PR passes the integration tests (ask a Datadog member to run the tests)
This PR passes the unit tests
This PR passes the installation tests (ask a Datadog member to run the tests)

ge0Aja · 2023-12-07T21:59:37Z

had to use python versions as strings for triggering GH actions
actions/runner#1989

tyrcho · 2023-12-08T13:13:42Z

aws/logs_monitoring/tools/integration_tests/external_lambda/serverless.yml

@@ -1,7 +1,7 @@
 service: forwarder-tests-external-lambda
 provider:
  name: aws
-  runtime: python2.7
+  runtime: python3.11


is this file really used ? I'm surprised in was on 2.X

caught that when I was doing the version changes, not sure if it is used.

tyrcho · 2023-12-08T13:15:58Z

aws/logs_monitoring/README.md

@@ -138,6 +138,10 @@ If you can't install the Forwarder using the provided CloudFormation template, y

 If you encounter issues upgrading to the latest version, check the Troubleshooting section.

+### Upgrade an older version to +3.97.0
+
+Since version 3.97.0 the Lambda function has been updated to require **Python 3.11**. If upgrading an older forwarder installation to 3.97.0 or above, ensure the AWS Lambda function is configured to use Python 3.11


it seems from most other files that we actually support 3.10 and 3.11 which is not really clear in this doc

we can update the readme to include both versions, I followed the convention in the readme file

tyrcho · 2023-12-08T13:16:43Z

aws/logs_monitoring/tools/Dockerfile_bundle

@@ -19,4 +19,3 @@ RUN rm -rf ./botocore*
 # like `os.execl`, which cause security scans to fail for certain customers.
 # These files are not directly used by the Forwarder.
 RUN rm ./ddtrace/commands/ddtrace_run.py
-RUN rm ./decorator.py


why this change ?

file not found error was thrown during ITests, it seems it was added during a dependency version bump which created this file.
Now, it is not created, so rm is failing

tyrcho · 2023-12-08T13:17:44Z

aws/logs_monitoring/tools/add_new_region.sh

@@ -12,7 +12,7 @@ set -e

 OLD_REGION='us-east-1'

-PYTHON_VERSIONS_FOR_AWS_CLI=("python3.9")
+PYTHON_VERSIONS_FOR_AWS_CLI=("python3.11")


do we support only one version here ? why an array then ?

it seems we support only the highest version, not sure why

RaphaelAllier · 2023-12-08T14:49:56Z

.github/workflows/lint.yml

@@ -11,7 +11,7 @@ jobs:
      - name: Setup Python
        uses: actions/setup-python@v3
        with:
-          python-version: 3.9
+          python-version: '3.10'


Why do we add quotes here?

RaphaelAllier · 2023-12-08T14:50:08Z

.github/workflows/test.yml

@@ -8,7 +8,7 @@ jobs:
    strategy:
      max-parallel: 4
      matrix:
-        python-version: [3.8, 3.9]
+        python-version: ['3.9', '3.10']


Same question, why do we use quotes now ?

see actions/runner#1989, same for the above

RaphaelAllier · 2023-12-08T14:51:22Z

aws/logs_monitoring/README.md

@@ -138,6 +138,10 @@ If you can't install the Forwarder using the provided CloudFormation template, y

 If you encounter issues upgrading to the latest version, check the Troubleshooting section.

+### Upgrade an older version to +3.97.0


You need to update the cloudformation template we use to deploy the forwarder to ensure we use Python 3.10 as well

RaphaelAllier · 2023-12-08T14:51:54Z

aws/logs_monitoring/setup.py

-    python_requires=">=3.7, <3.10",
-    install_requires=["datadog-lambda==3.39.0", "requests-futures==1.0.0"],
+    python_requires=">=3.9, <3.11",
+    install_requires=["datadog-lambda==4.77.0", "requests-futures==1.0.0"],


Where's that version change coming from ?

required dependency version update to support versions 3.10 -> 3.11

RaphaelAllier · 2023-12-08T14:52:56Z

aws/logs_monitoring/tools/integration_tests/recorder/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.9
+FROM python:3.11


This still mentions 3.11 (PR title states 3.10)

RaphaelAllier · 2023-12-08T14:53:36Z

aws/logs_monitoring/tools/publish_layers.sh

@@ -13,7 +13,7 @@ set -e
 # Makes sure any subprocesses will be terminated with this process
 trap "pkill -P $$; exit 1;" INT

-PYTHON_VERSIONS_FOR_AWS_CLI=("python3.9")
+PYTHON_VERSIONS_FOR_AWS_CLI=("python3.11")


Still mentions 3.11 vs 3.10 in title of the PR

Python 3.10 version is seen as 3.1 in github actions. See actions/runner#1989

Support Python version 3.10 for first version bump before adding support for Python 3.11

- Use the correct python runtime for tester and recorder docker images - Add a health check for the recorder service in the docker-compose file - Update service dependency - Pass the correct python runtime version when bundling the forwarder code for the Itests - Remove unused docker files as we are using the image version in the compose file

github-actions bot added the aws label Dec 7, 2023

ge0Aja force-pushed the georgi.ajaeiya/py310311 branch from 8102d49 to d11a039 Compare December 7, 2023 17:10

ge0Aja marked this pull request as ready for review December 7, 2023 22:00

ge0Aja requested a review from a team December 8, 2023 13:08

ge0Aja changed the title ~~Support python versions 3.10, 3.11~~ [AWSX-537] Support python versions 3.10, 3.11 Dec 8, 2023

tyrcho reviewed Dec 8, 2023

View reviewed changes

ge0Aja changed the title ~~[AWSX-537] Support python versions 3.10, 3.11~~ [AWSX-537] Support python versions 3.10 Dec 8, 2023

ge0Aja changed the title ~~[AWSX-537] Support python versions 3.10~~ [AWSX-537] Support python version 3.10 Dec 8, 2023

RaphaelAllier reviewed Dec 8, 2023

View reviewed changes

ge0Aja added 10 commits December 12, 2023 10:29

Support python versions 3.10, 3.11

f2aaad8

Remove a wrong rm statement from Dockerfile

5fbeff9

Fix python versions

9920d94

Update Itests

de7d4bb

Use strings for python versions in in yaml config files.

adb1b50

Python 3.10 version is seen as 3.1 in github actions. See actions/runner#1989

cleanup comments

9427b35

Update readme

9db34c5

Support Python 3.10

423e358

Support Python version 3.10 for first version bump before adding support for Python 3.11

Update python version range

d6c4663

Fix CLI & Cloudformation python versions

b7a1e2c

ge0Aja force-pushed the georgi.ajaeiya/py310311 branch from eaad826 to b7a1e2c Compare December 12, 2023 09:38

RaphaelAllier approved these changes Dec 13, 2023

View reviewed changes

ge0Aja merged commit 87e0a1a into master Dec 14, 2023
12 checks passed

ge0Aja deleted the georgi.ajaeiya/py310311 branch December 14, 2023 08:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AWSX-537] Support python version 3.10 #719

[AWSX-537] Support python version 3.10 #719

ge0Aja commented Dec 7, 2023 •

edited

Loading

ge0Aja commented Dec 7, 2023

tyrcho Dec 8, 2023

ge0Aja Dec 8, 2023

tyrcho Dec 8, 2023

ge0Aja Dec 8, 2023

tyrcho Dec 8, 2023

ge0Aja Dec 8, 2023

tyrcho Dec 8, 2023

ge0Aja Dec 8, 2023

RaphaelAllier Dec 8, 2023

RaphaelAllier Dec 8, 2023

ge0Aja Dec 8, 2023

RaphaelAllier Dec 8, 2023

RaphaelAllier Dec 8, 2023

ge0Aja Dec 8, 2023

RaphaelAllier Dec 8, 2023

RaphaelAllier Dec 8, 2023

		@@ -138,6 +138,10 @@ If you can't install the Forwarder using the provided CloudFormation template, y

		If you encounter issues upgrading to the latest version, check the Troubleshooting section.

		### Upgrade an older version to +3.97.0

[AWSX-537] Support python version 3.10 #719

[AWSX-537] Support python version 3.10 #719

Conversation

ge0Aja commented Dec 7, 2023 • edited Loading

What does this PR do?

Motivation

Testing Guidelines

Additional Notes

Types of changes

Check all that apply

ge0Aja commented Dec 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ge0Aja commented Dec 7, 2023 •

edited

Loading