chore: Add github workflow and script to check python depencies source #34549

salman2013 · 2024-04-19T14:52:32Z

Description: Add github action to run the python script from repo tools, which finds the python dependencies source and displays them if they belong to any third-party sources.

Resolves #33189

feanil

I think it makes sense for the workflow to live in this repo but the find_python_dependencies.py code should live in the repo_tools directory.

feanil · 2024-04-22T15:51:51Z

.github/workflows/check_python_dependencies.yml

+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.8'


Just start with 3.12 for this script.

I intentionally did that because i believe we have not added support of 3.12 in edx-platform yet,
if we try to update the python-version to 3.12 it results error

Traceback (most recent call last): File "/home/runner/work/edx-platform/edx-platform/scripts/find_python_dependencies.py", line 8, in <module> import requirements File "/opt/hostedtoolcache/Python/3.[12](https://github.com/openedx/edx-platform/actions/runs/8811123770/job/24184575317?pr=34549#step:10:13).3/x64/lib/python3.12/site-packages/requirements/__init__.py", line 19, in <module> from .parser import parse File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requirements/parser.py", line 23, in <module> from .requirement import Requirement File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requirements/requirement.py", line 24, in <module> from pkg_resources import Requirement as Req ModuleNotFoundError: No module named 'pkg_resources'

The reason for the above error is

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

https://docs.python.org/3/whatsnew/3.12.html

I have updated it.

feanil · 2024-04-22T15:53:06Z

.github/workflows/check_python_dependencies.yml

+
+      - name: Copy Python requirements file
+        run: |
+          for req_file in "requirements/edx/base.txt"; do


Do you think there will be more files in this list than just this one in the future? If so, can you make a list of one item that can be added to.

I have updated it.

feanil · 2024-04-22T16:24:39Z

scripts/find_python_dependencies.py

+$ python find_deps.py $OLIVE_DIRS
+"""
+
+import concurrent.futures


You can probably pare down these imports now that you've updated the code.

I have updated it.

feanil · 2024-04-22T16:27:04Z

scripts/find_python_dependencies.py

+        else:
+            print(f"Failed to retrieve data for package {package}. Status code:", response.status_code)    
+
+def process_directory():


Instead of processing a directory, why not just pass the path to a file in the main function, then we don't need to do all the directory work, just parse the requirements out of the file.

sure i have updated it.

feanil · 2024-05-09T12:42:56Z

.github/workflows/check_python_dependencies.yml

+      - name: Clone repo-tools repo
+        uses: actions/checkout@v4
+        with:
+          repository: salman2013/repo-tools
+          path: repo_tools
+          ref: salman/add-script-for-python-dependencies
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install setuptool
+        run: pip install setuptools
+
+      - name: Install requests module
+        run: pip install requests
+
+      - name: Install rich module
+        run: pip install rich
+
+      - name: Install requirements parser module
+        run: pip install requirements-parser  


Suggested change

- name: Clone repo-tools repo

uses: actions/checkout@v4

with:

repository: salman2013/repo-tools

path: repo_tools

ref: salman/add-script-for-python-dependencies

- name: Set up Python

uses: actions/setup-python@v5

with:

python-version: ${{ matrix.python-version }}

- name: Install setuptool

run: pip install setuptools

- name: Install requests module

run: pip install requests

- name: Install rich module

run: pip install rich

- name: Install requirements parser module

run: pip install requirements-parser

- name: Set up Python

uses: actions/setup-python@v5

with:

python-version: ${{ matrix.python-version }}

- name: Install repo-tools

run: pip install git+https://github.com/salman2013/repo-tools.git@salman/add-script-for-python-dependencies

@feanil There is one thing in this, i need to install requirements-parser and setuptools so there are two options should i add them here in CI or add them in repo-tool so that the script can find these packages. What do you suggest?

feanil · 2024-05-09T12:43:35Z

.github/workflows/check_python_dependencies.yml

+      - name: Make the script files executable
+        run: chmod +x repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py


Suggested change

- name: Make the script files executable

run: chmod +x repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py

This shouldn't be necessary if we're pip-installing repo-tools.

feanil · 2024-05-09T12:53:25Z

.github/workflows/check_python_dependencies.yml

+        run: chmod +x repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py
+
+      - name: Run Python script
+        run: python repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py requirements/edx/base.txt requirements/edx/testing.txt 


Suggested change

run: python repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py requirements/edx/base.txt requirements/edx/testing.txt

# Tracking ignored repos here: https://github.com/openedx/public-engineering/issues/174

run: python repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py requirements/edx/base.txt requirements/edx/testing.txt --ignore https://github.com/edx/codejail-includes --ignore https://github.com/edx/braze-client --ignore https://github.com/edx/edx-name-affirmation --ignore https://github.com/mitodl/edx-sga --ignore https://github.com/edx/token-utils --ignore https://github.com/edx/codejail-includes --ignore https://github.com/edx/braze-client --ignore https://github.com/edx/edx-name-affirmation --ignore https://github.com/mitodl/edx-sga --ignore https://github.com/edx/token-utils

I've made sure that all these repos are part of a tickte to further investigate and clean them up here: openedx/public-engineering#174 so we can ignore them for now and prevent future additions, we'll make sure to remove the repos from this list as they get removed from edx-platform.

feanil · 2024-05-17T17:04:31Z

.github/workflows/check_python_dependencies.yml

+        run: pip install setuptools  
+
+      - name: Run Python script
+        run: find_python_dependencies --dirs requirements/edx/base.txt --dirs requirements/edx/testing.txt --ignore https://github.com/edx/codejail-includes --ignore https://github.com/edx/braze-client --ignore https://github.com/edx/edx-name-affirmation --ignore https://github.com/mitodl/edx-sga --ignore https://github.com/edx/token-utils --ignore https://github.com/edx/codejail-includes --ignore https://github.com/edx/braze-client --ignore https://github.com/edx/edx-name-affirmation --ignore https://github.com/mitodl/edx-sga --ignore https://github.com/edx/token-utils


Nit, can you split this over multiple lines so that it's easier to read?

salman2013 · 2024-05-23T11:33:35Z

@feanil I think as per the documentation references, we should explicitly install the setuptool in environment where needed, like in our case I think in CI its best suited place instead of including it in repo-tool base.txt

python/cpython#95299
https://peps.python.org/pep-0632/

feanil · 2024-06-07T22:44:16Z

.github/workflows/check_python_dependencies.yml

+      - name: Install repo-tools
+        run: pip install git+https://github.com/salman2013/repo-tools.git@salman/add-script-for-python-dependencies
+
+      - name: Install requirements parser module
+        run: pip install requirements-parser  
+
+      - name: Install setuptool
+        run: pip install setuptools  


Suggested change

- name: Install repo-tools

run: pip install git+https://github.com/salman2013/repo-tools.git@salman/add-script-for-python-dependencies

- name: Install requirements parser module

run: pip install requirements-parser

- name: Install setuptool

run: pip install setuptools

- name: Install repo-tools

run: pip install edx-repo-tools[find_dependencies]

I think this is all you should need now.

@feanil repo-tool is requiring setuptool to run this github action on 3.12 so we have two options either we should keep it here like following step or install it in repo-tool but I believe it should be here where we need it as python has dropped its default support in 3.12

name: Install setuptool
run: pip install setuptools

I have updated other things and everything is working fine.

.github/workflows/check_python_dependencies.yml

edx-pipeline-bot · 2024-06-10T14:35:52Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot · 2024-06-10T14:56:42Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2024-06-10T14:56:42Z

2U Release Notice: This PR has been deployed to the edX production environment.

salman2013 marked this pull request as ready for review April 22, 2024 06:28

feanil requested changes Apr 22, 2024

View reviewed changes

salman2013 closed this Apr 24, 2024

salman2013 reopened this Apr 24, 2024

salman2013 closed this Apr 25, 2024

salman2013 reopened this Apr 25, 2024

salman2013 closed this Apr 25, 2024

salman2013 reopened this Apr 25, 2024

salman2013 requested a review from feanil May 6, 2024 11:09

feanil reviewed May 9, 2024

View reviewed changes

salman2013 closed this May 13, 2024

salman2013 reopened this May 13, 2024

salman2013 closed this May 13, 2024

salman2013 reopened this May 13, 2024

feanil reviewed May 17, 2024

View reviewed changes

salman2013 requested a review from a team as a code owner May 20, 2024 07:34

salman2013 closed this Jun 4, 2024

salman2013 reopened this Jun 4, 2024

salman2013 closed this Jun 4, 2024

salman2013 reopened this Jun 4, 2024

salman2013 closed this Jun 5, 2024

salman2013 reopened this Jun 5, 2024

salman2013 closed this Jun 5, 2024

salman2013 reopened this Jun 5, 2024

salman2013 closed this Jun 5, 2024

salman2013 reopened this Jun 5, 2024

chore: add workflow to find dependencies

87a7e87

salman2013 force-pushed the salman/add-ci-check-python-dependencies branch from 5628968 to 87a7e87 Compare June 5, 2024 12:05

salman2013 requested a review from feanil June 6, 2024 11:25

feanil reviewed Jun 7, 2024

View reviewed changes

.github/workflows/check_python_dependencies.yml Outdated Show resolved Hide resolved

salman2013 added 3 commits June 10, 2024 16:49

fix: fix the comments as per PR review

7ecf49e

fix: fix test cases

6a526bb

fix: code improvement as per PR reivew comment

35f6f50

salman2013 requested a review from feanil June 10, 2024 12:17

feanil approved these changes Jun 10, 2024

View reviewed changes

feanil merged commit 39a1369 into openedx:master Jun 10, 2024
46 checks passed

feanil mentioned this pull request Jun 20, 2024

Add CI check for dependencies from the edx GitHub org #33189

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Add github workflow and script to check python depencies source #34549

chore: Add github workflow and script to check python depencies source #34549

salman2013 commented Apr 19, 2024 •

edited by feanil

Loading

feanil left a comment

feanil Apr 22, 2024

salman2013 Apr 24, 2024

salman2013 May 6, 2024

feanil Apr 22, 2024

salman2013 May 6, 2024

feanil Apr 22, 2024

salman2013 May 6, 2024

feanil Apr 22, 2024

salman2013 Apr 24, 2024

feanil May 9, 2024

salman2013 May 13, 2024

feanil May 9, 2024

feanil May 9, 2024

feanil May 17, 2024

salman2013 commented May 23, 2024

feanil Jun 7, 2024

salman2013 Jun 10, 2024

salman2013 Jun 10, 2024

edx-pipeline-bot commented Jun 10, 2024

edx-pipeline-bot commented Jun 10, 2024

edx-pipeline-bot commented Jun 10, 2024

		- name: Make the script files executable
		run: chmod +x repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py

	run: python repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py requirements/edx/base.txt requirements/edx/testing.txt
	# Tracking ignored repos here: https://github.com/openedx/public-engineering/issues/174
	run: python repo_tools/edx_repo_tools/find_dependencies/find_python_dependencies.py requirements/edx/base.txt requirements/edx/testing.txt --ignore https://github.com/edx/codejail-includes --ignore https://github.com/edx/braze-client --ignore https://github.com/edx/edx-name-affirmation --ignore https://github.com/mitodl/edx-sga --ignore https://github.com/edx/token-utils --ignore https://github.com/edx/codejail-includes --ignore https://github.com/edx/braze-client --ignore https://github.com/edx/edx-name-affirmation --ignore https://github.com/mitodl/edx-sga --ignore https://github.com/edx/token-utils

chore: Add github workflow and script to check python depencies source #34549

chore: Add github workflow and script to check python depencies source #34549

Conversation

salman2013 commented Apr 19, 2024 • edited by feanil Loading

feanil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salman2013 commented May 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edx-pipeline-bot commented Jun 10, 2024

edx-pipeline-bot commented Jun 10, 2024

edx-pipeline-bot commented Jun 10, 2024

salman2013 commented Apr 19, 2024 •

edited by feanil

Loading