Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

toml install / requirements version boost #1242

Merged
merged 77 commits into from
Aug 28, 2023
Merged
Show file tree
Hide file tree
Changes from 64 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
4826ebd
bin execs replaced by toml scripts
WardDeb Aug 16, 2023
0c609de
replace version check for tests with toml version definition
WardDeb Aug 16, 2023
288ba70
remove _version file
WardDeb Aug 16, 2023
fb0a242
version checks with importlib
WardDeb Aug 16, 2023
45dea00
version check docs update to importlib
WardDeb Aug 16, 2023
6ca54b0
purge reqs & setup.py
WardDeb Aug 16, 2023
c986db9
init toml
WardDeb Aug 16, 2023
ce327bc
update cm for mpl > 3.7
WardDeb Aug 16, 2023
6352657
port nosetests to pytest
WardDeb Aug 16, 2023
2e7e4ac
escape sysargv check in case of pytests
WardDeb Aug 16, 2023
4b8d38c
deprecate nosetest in favor of pytest
WardDeb Aug 16, 2023
2cc7977
rollback heatmapperr imgs test until later
WardDeb Aug 16, 2023
df18b1d
flake8
WardDeb Aug 16, 2023
4294a3b
add conda env for workflows
WardDeb Aug 17, 2023
70004ab
switch test to micromamba
WardDeb Aug 17, 2023
427b0af
planemo tests to python 3.11, include samtools in test env
WardDeb Aug 17, 2023
e355cb1
shell defaults planemo test
WardDeb Aug 17, 2023
c273c83
pin allure version for planemo tests
WardDeb Aug 17, 2023
c9fc79d
pin python to 3.10 in test env
WardDeb Aug 17, 2023
f6e76d9
pin 3.10 in planemo wf as well
WardDeb Aug 17, 2023
15aeb57
test env channel priority into condarc
WardDeb Aug 17, 2023
1ce0424
planemo on 3.9, test on 3.11
WardDeb Aug 17, 2023
e4af3cd
complete rollback to 3.7 planemo with same deps as previous action
WardDeb Aug 17, 2023
911875e
pybw typo
WardDeb Aug 17, 2023
541e3bd
include pip install in planemo..
WardDeb Aug 17, 2023
c9b4fda
test with planemo test run python 3.10
WardDeb Aug 17, 2023
5232024
planemo tests with python 3.11 under galaxy
WardDeb Aug 17, 2023
d5fb5b6
py2bit as pip dep
WardDeb Aug 17, 2023
7729d15
python 3.10 for tests & planemo galaxy version
WardDeb Aug 18, 2023
65f4f11
plotcorr - parsed labels changed slightly due to mpl versions (assumed)
WardDeb Aug 18, 2023
4b4862c
typo in pltfingerprint comment, typo in bin script, include 1 extra s…
WardDeb Aug 18, 2023
3973e28
replace both expected heatmaps in planemo tests (mpl version boost ==…
WardDeb Aug 18, 2023
40b1e0c
replace plotprofile png in planemo test
WardDeb Aug 18, 2023
2d20885
test with NA in fp qualmets
WardDeb Aug 18, 2023
02d0d11
remove extra float + switch back to lower case nan qual metrics
WardDeb Aug 18, 2023
6fc8bae
include dist creation + test wheel install
WardDeb Aug 18, 2023
60e81f9
add name to test wheel wf
WardDeb Aug 18, 2023
250d4f7
name in diff place
WardDeb Aug 18, 2023
8c2d070
include build in building env
WardDeb Aug 18, 2023
755a084
purge checkout from testwheel
WardDeb Aug 18, 2023
2ee0a98
ship test wheels inside build-linux job
WardDeb Aug 18, 2023
c7374db
testwheels as separate job with artifact download
WardDeb Aug 18, 2023
4f8a447
add depends on
WardDeb Aug 18, 2023
9410a11
ls to see if download works
WardDeb Aug 18, 2023
d3d5604
list wdir after artifact dl
WardDeb Aug 18, 2023
e6bc30d
add checkout to test-wheels
WardDeb Aug 18, 2023
c836875
whl in dist folder
WardDeb Aug 18, 2023
8bcb173
tilde in pip install
WardDeb Aug 18, 2023
b84e4ed
importlib toml req for python 3.7
WardDeb Aug 19, 2023
2efc3f2
importlib escapes for python 3.7
WardDeb Aug 19, 2023
da21abe
flake fix try except
WardDeb Aug 19, 2023
a1eb555
test pypi build with micromamba
WardDeb Aug 19, 2023
e27b908
pypi push on tag
WardDeb Aug 19, 2023
9c2a86d
include pytest in test-wheels
WardDeb Aug 21, 2023
a11eedc
rounding qual metrics fingerprint galaxy test
WardDeb Aug 21, 2023
09a7a95
set version to 3.5.3
WardDeb Aug 21, 2023
1b3ae45
revert back to fingerprint qual metrics
WardDeb Aug 21, 2023
6b4ad36
rollback galaxy again
WardDeb Aug 21, 2023
f0ad1f0
update installation instructions doc
WardDeb Aug 21, 2023
e5a051d
qual metrics 14
WardDeb Aug 21, 2023
d7543b1
usage and -h for all executables
WardDeb Aug 22, 2023
3a5279c
flakes
WardDeb Aug 22, 2023
fff9904
switch pytest to -v
WardDeb Aug 22, 2023
ac43356
switch pytest to -v in init build as well
WardDeb Aug 22, 2023
70c8274
Update .github/workflows/planemo.yml
WardDeb Aug 24, 2023
8d3c964
python >=3.7, combine env/planemo envs, planemo test galaxy 23.1
WardDeb Aug 24, 2023
b0496a7
force conda prefix to micromamba env dir on runner
WardDeb Aug 24, 2023
558442d
include pytests for 3.11
WardDeb Aug 24, 2023
ad57435
bring back no dependency resolution planemo
WardDeb Aug 24, 2023
8df4889
parse toml for tool tests + verify version
WardDeb Aug 25, 2023
bfda737
fstring tests
WardDeb Aug 25, 2023
21d1326
1 env for test/builds, pip planemo, python min version 3.7, drop allu…
WardDeb Aug 25, 2023
c238997
setup-miniconda runner instead of micromamba for planemo test
WardDeb Aug 25, 2023
b80278a
print planemo version in shell runner
WardDeb Aug 25, 2023
5d00cb5
flake fixes tests
WardDeb Aug 25, 2023
04b7a67
drop channelprio / planemo fingerprint back to 13 (again)
WardDeb Aug 25, 2023
ba71cbe
update changelog
WardDeb Aug 25, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/env.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
channels:
- conda-forge
- bioconda
dependencies:
- python = 3.10
WardDeb marked this conversation as resolved.
Show resolved Hide resolved
- flake8
- pytest
- twine
- pip
- pip:
- build
16 changes: 16 additions & 0 deletions .github/planemo.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
channels:
- conda-forge
- bioconda
dependencies:
- python = 3.10
- numpy
- scipy
- flake8
- pysam
- deeptoolsintervals
- pytest
- planemo
- samtools
- allure-python-commons==2.12.0
- py2bit
- pyBigWig
23 changes: 18 additions & 5 deletions .github/workflows/planemo.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
name: Planemo
on: [push, pull_request]

env:
GALAXY_BRANCH: release_22.05
WardDeb marked this conversation as resolved.
Show resolved Hide resolved

defaults:
run:
shell: bash -l {0}

jobs:
planemo_test:
name: Planemo test
Expand All @@ -10,12 +16,19 @@ jobs:
matrix:
chunk: [1, 2, 3]
steps:
- uses: actions/checkout@v1
- uses: "dpryan79/github-actions/@master"
- uses: actions/checkout@v3
- uses: mamba-org/setup-micromamba@main
with:
environment-file: .github/planemo.yml
cache-downloads: true
environment-name: foo
WardDeb marked this conversation as resolved.
Show resolved Hide resolved
- name: pip install
run: |
micromamba activate foo
pip install .
- name: planemo
run: |
source activate foo
conda update -c conda-forge -c bioconda samtools
micromamba activate foo
./.planemo.sh ${{ matrix.chunk }} ${{ env.GALAXY_BRANCH }}
- uses: actions/upload-artifact@v3
with:
Expand All @@ -27,7 +40,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.7']
python-version: ['3.10']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could test multiple versions if we like.
Historically we tested for the lower bound. But we could test for lower and upper-supported python versions.

3.8 and 3.11?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've included 3.7 & 3.11, though I'm not sure how relevant it is at this stage (planemo_combine_outputs): The tests have already ran at that time, right ?

steps:
- uses: actions/download-artifact@v3
with:
Expand Down
34 changes: 16 additions & 18 deletions .github/workflows/pypi.yml
Original file line number Diff line number Diff line change
@@ -1,35 +1,33 @@
name: pypi

on:
create:
tags:

defaults:
run:
shell: bash -l {0}

jobs:
pypi:
name: upload to pypi
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Setup conda
- uses: actions/checkout@v3
- uses: mamba-org/setup-micromamba@main
with:
environment-file: .github/env.yml
cache-downloads: true
environment-name: foo
WardDeb marked this conversation as resolved.
Show resolved Hide resolved
- name: build
run: |
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
hash -r
conda config --set always_yes yes --set changeps1 no
- name: create env
run: |
export PATH=$HOME/miniconda/bin:$PATH
conda create -n foo -q --yes -c conda-forge -c bioconda python=3.7 twine
- name: sdist
run: |
export PATH=$HOME/miniconda/bin:$PATH
source activate foo
micromamba activate foo
rm -f dist/*
python setup.py sdist
python -m build
- name: upload
env:
TWINE_USERNAME: "__token__"
TWINE_PASSWORD: ${{ secrets.pypi_password }}
run: |
export PATH=$HOME/miniconda/bin:$PATH
source activate foo
micromamba activate foo
twine upload dist/*
78 changes: 64 additions & 14 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,21 @@
name: Test
on: [push, pull_request]

defaults:
run:
shell: bash -l {0}

jobs:
check_versions_matches:
name: Check deeptools version matches galaxy tools
runs-on: ubuntu-latest
if: github.base_ref == 'master'
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v3
- name: Check path
run: find /home/runner/work/deepTools/deepTools -name "_version.py"
run: find /home/runner/work/deepTools/deepTools -name "pyproject.toml"
- name: Get Version of Deeptools
run: echo "deeptools_version=$(grep "__version__" /home/runner/work/deepTools/deepTools/deeptools/_version.py | awk '{print substr($NF, 2, length($NF) - 2)}')" >> $GITHUB_ENV
run: echo "deeptools_version=$(grep "version" /home/runner/work/deepTools/deepTools/pyproject.toml | awk '{print substr($NF, 2, length($NF) - 2)}')" >> $GITHUB_ENV
- name: Get Version of Galaxy tools
run: echo "galaxy_deeptools_version=$(grep "token.*TOOL_VERSION" /home/runner/work/deepTools/deepTools/galaxy/wrapper/deepTools_macros.xml | awk -F '>|<' '{print $3}')" >> $GITHUB_ENV
- name: Versions
Expand All @@ -35,32 +40,77 @@ jobs:
name: Test on Linux
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: "dpryan79/github-actions/@master"
- uses: actions/checkout@v3
- uses: mamba-org/setup-micromamba@main
with:
environment-file: .github/env.yml
cache-downloads: true
environment-name: foo
WardDeb marked this conversation as resolved.
Show resolved Hide resolved
- name: pip install
run: |
micromamba activate foo
pip install .
- name: PEP8
run: |
source activate foo
micromamba activate foo
flake8 . --exclude=.venv,.build,build --ignore=E501,F403,E402,F999,F405,E722,W504,W605
- name: Test deepTools
run: |
source activate foo
nosetests --with-doctest -sv deeptools
micromamba activate foo
pytest -v
- name: make an artifact
run: |
source activate foo
micromamba activate foo
rm -f dist/*
python setup.py sdist
python -m build
- uses: actions/upload-artifact@master
with:
name: "Dist files"
path: "dist"
test-wheels:
name: test wheel
runs-on: ubuntu-latest
needs: build-linux
strategy:
matrix:
python-version: ['3.7','3.8','3.9','3.10']
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: "Dist files"
path: ~/dist/
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: install wheel
run: |
pip install ~/dist/*whl
deeptools -h
- name: pytest
run: |
pip install pytest
pytest -v
build-osx:
name: Test on OSX
runs-on: macOS-latest
steps:
- uses: actions/checkout@v1
- uses: "dpryan79/github-actions/@master"
- uses: actions/checkout@v3
- uses: mamba-org/setup-micromamba@main
with:
environment-file: .github/env.yml
cache-downloads: true
environment-name: foo
condarc: |
channels:
- conda-forge
- bioconda
- name: pip install
run: |
micromamba activate foo
pip install .
- name: Test deepTools
run: |
source activate foo
nosetests --with-doctest -sv deeptools
micromamba activate foo
pytest -v
2 changes: 1 addition & 1 deletion .planemo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,6 @@ else
fi

planemo lint ${wrappers}
planemo test --no_dependency_resolution --galaxy_branch $2 --install_galaxy ${wrappers} 2>&1
planemo test --no_dependency_resolution --galaxy_python_version 3.10 --galaxy_branch $2 --install_galaxy ${wrappers} 2>&1
WardDeb marked this conversation as resolved.
Show resolved Hide resolved
mkdir upload
mv tool_test_output* upload/
12 changes: 0 additions & 12 deletions bin/alignmentSieve

This file was deleted.

12 changes: 0 additions & 12 deletions bin/bamCompare

This file was deleted.

12 changes: 0 additions & 12 deletions bin/bamCoverage

This file was deleted.

12 changes: 0 additions & 12 deletions bin/bamPEFragmentSize

This file was deleted.

12 changes: 0 additions & 12 deletions bin/bigwigAverage

This file was deleted.

12 changes: 0 additions & 12 deletions bin/bigwigCompare

This file was deleted.

12 changes: 0 additions & 12 deletions bin/computeGCBias

This file was deleted.

14 changes: 0 additions & 14 deletions bin/computeMatrix

This file was deleted.

12 changes: 0 additions & 12 deletions bin/computeMatrixOperations

This file was deleted.

12 changes: 0 additions & 12 deletions bin/correctGCBias

This file was deleted.

12 changes: 0 additions & 12 deletions bin/deeptools

This file was deleted.

Loading