Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CI/CD pipelines Part 1 #2577

Merged
merged 17 commits into from
Nov 3, 2024
Merged

Update CI/CD pipelines Part 1 #2577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
df24b59
reduce tests to failing macos
dennisbader Nov 2, 2024
7a0f843
retry
dennisbader Nov 2, 2024
b70594a
retry
dennisbader Nov 2, 2024
cc66d1e
retry
dennisbader Nov 2, 2024
000f480
add back cache file
dennisbader Nov 2, 2024
b004399
add comment
dennisbader Nov 2, 2024
ceb7232
try different caching
dennisbader Nov 2, 2024
79733a3
fix bug
dennisbader Nov 2, 2024
7a78674
remove venv
dennisbader Nov 2, 2024
23bfaf5
add back codecov upload and ubuntu os
dennisbader Nov 2, 2024
b0fb3e0
add linting, docs, and example checks
dennisbader Nov 2, 2024
1a698f0
fix papermill missing
dennisbader Nov 2, 2024
5f7db95
change to exmaples workdir for example checks
dennisbader Nov 2, 2024
1774468
improve caching, update merge workflow
dennisbader Nov 3, 2024
79807f1
update merge workflow to use cache
dennisbader Nov 3, 2024
993b579
add update cache step whenever a branch is merged to master
dennisbader Nov 3, 2024
d5db286
Merge branch 'master' into fix/mac_cicd
dennisbader Nov 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 86 additions & 87 deletions .github/workflows/develop.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,30 +9,23 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: "1. Clone repository"
uses: actions/checkout@v2
- name: "Clone repository"
uses: actions/checkout@v4

- name: "2. Set up Python 3.9"
uses: actions/setup-python@v1
- name: "Set up Python 3.9"
uses: actions/setup-python@v5
with:
python-version: '3.9'

# downloading gradle multiple times in parallel can yield to connection errors
- name: "3. Cache gradle distribution"
uses: actions/cache@v2
with:
path: ~/.gradle/wrapper/dists
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties') }}

- name: "3.1 Cache gradle packages"
uses: actions/cache@v2
with:
path: ~/.gradle/caches
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'build.gradle') }}
- name: "Install Dev Dependencies"
run: |
python -m pip install --upgrade pip
pip install -r requirements/dev.txt

- name: "4. Lint"
- name: "Lint"
run: |
./gradlew lint
pre-commit run --all-files


tests:
runs-on: ${{ matrix.os }}
Expand All @@ -43,47 +36,39 @@ jobs:
flavour: ['all']

steps:
- name: "1. Clone repository"
uses: actions/checkout@v2
- name: "Clone repository"
uses: actions/checkout@v4

- name: "2. Set up Python ${{ matrix.python-version }}"
uses: actions/setup-python@v1
- name: "Set up Python ${{ matrix.python-version }}"
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

# downloading gradle multiple times in parallel can yield to connection errors
- name: "3. Cache gradle distribution"
uses: actions/cache@v2
- name: "Cache python environment"
uses: actions/cache@v4
id: pythonenv-cache
with:
path: ~/.gradle/wrapper/dists
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties') }}
path: ${{ env.pythonLocation }}
key: ${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('requirements/*.txt') }}

- name: "3.1 Cache gradle packages"
uses: actions/cache@v2
with:
path: ~/.gradle/caches
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'build.gradle') }}
- name: "Setup Pip"
run: |
python -m pip install --upgrade pip

- name: "4. Setup pip"
- name: "Install Dependencies"
run: |
./gradlew installPipLatest
# install latest dependencies (potentially updating cached dependencies)
pip install -U -r requirements/dev-all.txt

- name: "5. Install libomp (for LightGBM)"
- name: "Install libomp (for LightGBM)"
run: |
./.github/scripts/libomp-${{ runner.os }}.sh

- name: "6. Attach cache for pip"
uses: actions/cache@v1
id: cache
with:
path: ~/.cache/pip
key: tests-${{ runner.os }}-${{ matrix.python-version }}-pip-${{ hashFiles('requirements-latest.txt') }}

- name: "7. Tests"
- name: "Run tests"
run: |
./gradlew "test_${{matrix.flavour}}"
pytest --durations=50 --cov=darts --cov-config=.coveragerc --cov-report=xml darts/tests

- name: "8. Codecov upload"
- name: "Codecov upload"
if: ${{ matrix.flavour == 'all' }}
uses: codecov/codecov-action@v4
with:
Expand All @@ -93,73 +78,87 @@ jobs:
docs:
runs-on: ubuntu-latest
steps:
- name: "1. Clone repository"
uses: actions/checkout@v2
- name: "Clone repository"
uses: actions/checkout@v4

- name: "2. Set up Python 3.9"
uses: actions/setup-python@v1
- name: "Set up Python 3.9"
uses: actions/setup-python@v5
with:
python-version: '3.9'

- name: "3. Install pandoc"
# only restore cache but do not upload
- name: "Restore cached python environment"
uses: actions/cache/restore@v4
id: pythonenv-cache
with:
path: ${{ env.pythonLocation }}
key: ${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('requirements/*.txt') }}

- name: "Install pandoc"
run: |
sudo apt-get install -y pandoc

# downloading gradle multiple times in parallel can yield to connection errors
- name: "4. Cache gradle distribution"
uses: actions/cache@v2
with:
path: ~/.gradle/wrapper/dists
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties') }}
- name: "Setup Pip"
run: |
python -m pip install --upgrade pip

- name: "4.1 Cache gradle packages"
uses: actions/cache@v2
with:
path: ~/.gradle/caches
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'build.gradle') }}
- name: "Install Dependencies"
run: |
# install latest dependencies (potentially updating cached dependencies)
pip install -U -r requirements/dev-all.txt

- name: "5. Setup pip"
- name: "Install libomp (for LightGBM)"
run: |
./gradlew setupPip
./.github/scripts/libomp-${{ runner.os }}.sh

- name: "6. Attach cache for pip"
uses: actions/cache@v1
id: cache
with:
path: ~/.cache/pip
key: tests-${{ runner.os }}-3.9-pip-${{ hashFiles('requirements/core.txt', 'requirements/release.txt') }}
- name: "Install Locally"
run: |
pip install .

- name: "7. Build docs"
- name: "Build docs"
run: |
./gradlew buildDocs
make --directory ./docs build-all-docs

check-examples:
runs-on: ubuntu-latest
strategy:
matrix:
example-name: [03-FFT-examples.ipynb, 04-RNN-examples.ipynb, 00-quickstart.ipynb, 02-data-processing.ipynb, 01-multi-time-series-and-covariates.ipynb]
steps:
- name: "1. Clone repository"
uses: actions/checkout@v2
- name: "Clone repository"
uses: actions/checkout@v4

- name: "2. Set up Python 3.9"
uses: actions/setup-python@v1
- name: "Set up Python 3.9"
uses: actions/setup-python@v5
with:
python-version: '3.9'

# downloading gradle multiple times in parallel can yield to connection errors
- name: "3. Cache gradle distribution"
uses: actions/cache@v2
# only restore cache but do not upload
- name: "Restore cached python environment"
uses: actions/cache/restore@v4
id: pythonenv-cache
with:
path: ~/.gradle/wrapper/dists
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties') }}
path: ${{ env.pythonLocation }}
key: ${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('requirements/*.txt') }}

- name: "3.1 Cache gradle packages"
uses: actions/cache@v2
with:
path: ~/.gradle/caches
key: tests-${{ runner.os }}-gradle-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'build.gradle') }}
- name: "Setup Pip"
run: |
python -m pip install --upgrade pip

- name: "Install Dependencies"
run: |
# install latest dependencies (potentially updating cached dependencies)
pip install -U -r requirements/dev-all.txt

- name: "Install libomp (for LightGBM)"
run: |
./.github/scripts/libomp-${{ runner.os }}.sh

- name: "Install Locally"
run: |
pip install .

- name: "4. Run examples ${{matrix.example-name}}"
- name: "Run example ${{matrix.example-name}}"
working-directory: ./examples
run: |
./gradlew checkExample -PexampleName=${{matrix.example-name}}
papermill ${{matrix.example-name}} ${{matrix.example-name}}
Loading
Loading