Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect and check test coverage stats #49

Merged
merged 8 commits into from
Oct 28, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Collect and check test coverage stats
ncoghlan committed Oct 28, 2024
commit add33d988d965cda635150b48206c3d109f2631e
51 changes: 50 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -32,7 +32,7 @@ permissions:
contents: read

jobs:
tox:
tests:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false # Always report results for all targets
@@ -131,3 +131,52 @@ jobs:
path: |
export/tests
retention-days: 3 # Just for debugging, don't need to keep these long term

- name: Upload coverage data
uses: actions/upload-artifact@v4
with:
name: coverage-data-${{ matrix.os }}-py${{ matrix.python-version }}
path: .coverage.*
include-hidden-files: true
if-no-files-found: ignore

# Coverage check based on https://hynek.me/articles/ditch-codecov-python/
coverage:
name: Combine & check coverage
if: always()
needs: tests
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
# Use latest Python, so it understands all syntax.
python-version: "3.13"
- uses: hynek/setup-cached-uv@v2

- uses: actions/download-artifact@v4
with:
pattern: coverage-data-*
merge-multiple: true

- name: Combine coverage & fail if it's <100%
run: |
uv tool install 'coverage[toml]'

coverage combine
coverage html --skip-covered --skip-empty

# Report and write to summary.
coverage report --format=markdown >> $GITHUB_STEP_SUMMARY

# Report again and fail if under 99%.
# (some error handling paths are not yet exercised in CI)
coverage report --fail-under=99

- name: Upload HTML report if check failed
uses: actions/upload-artifact@v4
with:
name: html-report
path: htmlcov
if: ${{ failure() }}
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -14,4 +14,4 @@ __pycache__
*.egg-info
*.dist-info
/dist/

.coverage*
18 changes: 18 additions & 0 deletions ci-constraints.txt
Original file line number Diff line number Diff line change
@@ -52,6 +52,24 @@ click==8.1.7 \
colorama==0.4.6 \
--hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \
--hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6
coverage==7.6.4 \
--hash=sha256:023bf8ee3ec6d35af9c1c6ccc1d18fa69afa1cb29eaac57cb064dbb262a517f9 \
--hash=sha256:1032e178b76a4e2b5b32e19d0fd0abbce4b58e77a1ca695820d10e491fa32b08 \
--hash=sha256:12394842a3a8affa3ba62b0d4ab7e9e210c5e366fbac3e8b2a68636fb19892c2 \
--hash=sha256:182e6cd5c040cec0a1c8d415a87b67ed01193ed9ad458ee427741c7d8513d963 \
--hash=sha256:29fc0f17b1d3fea332f8001d4558f8214af7f1d87a345f3a133c901d60347c73 \
--hash=sha256:2b6b4c83d8e8ea79f27ab80778c19bc037759aea298da4b56621f4474ffeb117 \
--hash=sha256:51b44306032045b383a7a8a2c13878de375117946d68dcb54308111f39775a25 \
--hash=sha256:5915fcdec0e54ee229926868e9b08586376cae1f5faa9bbaf8faf3561b393d52 \
--hash=sha256:73d2b73584446e66ee633eaad1a56aad577c077f46c35ca3283cd687b7715b0b \
--hash=sha256:8902dd6a30173d4ef09954bfcb24b5d7b5190cf14a43170e386979651e09ba19 \
--hash=sha256:a181e99301a0ae128493a24cfe5cfb5b488c4e0bf2f8702091473d033494d04f \
--hash=sha256:a6b1e54712ba3474f34b7ef7a41e65bd9037ad47916ccb1cc78769bae324c01a \
--hash=sha256:b0ac3d42cb51c4b12df9c5f0dd2f13a4f24f01943627120ec4d293c9181219ba \
--hash=sha256:b369ead6527d025a0fe7bd3864e46dbee3aa8f652d48df6174f8d0bac9e26e0e \
--hash=sha256:dacbc52de979f2823a819571f2e3a350a7e36b8cb7484cdb1e289bceaf35305f \
--hash=sha256:df57bdbeffe694e7842092c5e2e0bc80fff7f43379d465f932ef36f027179806 \
--hash=sha256:f3ddf056d3ebcf6ce47bdaf56142af51bb7fad09e4af310241e9db7a3a8022e1
dep-logic==0.4.9 \
--hash=sha256:06faa33814e5ff881922f644284a608d7da7946462760f710217d829ae864a0e \
--hash=sha256:5d455ea2a3da4fea2be6186d886905c57eeeebe3ea7fa967f599cb8e0f01d5c9
1 change: 1 addition & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -13,6 +13,7 @@ chardet==5.2.0
charset-normalizer==3.4.0
click==8.1.7
colorama==0.4.6
coverage[toml]==7.6.4
dep-logic==0.4.9
distlib==0.3.9
docutils==0.21.2
59 changes: 58 additions & 1 deletion pdm.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -65,6 +65,7 @@ dev = [
"ruff>=0.5.4",
"mypy>=1.11.0",
"pytest-subtests>=0.13.1",
"coverage[toml]>=7.6.4",
# Use exact pin for dev as runtime environments are sensitive to the exact pbs version
"pbs-installer==2024.10.10",
# Uses exact pin for dev as lock file regeneration is sensitive to the exact uv version
@@ -91,6 +92,17 @@ verbosity_assertions = 2
tmp_path_retention_policy = "failed"
tmp_path_retention_count = 1

[tool.coverage.run]
source = [
"venvstacks",
"tests/",
]
omit = [
# There is quite a bit of test support code that
# only runs to help diagnose failures in CI
"tests/support.py",
]

[tool.ruff]
# Assume Python 3.11
target-version = "py311"
13 changes: 13 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
@@ -47,6 +47,19 @@ Tests which take more than a few seconds to run should be marked as slow:
def test_locking_and_publishing(self) -> None:
...

The slow tests are part of the test suite because the fast tests only
get to just over 60% coverage of `venvstacks.stacks` and less than
20% coverage of `venvstacks.pack_venv`. The combined fast coverage
on a single platform (Linux for these numbers) is just over 60%.

When the slow tests are included, even running on a single platform,
those numbers increase to nearly 90% coverage of `venvstacks.stacks`,
nearly 70% coverage of `venvstacks.pack_venv`, and just under 90%
combined across the test suite and package source code.

When the results across all platforms are combined, the overall
numbers are TBD...


Marking tests with committed output files
-----------------------------------------
2 changes: 2 additions & 0 deletions tests/test_cli_invocation.py
Original file line number Diff line number Diff line change
@@ -150,6 +150,7 @@ def test_implicit_help(self, mocked_runner: MockedRunner) -> None:
assert result.exception is None, report_traceback(result.exception)
assert result.exit_code == 0

# TODO: Collect test coverage stats for this entry point invocation
@requires_venv("Entry point test")
def test_entry_point_help_raw(self) -> None:
expected_entry_point = Path(sys.executable).parent / "venvstacks"
@@ -168,6 +169,7 @@ def test_entry_point_help_raw(self) -> None:
assert result.returncode == 0
assert result.stdout is not None

# TODO: Collect test coverage stats for this entry point invocation
@requires_venv("Entry point test")
def test_entry_point_help(self) -> None:
expected_entry_point = Path(sys.executable).parent / "venvstacks"
13 changes: 12 additions & 1 deletion tox.ini
Original file line number Diff line number Diff line change
@@ -13,10 +13,20 @@ labels =
groups = dev
allowlist_externals = pytest
passenv =
CI
VENVSTACKS_*
commands =
pytest {posargs:-m "not slow"} tests/

[testenv:coverage]
groups = dev
allowlist_externals = coverage
passenv =
CI
VENVSTACKS_*
commands =
coverage run --parallel -m pytest {posargs} tests/

[testenv:format]
groups = dev
allowlist_externals = ruff
@@ -60,4 +70,5 @@ commands =
python =
3.11 = py3.11
3.12 = py3.12
3.13 = py3.13
# Collect coverage stats on the newest version
3.13 = coverage