Skip to content

Commit

Permalink
test: Use coverage over pytest-cov (#2054)
Browse files Browse the repository at this point in the history
* Remove 'pytest-cov' in favor of 'coverage[toml]' in 'test' extra.
* Remove .coveragerc in favor of setting coverage options in pyproject.toml.
* Make running without coverage the default for the tests nox session. Running under coverage
  can be enabled with an optional 'coverage' positional argument.
* Add coverage session to nox that displays a coverage report sorted by ascending coverage
  and generates XML and HTML versions of the report.
* Update CI workflows to run with coverage only if needed. 
   - Apply stylistic changes to the order of arguments to pytest to match across the codebase.
* Add steps to CI to generate the stdout coverage report and XML report after each segment
  of the tests.
* Add section to development docs on coverage.
  • Loading branch information
matthewfeickert authored Nov 2, 2022
1 parent 81a429e commit 2998959
Show file tree
Hide file tree
Showing 9 changed files with 105 additions and 28 deletions.
3 changes: 0 additions & 3 deletions .coveragerc

This file was deleted.

26 changes: 21 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,19 @@ jobs:
- name: List installed Python packages
run: python -m pip list

- name: Test with pytest
- name: Test with pytest and coverage
run: |
pytest --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
coverage run --module pytest --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py
- name: Launch a tmate session if tests fail
if: failure() && github.event_name == 'workflow_dispatch'
uses: mxschmitt/action-tmate@v3

- name: Coverage report for core project
run: |
coverage report
coverage xml
# Report coverage for oldest and newest Python tested to deal with version differences
- name: Report core project coverage with Codecov
if: >-
Expand All @@ -67,7 +72,12 @@ jobs:

- name: Test Contrib module with pytest
run: |
pytest tests/contrib --mpl --mpl-baseline-path tests/contrib/baseline
coverage run --append --module pytest tests/contrib --mpl --mpl-baseline-path tests/contrib/baseline
- name: Coverage report with contrib
run: |
coverage report
coverage xml
- name: Report contrib coverage with Codecov
if: github.event_name != 'schedule' && matrix.python-version == '3.10' && matrix.os == 'ubuntu-latest'
Expand All @@ -78,13 +88,19 @@ jobs:

- name: Test docstring examples with doctest
if: matrix.python-version == '3.10'
run: pytest src/ README.rst
run: coverage run --data-file=.coverage-doctest --module pytest src/ README.rst

- name: Coverage report for doctest only
if: matrix.python-version == '3.10'
run: |
coverage report --data-file=.coverage-doctest
coverage xml --data-file=.coverage-doctest -o doctest-coverage.xml
- name: Report doctest coverage with Codecov
if: github.event_name != 'schedule' && matrix.python-version == '3.10' && matrix.os == 'ubuntu-latest'
uses: codecov/codecov-action@v3
with:
files: ./coverage.xml
files: doctest-coverage.xml
flags: doctest

- name: Run benchmarks
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/dependencies-head.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- name: Test with pytest
run: |
pytest --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
pytest --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py
scipy:

Expand Down Expand Up @@ -65,7 +65,7 @@ jobs:
- name: Test with pytest
run: |
pytest --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
pytest --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py
iminuit:

Expand All @@ -91,7 +91,7 @@ jobs:
python -m pip list
- name: Test with pytest
run: |
pytest --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
pytest --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py
uproot4:

Expand All @@ -116,7 +116,7 @@ jobs:
python -m pip list
- name: Test with pytest
run: |
pytest --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
pytest --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py
matplotlib:

Expand Down Expand Up @@ -177,4 +177,4 @@ jobs:
python -m pip list
- name: Test with pytest
run: |
pytest --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
pytest --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py
2 changes: 1 addition & 1 deletion .github/workflows/lower-bound-requirements.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,4 +39,4 @@ jobs:
# free. Though still show warnings by setting warning control to 'default'.
export PYTHONWARNINGS='default'
# Run on tests/ to skip doctests of src given examples are for latest APIs
pytest --override-ini filterwarnings= --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py tests/
pytest --override-ini filterwarnings= --ignore tests/contrib --ignore tests/benchmarks --ignore tests/test_notebooks.py tests/
2 changes: 1 addition & 1 deletion .github/workflows/release_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install --pre pyhf[backends,xmlio]
python -m pip install pytest pytest-cov
python -m pip install pytest
python -m pip list
- name: Canary test public API
Expand Down
30 changes: 30 additions & 0 deletions docs/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,36 @@ For example, to run ``doctest`` on the JAX backend run
pytest src/pyhf/tensor/jax_backend.py
Coverage
~~~~~~~~

To measure coverage for the codebase run the tests under ``coverage`` with

.. code-block:: console
coverage run --module pytest
or pass ``coverage`` as a positional argument to the ``nox`` ``tests`` session

.. code-block:: console
nox --session tests --python 3.10 -- coverage
Coverage Report
^^^^^^^^^^^^^^^

To generate a coverage report after running the tests under ``coverage`` run

.. code-block:: console
coverage
or to also generate XML and HTML versions of the report run the coverage ``nox`` session

.. code-block:: console
nox --session coverage
Documentation
-------------

Expand Down
43 changes: 36 additions & 7 deletions noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,25 @@ def tests(session):
Examples:
$ nox --session tests --python 3.10
$ nox --session tests --python 3.10 -- contrib
$ nox --session tests --python 3.10 -- tests/test_tensor.py
$ nox --session tests --python 3.10 -- contrib # run the contrib module tests
$ nox --session tests --python 3.10 -- tests/test_tensor.py # run specific tests
$ nox --session tests --python 3.10 -- coverage # run with coverage but slower
"""
session.install("--upgrade", "--editable", ".[test]")
session.install("--upgrade", "pytest")

# Allow tests to be run with coverage
if "coverage" in session.posargs:
runner_commands = ["coverage", "run", "--append", "--module", "pytest"]
session.posargs.pop(session.posargs.index("coverage"))
session.install("--upgrade", "coverage[toml]")
else:
runner_commands = ["pytest"]

def _contrib(session):
if sys.platform.startswith("linux"):
session.run(
"pytest",
*runner_commands,
"tests/contrib",
"--mpl",
"--mpl-baseline-path",
Expand All @@ -58,21 +67,41 @@ def _contrib(session):
return

if session.posargs:
session.run("pytest", *session.posargs)
session.run(*runner_commands, *session.posargs)
else:
# defaults
default_runner_commands = runner_commands.copy()
if "--append" in default_runner_commands:
default_runner_commands.pop(default_runner_commands.index("--append"))
session.run(
"pytest",
"--ignore",
"tests/benchmarks/",
*default_runner_commands,
"--ignore",
"tests/contrib",
"--ignore",
"tests/benchmarks",
"--ignore",
"tests/test_notebooks.py",
)
_contrib(session)


@nox.session(reuse_venv=True)
def coverage(session):
"""
Generate coverage report
"""
session.install("--upgrade", "pip")
session.install("--upgrade", "coverage[toml]")

session.run("coverage", "report")
session.run("coverage", "xml")
htmlcov_path = DIR / "htmlcov"
if htmlcov_path.exists():
session.log(f"rm -r {htmlcov_path}")
shutil.rmtree(htmlcov_path)
session.run("coverage", "html")


@nox.session(reuse_venv=True)
def regenerate(session):
"""
Expand Down
15 changes: 10 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,9 @@ minversion = "6.0"
xfail_strict = true
addopts = [
"-ra",
"--cov=pyhf",
"--cov-branch",
"--showlocals",
"--strict-markers",
"--strict-config",
"--cov-report=term-missing",
"--cov-report=xml",
"--cov-report=html",
"--doctest-modules",
"--doctest-glob='*.rst'",
]
Expand Down Expand Up @@ -93,6 +88,16 @@ filterwarnings = [
'ignore:Call to deprecated create function:DeprecationWarning', # protobuf via tensorflow
]

[tool.coverage.run]
source = ["pyhf"]
branch = true
omit = ["*/pyhf/typing.py"]

[tool.coverage.report]
precision = 1
sort = "cover"
show_missing = true

[tool.nbqa.mutate]
pyupgrade = 1

Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
+ [
'scikit-hep-testdata>=0.4.11',
'pytest>=6.0',
'pytest-cov>=2.5.1',
'coverage[toml]>=6.0.0',
'pytest-mock',
'requests-mock>=1.9.0',
'pytest-benchmark[histogram]',
Expand Down

0 comments on commit 2998959

Please sign in to comment.