Codecov.io report doesn't match Coverage.py when branch coverage is enabled

[kinow]

Not sure if the same issue as #125, but I noticed that when I have branch coverage in coverage.py, the reported coverage in Codecov doesn't match (without coverage both work exactly the same).

The issue appears to be that Codecov doesn't compute branch-coverage the same way as Coverage.py.

This is the reported coverage without branch-coverage in both Coverage.py and Codecov (they match).

And this is the reported coverage with branch-coverage:

After comparing the reports with branch-coverage, it looks to me that Coverage.py looks at branches in the whole source code, including missed statements. While Codecov.io appears to ignore missed statements.

The file data_store_mgr.py, for example, has the exact same coverage in Codecov.io when branch analysis is enabled and when it is disabled.

But with coverage.py that number is different. And the reason is that there are missed branches in the missed statements. So coverage.py considers those branches as missed too, while Codecov.io appears to not take those into consideration - which means it reports the same as if there was no branch analysis enabled, as there are no missed branches.

It would make more sense, in my opinion, to include the missed branches of the missed statements too - as the user specifically said s/he is interested in branch-coverage.

Cheers
Bruno

[blueyed]

That should work in general.
Make sure that you are generating the report / coverage.xml using the same options. I recommend to generate it manually (using .coveragerc), and then specify it with codecov-python (-f coverage.xml).
Also coverage report -m or similar can be added to CI to have a report there for comparison.

[kinow]

That should work in general.
Make sure that you are generating the report / coverage.xml using the same options. I recommend to generate it manually (using .coveragerc), and then specify it with codecov-python (-f coverage.xml).
Also coverage report -m or similar can be added to CI to have a report there for comparison.

Hi @blueyed

It does work in general. But if you configure if to have branch coverage, manually or not, the coverage generated values do not match the values displayed in Codecov UI.

We have kept Codecov in the UI (where we are using nyc + mocha) where it's giving us the right values, but have disabled the Python reporting as developers were finding it confusing to have mismatching values reported locally and on Codecov.

We compared the report locally, CI by downloading the generated XML, and the final value reported in Codecov. Local and CI values include the right branching information, and report the same percentage values. What we noticed that Codecov was the one reporting different numbers.

We had migrated from coveralls before adding branching, but it's on our roadmap to look if coveralls or SonarQube do the same or not, or if Codecov fixed the reporting numbers.

Thanks
Bruno

[thomasrockhu]

@kinow, can you open a ticket under support in our community boards?

[blueyed]

@kinow FWIW I am using branch coverage for/with many projects, and codecov reports it properly.

I am not saying there might not be an issue somewhere of course. It still is more an issue with the server/backend then - so creating a support ticket is the way to go I think, yes.

[ssbarnea]

I am afraid that if you use pyproject.toml, this feature is not working. See #107 (comment)