Wrong mutation coverage reported

[mflorea]

Characteristics

• Issue Type: [bug]
• Reproducibility: [always]
• Severity: [major]
• Tool/Service/Component: [pitest-maven:1.4.10:mutationCoverage]
• Execution Environment: [Ubuntu 19.04]

Description

When I build xwiki-commons-diff-xml using:

> mvn clean install -Pquality -Dxwiki.pitest.skip=false -Djacoco.skip=true

I get:

Mutation score of 60 is below threshold of 96

The difference is quite significant. I checked the issues reported by Descartes and some of them seem to be wrong. For instance:

org.xwiki.diff.xml.internal.AbstractXMLDiffMarker
    applyPatch(org.w3c.dom.Element, org.xwiki.diff.Patch)
    This method is pseudo-tested.
    The body of this method can be removed and no test case fails.
    The following test cases execute the method:
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[14] src/test/resources/html/nestedDiffBlocks.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[11] src/test/resources/html/attributeDeletedInline.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[13] src/test/resources/html/childrenModifiedInline.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[20] src/test/resources/html/attributeInsertedBlock.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[10] src/test/resources/html/nodeIndexChange.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[4] src/test/resources/html/listItemInserted.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[3] src/test/resources/html/childrenDeletedInline.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[21] src/test/resources/html/childrenInsertedBlock.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[12] src/test/resources/html/attributeDeletedBlock.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[18] src/test/resources/html/childrenInsertedInline.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[17] src/test/resources/html/childrenModifiedBlock.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[1] src/test/resources/html/childrenDeletedBlock.test
        org.xwiki.diff.xml.internal.UnifiedHTMLDiffManagerTest.[8] src/test/resources/html/attributeInsertedInline.test

If I comment out the body of AbstractXMLDiffMarker#applyPatch(org.w3c.dom.Element, org.xwiki.diff.Patch) and run the tests I get lots of failures, including some from the test cases listed above (e.g. childrenDeletedInline.test). So it looks like pitest/descartes is detecting the test cases correctly but not the fact that they fail.

My test is a bit special because it uses:

@ParameterizedTest
@MethodSource("getTestFiles")

But this hasn't created problems before. For instance, at this commit https://github.com/xwiki/xwiki-commons/tree/300ad03da64dad2749267a70d692e2dd14429351/xwiki-commons-core/xwiki-commons-diff/xwiki-commons-diff-xml the reported mutation coverage was OK (96). Basically the issue occurred at this commit xwiki/xwiki-commons@d2101de#diff-7ddf5975d3bead4599efe91f451a4934 but I don't see what could have caused the significant drop in mutation coverage.

Other files and URLs

Relationships

Help on issue template

Preview to follow the link or open file .github/ISSUE_DOC.md

[vmassol]

@oscarlvp hi, let me start by wishing you a happy new year 2020! Plenty of research success to you :)

Is there any chance you could help us diagnose this issue?

Thanks a lot
-Vincent

[oscarlvp]

@vmassol best wishes for you too!
Yes, I have this issue in my priority list :)
As soon as I have an answer I'll let you know.

[oscarlvp]

@mflorea is the build information available from your CI? I would like to inspect the report produced by PIT to see if the test is not being executed.

[oscarlvp]

I checked here https://ci.xwiki.org/view/All/job/xwiki-commons_pitest/683/org.xwiki.commons$xwiki-commons-diff-xml/console, it seems that the build does not keep the report. I will check it locally.

[vmassol]

Thanks @oscarlvp . Let me know if you need my help somewhere.

[oscarlvp]

@mflorea @vmassol This issue seems to be solved in version 0.12 of pitest-junit5-plugin.
I ran the analysis on xwiki-commons-diff-xml with version 0.12 and got a 98% mutation score and applyPatch was not pseudo-tested.
Please update the pitest-junit5-plugin to the latest version and confirm if the issue is solved on your side.

[vmassol]

Thanks @oscarlvp . Actually I've upgraded to 0.12 yesterday but because I was testing if #117 would be fixed by it (it's not unfortunately). I didn't think about testing this issue. I'll try to do that tomorrow. Thanks a lot for investigating it!

If you have some energy left, #117 is an even bigger issue for us ATM. We suddenly have plenty of test modules (junit5 ones probably) suddenly failing pitest/descartes. Could be a regression somewhere.

Thanks!

[tmortagne]

@oscarlvp the build errors you are reporting suggest you are building with a not very up to date version of xwiki-commons

Also I just reverted reflections version because of a regressions in it so better pulling again.

[oscarlvp]

@tmortagne Sorry, the comment before was meant to be in #117

[tmortagne]

OK

[vmassol]

I confirm that it works fine with version 0.12. Thanks @oscarlvp !