Should the coverage badge be mandatory?

[mateuszrzeszutek]

See the repository requirements:

• MUST have a README.md
  • MUST have badges on README.md with code coverage and build status

In Java it is not possible to collect coverage information for our custom instrumentations (technical reasons; the coverage tool cannot measure javaagent code), so no matter how many tests we have the number will always be 0. Same for our smoke tests.
Unit testing the agent module is impossible and does not make any sense anyway, the only modules that it's actually possible to collect coverage are custom and bootstrap. But these two modules are only a fraction of the code in this repo, presenting a coverage percentage based on just the two of them would be not telling the truth to whoever uses our javaagent (because we don't know the real coverage number).

Should we relax the requirement on the coverage badge to SHOULD?

[iNikem]

Technically, we can try to run our tests with both otel javaagent and jacoco. I believe we even did that in Plumbr, @vovencij @agoallikmaa ?

[mateuszrzeszutek]

Hmm, it's worth checking this out.

[vovencij]

Plumbr's acceptance tests (where we started an app-server with an agent and verified that everything is captured as expected) were definitely run WITHOUT jacoco. I'm not sure it even can handle all the relocated/obfuscated code.

[mateuszrzeszutek]

Hmm, that's bad news - some of our instrumentations can only be tested in smoke tests. And even if we could test them with instrumentation unit tests, these work on a similar principle - the code that's being tested is packaged into the javaagent.

[owais]

Should the coverage badge be mandatory?

Only if it is green :)

[mateuszrzeszutek]

OK, case closed then 🤣

[MrAlias]

If there are legitimate reasons, which it sounds like there are, why a requirement may not be satisfied and not cause compatibility or security issues in doing so, it seems that yes the requirement should be made a recommendation.

[MrAlias]

Opened #35 to address this.

[jkwatson]

I have some pretty strong philosophical issues with using code coverage metrics as anything more than just raw information, so I'd recommend against putting badges about them in the repository READMEs.