Add coverage testing on Jenkins

[Pansysk75]

Runs coverage tests on Jenkins, and uploads results to Codacy.
Relies on fetching grcov utility.

[hkaiser]

Please ignore the cmake formatting issue, that one has trickled through from master.

[Pansysk75]

retest lsu

[StellarBot]

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	(=)	??	(=)

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-09-06T17:05:04+00:00
HPX Commit	dcb5415	0f53ba5
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Clustername	rostam	rostam
Envfile
Datetime	2023-05-10T14:50:18.616050-05:00	2023-09-06T12:15:03.169581-05:00

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	(=)

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-09-06T17:05:04+00:00
HPX Commit	dcb5415	0f53ba5
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Clustername	rostam	rostam
Envfile
Datetime	2023-05-10T14:52:35.047119-05:00	2023-09-06T12:17:14.408419-05:00

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	(=)	(=)	(=)
Stream Benchmark - Scale	(=)	-	=
Stream Benchmark - Triad	(=)	(=)	(=)
Stream Benchmark - Copy	(=)	-	(=)

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-09-06T17:05:04+00:00
HPX Commit	dcb5415	0f53ba5
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Clustername	rostam	rostam
Envfile
Datetime	2023-05-10T14:52:52.237641-05:00	2023-09-06T12:17:31.433601-05:00

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
+1.45%	∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (7849eef)	148709	124399	83.65%
Head commit (2b3fdde)	188085 (+39376)	160070 (+35671)	85.11% (+1.45%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6339)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

[Pansysk75]

Coverage summary from Codacy

...

Finally this worked :) I will disable this guy, we don't need a comment.

I'll add the ReadMe badge as well, and check whether I forgot to include some part of HPX in the build.

Currently, it runs for every PR, which takes 2 to 3 hours on Rostam (because it builds and runs all tests). I think that's fine for now, but we could get away only running it on master, to save resources.

Also, currently, it reports as failed if some build/run failed, but it will intentionally still attempt to get the coverage results.

[Pansysk75]

@hkaiser This is ready to merge, the only caveat being that things like APEX and CUDA (and probably others) aren't included in the tests that are currently being ran, so those parts of the codebase might not show up as "covered" by tests. Same goes for compile time tests.

I suggest merging this first so that we can a few runs and thus get a better image in the dashboard. We can tweak it later if needed.

Oh, also @hkaiser you'll need to disable the commenting because you are the admin in Codacy. It's in the settings tab, under "Integrations". Thanks!

[hkaiser]

@hkaiser This is ready to merge, the only caveat being that things like APEX and CUDA (and probably others) aren't included in the tests that are currently being ran, so those parts of the codebase might not show up as "covered" by tests. Same goes for compile time tests.

I suggest merging this first so that we can a few runs and thus get a better image in the dashboard. We can tweak it later if needed.

Thanks @Pansysk75! This is great.

Oh, also @hkaiser you'll need to disable the commenting because you are the admin in Codacy. It's in the settings tab, under "Integrations". Thanks!

Should we really disable the comments?

[hkaiser]

LGTM, thanks!

[hkaiser]

bors merge

[Pansysk75]

Thanks @Pansysk75! This is great.

My pleasure :))

Should we really disable the comments?

We can give them a chance, I don't mind at all. You can always toggle them directly from the dashboard.

[bors]

This PR was included in a batch that successfully built, but then failed to merge into master (it was a non-fast-forward update). It will be automatically retried.

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• Bors

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
+1.40%	∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (7849eef)	148709	124399	83.65%
Head commit (610f270)	188085 (+39376)	159969 (+35570)	85.05% (+1.40%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6339)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences