twister: coverage: Data collection and reporting per-test instance

[golowanow]

Twister code coverage data collection and reporting improvements:

1. The new --coverage-split mode extend Twister code coverage operations to collect and report statistics on each test instance execution individually in addition to the default reporting mode of data aggregation for all the test instances in current scope of the Twister run.
   The split mode allows to identify precisely what amount of code coverage each test case provides and to analyze its contribution to the overall test plan's coverage. Each build directory will have its own coverage report and data files, so the overall disk space and the total execution time increase.
   The split mode reads the data files while the tests are running in parallel, so the report aggregation stage doesn't need to repeat the same at its turn.
   Another new --disable-coverage-aggregation option allows to execute only the --coverage-split mode if the summary statistics are not needed.

2. New Twister option --coverage-sections allows to select what coverage data to include into the twister.json file for its further analysis together with the test meta-data and test results.
   The choices are to select either the coverage summary, or the detailed branch coverage, ztest coverage, or all the coverage data collected. Also the coverage run 'status' and coverage 'tool' values are added with any non-default choice. The 'environment' top object has additional 'gcov_tool' proverty to keep the command line argument selection.
   Depending on --coverage-split and --disable-coverage-aggregation options, the coverage data is attached either to its test suite object or/and at the report's top level as the aggregated summary of the test plan execution scope.
   Currently this mode is fully supported for the default gcovr reporting tool only, as it is based on the GCOVR json reports. For lcov only coverage run status is reported.

All the new modes are disabled by default for compatibility.

3. Group Twister code coverage command line arguments for better usability. Add more details in descriptions.

Resolves #67058

[golowanow]

for example, with ./scripts/twister -vv -j 1 -p unit_testing --coverage --coverage-tool gcovr -T tests/unit --coverage-split --coverage-report summary:

twister.json

{
    "environment":{
        "os":"posix",
        "zephyr_version":"zephyr-v3.5.0-2849-g38e4b5f370df",
        "toolchain":"zephyr",
        "commit_date":"2023-12-09T22:30:12+01:00",
        "run_date":"2023-12-11T09:00:36+00:00",
        "gcov_tool":"gcov"
    },
    "testsuites":[
        {
            "name":"tests/unit/crc/utilities.crc",
            "arch":"unit",
            "platform":"unit_testing",
            "path":"tests/unit/crc",
            "run_id":"037ce42604eb9fb89aef5d37a15e7e2a",
            "runnable":true,
            "retries":0,
            "status":"passed",
            "execution_time":"0.01",
            "build_time":"2.55",
            "testcases":[
                {
                    "identifier":"utilities.crc.crc32c",
                    "execution_time":"0.00",
                    "status":"passed"
                },
.......
            "coverage":{
                "status":true,
                "tool":"gcovr",
                "summary":{
                    "branch_covered":121,
                    "branch_percent":48.0,
                    "branch_total":252,
                    "files":[
                        {
                            "branch_covered":0,
                            "branch_percent":null,
                            "branch_total":0,
                            "filename":"include/zephyr/sys/crc.h",
                            "function_covered":1,
                            "function_percent":100.0,
                            "function_total":1,
                            "line_covered":2,
                            "line_percent":100.0,
                            "line_total":2
                        },
.........
                    ],
                    "function_covered":38,
                    "function_percent":63.3,
                    "function_total":60,
                    "gcovr/summary_format_version":"0.5",
                    "line_covered":315,
                    "line_percent":55.6,
                    "line_total":567,
                    "root":"../../../../../.."
                }
            }
        },
.........
    ],
    "coverage":{
        "status":true,
        "tool":"gcovr",
        "summary":{
            "branch_covered":1067,
            "branch_percent":76.1,
            "branch_total":1403,
            "files":[
                {
                    "branch_covered":4,
                    "branch_percent":100.0,
                    "branch_total":4,
                    "filename":"include/zephyr/sys/cbprintf.h",
                    "function_covered":5,
                    "function_percent":100.0,
                    "function_total":5,
                    "line_covered":16,
                    "line_percent":100.0,
                    "line_total":16
                },
.........
            ],
            "function_covered":191,
            "function_percent":86.8,
            "function_total":220,
            "gcovr/summary_format_version":"0.5",
            "line_covered":2093,
            "line_percent":83.8,
            "line_total":2499,
            "root":".."
        }
    }
}
```

[golowanow]

@gchwier, @gopiotr, @hakehuang, @nashif, @PerMac up to your attention

[golowanow]

rebased to resolve conflicts with #66406

[hakehuang]

@golowanow , maybe change the PR title to enable json report with coverage for each file?

[golowanow]

@golowanow , maybe change the PR title to enable json report with coverage for each file?

That title will be misleading as gcovr coverage report already has details for each file, and now it is included into twister.json.
The example snippet above is to show the gcovr 'summary' mode results embedded.

[hakehuang]

LGTM, with some code suggestion on group coverage options. beside the conflict need to be resolved

[golowanow]

rebased to resolve conflicts with #66487 and with some minor improvements:

• command line arguments re-arranged;
• additional properties added to the json (to keep which gcov-tool and gcovr/lcov was used).

[golowanow]

rebased resolving merge conflict with #66898

[golowanow]

rebased resolving merge conflict with #66905

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[PerMac]

@nordic-piks Maybe you'd like to have a look at this?

[golowanow]

rebased to resolve merge conflicts (with #67129), some minor corrections and adjustments to twister tests.

@hakehuang, please refresh your +1

[golowanow]

@gchwier, @gopiotr, @nashif, @PerMac, any objections or comments on this PR ?
some more details are also in the RFC #67058

[katgiadla]

@gchwier, @gopiotr, @nashif, @PerMac, any objections or comments on this PR ? some more details are also in the RFC #67058

I want to check it before PR merge. Please give me about 23 days.

[golowanow]

I want to check it before PR merge. Please give me about 23 days.

@katgiadla, did you have time to check ?

[katgiadla]

I want to check it before PR merge. Please give me about 23 days.

@katgiadla, did you have time to check ?

No, I hadn't. I'll check it at this week.

[golowanow]

@katgiadla, do you have any feedback to share ?

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[kartben]

This needs rebasing but not sure what's the current status of this PR?