Feature/task-and-output-test-coverage #48
Conversation
… of https://github.com/DNAstack/wdl-ci into feature/task-and-output-test-coverage
Create a dictionary of absent outputs rather than present ones to simplify iterating over; adjust iterating over keys in the dictionary of missing outputs and corresponding exception
There was a problem hiding this comment.
Good first pass.
The main issue I see is that this will miss outputs that have no tests defined, since we're relying on the config file to tell us what tasks and outputs to test.
Can you try reading through WDL files in the directory and using that to build out an idea of what outputs each task has, and which outputs are missing tests?
Then we'd want to get an understanding of % coverage, rather than the absolute number of tests for each output/task. i.e. so long as an output is covered by 1 test, we can say that it is 100% covered. If 3/4 outputs for a task have at least 1 test, that task has 75% test coverage.
Output could look something like this:
workflowNameA: 84% # this is the workflow coverage
taskA: 75% # this is the task coverage
taskB: 63%
taskC: 100%
taskD: 100%
workflowNameB: 100%
taskA: 100%
taskB: 100%
...
Total coverage 89% # this is the total coverage across all outputs from all tasks in all workflows
Task coverage = sum(# of task outputs with at least 1 test) / (# of task outputs)
Workflow coverage = sum(# of task outputs in all tasks with at least 1 test) / (# of task outputs in all tasks)
…dle-output-key-error
Feature/handle-output-key-error
Instead of relying on the config file to tell us what tasks/outputs to test, the WDL files in the directory tell us what needs testing and then the config file is checked for associated tests. Currently, there are some edge cases that need handling; 1. Compound outputs and 2. Workflows from the custom_tests directory are included in the coverage calculation, which is not correct.
… coverage script to facilitate testing
…setUp; add more TODO tests
…d based on workflow_name filter
…name; calculate task coverage using dict instead of list; add TODO
…utputs to compute coverage for
…er that resulted in confusing output Previously, workflow_name_filter was being checked as part of the total coverage being returned; this was confusing as if it was set, a print message would tell the user there was no outputs to compute coverage for. It makes more sense to return coverage at all levels (task, workflow, and total) provided there are outputs to test
untested optional outputs
Features
wdl-ci coveragecommand to calculate and output task-level, workflow-level, and overall test coverage.If there are any tasks or outputs with no tests, a warning is provided to the user, listing the relevant outputs and associated tasks. I left a
##TODOin here regarding optional inputs.