Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

Fail to test run when user/kernel test specification mismatch #20

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ndvbrk
Copy link

@ndvbrk ndvbrk commented Jan 31, 2021

Hybrid tests are great, but manually maintaining the mapping of user/kernel tests is hard.
We've seen scenarios in which user space test were left orphaned/unmatched after an innocent refactor to the kernel side.

Our expectation were for KTF to detect this and fail CI, rather than ignoring this issue.
What happened instead was that KTF decided to skip those problematic test suites entirely, which caused an additional unwanted effect: later test would be executed under the wrong name (not sure why).

The way in which User-Kernel communicate over netlink is wonderful, and I found no parsing error.
The function KernelTestMgr::get_test_names() was able to detect this exact scenario, only the current problem resolution is strange. Why skip?

What puzzled me the most was "second.test_names.size() == 0", which means that the call to the log would be done only on empty test suites, while the log itself refers to the opposite. (Our errors were skipped silently as a result)

Our suggested change is to send a helpful log message for each the problematic tests, and then exit immediately. Tested locally that it works, but perhaps it's been fixed since I'm running an outdated version.

@knuto
Copy link
Contributor

knuto commented Jan 31, 2021

Thanks for the patch and your comments - I think you are onto something here - let me have a look.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants