-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a timeout to TestWorld #1467
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
having a timeout makes sense to me, I've seen a couple of tests that take more than 60 secs to run.
4d4df47
to
8155640
Compare
Further increase timeout for dkzp_validator tests
dae5c6d
to
135fbe1
Compare
It is much slower than in the single thread config.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1467 +/- ##
==========================================
- Coverage 93.84% 93.81% -0.03%
==========================================
Files 234 234
Lines 41826 41867 +41
==========================================
+ Hits 39250 39276 +26
- Misses 2576 2591 +15 ☔ View full report in Codecov by Sentry. |
The new timeout applies to each TestWorld protocol execution using one of the helpers ( After a few rounds of increasing timeouts due to failures in CI (wow the CI machines are slow), there was only one test that seemed to need more than a 60s timeout -- the I don't think it's worth being aggressive with timeouts for unit tests (the goal is to catch hangs, not identify performance regressions), so I'm happy to increase the timeouts further if there are places where you think they are too aggressive. |
There is a risk that this will introduce flakiness, if there are tests where the tail of the runtime distribution exceeds the timeout. On the other hand, we know that stalls are a predominant failure mode of our protocols, and a clear failure message is a lot nicer than a test job that hangs until somebody notices and kills it.