Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

redis test flakeyness #297

Open
wez opened this issue Sep 30, 2024 · 3 comments
Open

redis test flakeyness #297

wez opened this issue Sep 30, 2024 · 3 comments

Comments

@wez
Copy link
Collaborator

wez commented Sep 30, 2024

get to the bottom of this:

thread 'throttle::test::redis_script_throttle_1_000' panicked at crates/throttle/src/throttle.rs:316:9:
throttled after 1160 iterations for limit 1000. diff=160 is not within tolerance 20

I'll bump the tolerance for now so that we're able to produce builds in the mean time
wez added a commit that referenced this issue Sep 30, 2024
@wez
throttle: increase tolerance of new redis throttle test case
1ae88c1 
This is acting a bit flakey, and the magnitude expressed as
a percentage is sometimes a bit on the high side at larger
than 10%, so it would be good to understand what is really
going on.

For now we can bump the tolerance so that we're able
to continue producing builds.

refs: #297
@djc
Copy link
Contributor

djc commented Oct 2, 2024

Just wondering, have you observed this locally, or only in CI so far?

@wez
Copy link
Collaborator Author

wez commented Oct 2, 2024

only CI so far. I'm leaning towards this being a bit of a noisy neighbor issue, but I did spot a flakey run yesterday where the variance was over 20% and that feels a little too high from a statistical significance perspective.

@wez
Copy link
Collaborator Author

wez commented Oct 2, 2024

or this one with huge variance:

thread 'throttle::test::redis_script_throttle_1_000' panicked at crates/throttle/src/throttle.rs:316:9:
throttled after 1745 iterations for limit 1000. diff=745 is not within tolerance 200

wez added a commit that referenced this issue Nov 19, 2024
@wez
throttle: constraint burst value in tests
6a0f905 
I'm hoping that this will help to reduce flakeyness in CI

refs: #297
wez added a commit that referenced this issue Nov 19, 2024
@wez
throttle: make tolerance <= to reduce flakes
a827e98 
We're fine with being that close

refs: #297
wez added a commit that referenced this issue Dec 12, 2024
@wez
throttle: increase tolerance in test cases
8aee36a 
refs: #297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wez @djc