Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Enhance integration test to avoid test failures with different seeds #9241

Open
5 of 6 tasks
thirtiseven opened this issue Sep 18, 2023 · 1 comment
Open
5 of 6 tasks
Assignees
Labels
test Only impacts tests

Comments

@thirtiseven
Copy link
Collaborator

thirtiseven commented Sep 18, 2023

Is your feature request related to a problem? Please describe.

Currently, the random seed in integration tests is set to 0 by default, and all cases can pass with this seed. However, if we change the seed to other values, some cases will fail.

I wish we could improve the integration tests to avoid this, e.g. use a dynamic seed for each run, or test with more data for those cases that are easy to fail.

Some failed cases:

We should fix them or mark them as XFAIL first.

Describe the solution you'd like
Set random seeds for each test run and log the seed for reproducibility.

Describe alternatives you've considered
Or we can make the easy to fail cases stronger, e.g. more data for aggregate tests and tests with long generated strings, more precise data range for fallback/exception tests.

Additional context
Some similar issues that has been fixed: #8716 and #8607

@thirtiseven thirtiseven added the ? - Needs Triage Need team to review and classify label Sep 18, 2023
@thirtiseven thirtiseven changed the title [FEA] Enhance integration tests to avoid tests failures with different seeds [FEA] Enhance integration test to avoid test failures with different seeds Sep 18, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Sep 19, 2023
@revans2
Copy link
Collaborator

revans2 commented Sep 21, 2023

The init_cap issue is something that CUDF says that they will fix. All of the others have had fixes merged in.

@mattahrens mattahrens added the test Only impacts tests label Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Only impacts tests
Projects
None yet
Development

No branches or pull requests

3 participants