Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky pytest tests #72

Closed
jjnesbitt opened this issue Sep 26, 2024 · 0 comments · Fixed by #87
Closed

Fix flaky pytest tests #72

jjnesbitt opened this issue Sep 26, 2024 · 0 comments · Fixed by #87
Assignees

Comments

@jjnesbitt
Copy link
Member

jjnesbitt commented Sep 26, 2024

The test_populate test downloads external data as a part of the test. This means that if the download fails for any reason (very common with web requests, especially if the host is unreliable), the test will fail. This isn't good practice, as it means that tests can pass or fail independent of any changed code in our codebase. We should aim for our tests to be reproducible, that is, given the same commit in our code base, the test should produce the same result, every time.

To allow for our tests to be reproducible, we can employ the following strategies:

  1. Test at the appropriate level. For example, the populate management script really just wraps ingest_use_case. We should just test that function directly, to allow for more robust tests, instead of relying on proxy functions like call_command, which we don't have much control over.
  2. Mock data. Rather than relying on the downloading of datasets in our tests, we can mock or generate the minimum data we need to accomplish the task.
  3. Limit test scope. In the above example, ingest_use_case really just calls ingest_datasets, ingest_projects, and ingest_charts, sequentially and in that order. We should realistically just test those functions directly, which will give us more control, and isolate the setup to each individual use case. If we then want to test ingest_use_case overall, we don't need to test every part of all the subsequent functions, since those are already covered by the individual tests. We can minimally just test that all 3 of them ran successfully.
@jjnesbitt jjnesbitt changed the title Fix flaky pytest tests Fix flaky pytest test Oct 1, 2024
@annehaley annehaley added this to the Miscellaneous Improvements milestone Oct 1, 2024
@jjnesbitt jjnesbitt changed the title Fix flaky pytest test Fix flaky pytest tests Oct 1, 2024
@annehaley annehaley linked a pull request Nov 12, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants