Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TestHooksInSourceFreshnessDisabled flaky test #10308

Merged
merged 1 commit into from
Jun 13, 2024

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Jun 13, 2024

resolves #10309

Problem

TestHooksInSourceFreshnessDisabled passes when run via tests/functional/sources/test_source_freshness.py, but fails independently.

  • More specifically, test always passes when run with TestSourceFreshnessProjectHooksNotRun, likely due to a shared global (deprecations).
pytest tests/functional/sources/test_source_freshness.py::TestSourceFreshnessProjectHooksNotRun tests/functional/sources/test_source_freshness.py::TestHooksInSourceFreshnessDisabled

Solution

  • Safely cleanup deprecations in TestSourceFreshnessProjectHooksNotRun
  • Make log matching more precise + consistent across the hook-related functional test assertions

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
  • This PR includes type annotations for new and modified functions

@cla-bot cla-bot bot added the cla:yes label Jun 13, 2024
Copy link

codecov bot commented Jun 13, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.70%. Comparing base (8639290) to head (0f478a0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10308      +/-   ##
==========================================
- Coverage   88.76%   88.70%   -0.06%     
==========================================
  Files         180      180              
  Lines       22483    22483              
==========================================
- Hits        19957    19944      -13     
- Misses       2526     2539      +13     
Flag Coverage Δ
integration 86.00% <ø> (-0.11%) ⬇️
unit 61.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@MichelleArk MichelleArk added the Skip Changelog Skips GHA to check for changelog file label Jun 13, 2024
@MichelleArk MichelleArk changed the title deflake source freshness hooks test Fix TestHooksInSourceFreshnessDisabled flaky test Jun 13, 2024
def global_deprecations(self):
deprecations.reset_deprecations()
yield
deprecations.reset_deprecations()
Copy link
Contributor Author

@MichelleArk MichelleArk Jun 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

root cause: this clean-up call was missing from test_hooks_do_run_for_source_freshness initially, meaning the global active_deprecations was preserved across tests, triggering deprecations erronously (and cluttering up the logs)

@MichelleArk MichelleArk marked this pull request as ready for review June 13, 2024 15:15
@MichelleArk MichelleArk requested a review from a team as a code owner June 13, 2024 15:15
Copy link
Contributor

@McKnight-42 McKnight-42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh that's interesting, updates look fine to me

Copy link
Member

@aranke aranke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@MichelleArk MichelleArk merged commit d4a6482 into main Jun 13, 2024
67 checks passed
@MichelleArk MichelleArk deleted the deflake-source-freshness-hooks-test branch June 13, 2024 15:25
github-actions bot pushed a commit that referenced this pull request Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.8.latest cla:yes Skip Changelog Skips GHA to check for changelog file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Flaky test] tests/functional/sources/test_source_freshness.py::TestHooksInSourceFreshnessDefault is flaky
3 participants