GCP: Don't try to read tiles which very probably don't exist #69
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mhutchinson
approved these changes
Jul 17, 2024
AlCutter
force-pushed
the
prewarm_probably_empty_tiles
branch
from
fed9adb to
91000ae
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR detects when the cache is requesting tiles which are beyond the extent of the current tree and skips the read.
These "future" tiles are very likely not to exist, and, since they're requested by the visitor, we end up trying to fetching serially which adds a fair chunk of latency to an integration run. Not fetching these future tiles speeds the integration up by around 25%.
While it's possible that these tiles could occasionally exist (e.g. due to an integration run crashing previously), not reading should still be safe as the storage design makes integration an idempotent operation, and this is enforced when tiles are finally flushed to GCS at the end of the integration run.
Toward #23