Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performance of stale image check #2528

Merged
merged 2 commits into from
Dec 23, 2024
Merged

Fix performance of stale image check #2528

merged 2 commits into from
Dec 23, 2024

Conversation

Eric-Arellano
Copy link
Collaborator

Closes #2518.

We were before checking every image inside await Promise.all() and only logging results at the end. This is too slow to keep that much in memory, and it results in the experience feeling laggy to not get results until the very end.

Instead, it makes more sense to check one image at a time and eagerly report violations.

Other improvements in this PR:

  • Use ripgrep rather than git grep. The downside is requiring users have ripgrep installed, but this is a manual script we run with npx tsx, so it should be fine.
  • Log progress every 20 images
  • Add an allowlist.
example run
❌ image is unused: public/images/migration/dedicated_queue_no_jobs1.jpg
❌ image is unused: public/images/migration/switcher-small2.png
❌ image is unused: public/images/guides/normal_queue_with_providers1.jpg
❌ image is unused: public/images/guides/paulibasis.png
⏳ Checked 20 / 3519 images
❌ image is unused: public/images/qiskit-ibm-runtime/noisy-sim-circuit.png
❌ image is unused: public/images/qiskit-ibm-runtime/noisy-sim-estimator-circuit.png
❌ image is unused: public/images/qiskit-ibm-runtime/noisy-sim-estimator-ideal.png
❌ image is unused: public/images/qiskit-ibm-runtime/noisy-sim-estimator-noisy.png
❌ image is unused: public/images/qiskit-ibm-runtime/noisy-sim-sampler-ideal.png
⏳ Checked 40 / 3519 images
❌ image is unused: public/images/qiskit-ibm-runtime/noisy-sim-sampler-noisy.png
❌ image is unused: public/images/api/qiskit-addon-cutting/qiskit_addon_cutting-instructions-1.svg

Copy link
Member

@frankharkins frankharkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

scripts/js/commands/checkStaleImages.ts Outdated Show resolved Hide resolved
if (unused.length === 0) {
let allGood = true;
let numFilesChecked = 0;
for (const fp of paths) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The await Promise.all approach should work fine here. There won't be a race condition with numFilesChecked since Node is single-threaded. I'd expect it to make a difference since there's a lot of time waiting for commands to run. Just move L27 -> L34 so there's no await between incrementing and logging.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this would work well because we could trigger too many concurrent calls to ripgrep, which can peg the system.

Co-authored-by: Frank Harkins <[email protected]>
@Eric-Arellano Eric-Arellano added this pull request to the merge queue Dec 23, 2024
Merged via the queue into main with commit a568f22 Dec 23, 2024
3 checks passed
@Eric-Arellano Eric-Arellano deleted the EA/fix-stale-image branch December 23, 2024 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Fix performance of checkStaleImages.ts
2 participants