Determine internal vs external links at parse time #884
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Eric-Arellano
commented
Feb 26, 2024
| @@ -83,8 +83,10 @@ async function main() { | |||
|
|
|||
| const fileBatches = await determineFileBatches(args); | |||
| const otherFiles = [ | |||
| ...(await globby("public/**/*")).map((fp) => new File(fp, [])), | |||
| ...SYNTHETIC_FILES.map((fp) => new File(fp, [], true)), | |||
| ...(await globby("public/{images,videos}/**/*")).map( | |||
There was a problem hiding this comment.
This would include the objects.inv files, which was wrong.
| for (const filePath of this.toCheck) { | ||
| const parsed = await parseFile(filePath); | ||
| files.push(new File(filePath, parsed.anchors)); | ||
| if (!IGNORED_FILES.has(filePath)) { |
There was a problem hiding this comment.
All three ignores are moved into addLinksToMap.
| internalLinks.push(new InternalLink(linkPath, originFiles)); | ||
| } | ||
| addLinksToMap(filePath, parsed.internalLinks, internalLinksToOriginFiles); | ||
| if (loadExternalLinks) { |
There was a problem hiding this comment.
We don't waste our time storing external links if not necessary. That should slightly decrease memory usage and improve performance.
| results.push(await link.check()); | ||
| } | ||
| // For loop reduces the risk of rate-limiting. | ||
| for (let link of externalLinkList) { |
There was a problem hiding this comment.
externalLinkList will be empty if we're not checking external links.
arnaucasau
approved these changes
Feb 26, 2024
Co-authored-by: Arnau Casau <[email protected]>
frankharkins
approved these changes
Feb 26, 2024
scripts/lib/links/FileBatch.ts
Outdated
Comment on lines
141
to
148
| if ( | ||
| IGNORED_FILES.has(filePath) || | ||
| ALWAYS_IGNORED_URLS.has(link) || | ||
| FILES_TO_IGNORES[filePath]?.includes(link) | ||
| ) { | ||
| return; | ||
| } | ||
|
|
frankharkins
pushed a commit
to frankharkins/documentation
that referenced
this pull request
Jul 22, 2024
Part of Qiskit#876 to split the internal and external link checkers into distinct programs. To do this, it's useful to split out internal vs. external links at parse time, whereas before we did it inside `FileBatch`. The external link checker won't use `FileBatch`. This is only a refactor and doesn't change the program, other than now using `HEAD` for external link requests rather than `GET`. We also now use a `Set` for anchors so that checking if an anchor is included is faster. --------- Co-authored-by: Arnau Casau <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #876 to split the internal and external link checkers into distinct programs. To do this, it's useful to split out internal vs. external links at parse time, whereas before we did it inside
FileBatch. The external link checker won't useFileBatch.This is only a refactor and doesn't change the program, other than now using
HEADfor external link requests rather thanGET. We also now use aSetfor anchors so that checking if an anchor is included is faster.