Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase efficiency of wiki context creation #113

Open
steffencruz opened this issue Feb 21, 2024 · 1 comment
Open

Increase efficiency of wiki context creation #113

steffencruz opened this issue Feb 21, 2024 · 1 comment
Assignees
@steffencruz

Comments

@steffencruz
Copy link
Collaborator

At present, 40-50% of wiki articles are rejected because they do not contain sections with greater than our minimum number of words. This introduces extra latency as we sequentially download additional articles until finding a suitable one.

We should instead download batches of pages concurrently so that we don't need to wait for sequential processing times.
@steffencruz steffencruz self-assigned this Feb 22, 2024
@steffencruz
Copy link
Collaborator Author

@p-ferreira related to your current work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@steffencruz steffencruz

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@steffencruz