Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Rate Limit Handling in GithubFetcher #99

Open
cdboer opened this issue Aug 27, 2023 · 0 comments
Open

Implement Rate Limit Handling in GithubFetcher #99

cdboer opened this issue Aug 27, 2023 · 0 comments
Assignees
Labels
feature Implement this feature github

Comments

@cdboer
Copy link
Collaborator

cdboer commented Aug 27, 2023

Is your feature request related to a problem? Please describe.
When using the GithubFetcher, I encounter issues when the rate limit for the GitHub API is reached. Currently, as soon as the rate limit is exceeded, the fetcher stops its operation and throws an error. This means manual intervention is required to restart the fetching process once the rate limit is reset, causing interruptions and inefficiencies in the data retrieval process.

Describe the solution you'd like
Instead of the fetcher stopping and throwing an error immediately upon hitting the rate limit, I propose that the GithubFetcher should intelligently detect when the rate limit has been reached and then pause its operation. It should wait until the rate limit reset time has passed before continuing with its requests. This would allow for continuous and smooth data fetching without the need for manual restarts. Ideally, there would be a logging mechanism to inform the user when the fetcher is in the "waiting state" due to rate limit constraints, so they are aware of the delay.

Additional context
GitHub's API provides headers with every response that tell you how many requests you have remaining in the current rate limit window, as well as the time when the window resets. This information can be used to implement the proposed feature.

github.GithubException.RateLimitExceededException: 403 {"message": "API rate limit exceeded for user ID XXXXXX.", "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
@cdboer cdboer added the feature Implement this feature label Aug 27, 2023
@cdboer cdboer self-assigned this Aug 27, 2023
@cdboer cdboer added the github label Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Implement this feature github
Projects
None yet
Development

No branches or pull requests

1 participant