You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
When using the GithubFetcher, I encounter issues when the rate limit for the GitHub API is reached. Currently, as soon as the rate limit is exceeded, the fetcher stops its operation and throws an error. This means manual intervention is required to restart the fetching process once the rate limit is reset, causing interruptions and inefficiencies in the data retrieval process.
Describe the solution you'd like
Instead of the fetcher stopping and throwing an error immediately upon hitting the rate limit, I propose that the GithubFetcher should intelligently detect when the rate limit has been reached and then pause its operation. It should wait until the rate limit reset time has passed before continuing with its requests. This would allow for continuous and smooth data fetching without the need for manual restarts. Ideally, there would be a logging mechanism to inform the user when the fetcher is in the "waiting state" due to rate limit constraints, so they are aware of the delay.
Additional context
GitHub's API provides headers with every response that tell you how many requests you have remaining in the current rate limit window, as well as the time when the window resets. This information can be used to implement the proposed feature.
github.GithubException.RateLimitExceededException: 403 {"message": "API rate limit exceeded for user ID XXXXXX.", "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
When using the GithubFetcher, I encounter issues when the rate limit for the GitHub API is reached. Currently, as soon as the rate limit is exceeded, the fetcher stops its operation and throws an error. This means manual intervention is required to restart the fetching process once the rate limit is reset, causing interruptions and inefficiencies in the data retrieval process.
Describe the solution you'd like
Instead of the fetcher stopping and throwing an error immediately upon hitting the rate limit, I propose that the GithubFetcher should intelligently detect when the rate limit has been reached and then pause its operation. It should wait until the rate limit reset time has passed before continuing with its requests. This would allow for continuous and smooth data fetching without the need for manual restarts. Ideally, there would be a logging mechanism to inform the user when the fetcher is in the "waiting state" due to rate limit constraints, so they are aware of the delay.
Additional context
GitHub's API provides headers with every response that tell you how many requests you have remaining in the current rate limit window, as well as the time when the window resets. This information can be used to implement the proposed feature.
The text was updated successfully, but these errors were encountered: