Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rotate through all given proxies before letting yagoogle handle the waiting #70

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

sporetec
Copy link

This rework lets pagodo:

  • Have a new flag and argument to divide the wanted wait time by the amount of unblocked proxies and recalculate on a proxy being blocked. Default behavior stays unchanged
  • Rotate through all given proxies before letting yagoogle handle the 429 to reduce execution time

@opsdisk
Copy link
Owner

opsdisk commented Aug 17, 2022

Thanks for submitting a PR @sporetec! Just a heads up, it may be a week or two until I can take a good look at it.

@opsdisk
Copy link
Owner

opsdisk commented Sep 16, 2022

Hey @sporetec - haven't forgotten about this! Hoping to dig into the PR soon.

@opsdisk
Copy link
Owner

opsdisk commented Sep 29, 2022

@sporetec - Would it be possible to tackle this feature first?

"Rotate through all given proxies before letting yagoogle handle the 429 to reduce execution time"

To be clear, you don't want the script to automatically go to sleep if an HTTP 429 is detected, especially if there are other proxies that can be tried?

As it is currently written, since yagooglesearch_manages_http_429s is not set during the instantiation of a yagooglesearch.SearchClient object (like is done here https://github.com/opsdisk/pagodo/blob/master/pagodo.py#L200), it defaults to True https://github.com/opsdisk/yagooglesearch/blob/master/yagooglesearch/__init__.py#L82

@DannyCork
Copy link

would be good to get this into master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants