Is there a method for pause/continue or stop crawl? #2414
-
|
it seems like only it won't stop until all the requests are processed . Is there a method to control when to stop manually？ |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 8 replies
-
|
If you're running it from the command line, just interrupting the crawler with Ctrl+C will actually pause it. You can then resume it with |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I want to run the crawler with different host consist of a lot of pages, and i want to stop it after some time every day because the amount of page is very large. For example, I crawl A.com and B.com at first day for one hour, then next day crawl it's remain uncrawled page, repeat day and day. |
Beta Was this translation helpful? Give feedback.
-
|
Have you considered using the apify platform for this? Pausing and resuming crawlee app is available at the push of a button there. |
Beta Was this translation helpful? Give feedback.
-
|
sorry, it's a internal project, many private dependencies |
Beta Was this translation helpful? Give feedback.
-
|
Then your second best option may be to run crawlee in a subprocess and send sigint to it when you want to pause it - that is essentially the same as what happens when you press CTRL+C in the command line. I don't think you can pause the crawler programmatically without heavily relying on implementation details. |
Beta Was this translation helpful? Give feedback.
Then your second best option may be to run crawlee in a subprocess and send sigint to it when you want to pause it - that is essentially the same as what happens when you press CTRL+C in the command line. I don't think you can pause the crawler programmatically without heavily relying on implementation details.