dyer is designed for reliable, flexible and fast web-crawling, providing some high-level, comprehensive features without compromising speed.
dyer provides some high-level features:
- asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
- Event-driven, once you set the initials and recursive generator,
dyerwill handle the rest of it. - User-friendly and flexible,
dyeroffers high-level, flexible, easy to use wrappers and APIs what does a lot for you.
Get started by installing dyer-cli and looking over the examples.