Consider using deltafetch

[ojongerius]

This looks interesting to speed things up, have not looked at it closely, hope it looks at more than the URL ;)

This is a Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls of the same spider, thus producing a "delta crawl" containing only new items.

This also speeds up the crawl, by reducing the number of requests that need to be crawled, and processed (typically, item requests are the most CPU intensive).

https://github.com/scrapy-plugins/scrapy-deltafetch