Custom Splash responses

[kmike]

This PR provides an easier API to process data returned by Splash.

TODO:

• fix saving/loading Splash responses to/from cache;
• magical JSON response handling:
  • set response.headers from 'headers';
  • set response.headers from 'cookies';
  • set response.body either from 'html' or 'body';
  • set response.status from 'http_status';
  • set response.url from 'url.
• add shortcuts to access base64-encoded values (e.g. 'png' / 'jpeg' keys);
• re-check that SplashRequest works correctly with persistent request queues: it works, but __repr__ is lost; see Disk queues don't preserve Request class scrapy/scrapy#1890.
• check how custom Splash responses interact with builtin Scrapy middlewares, esp. when headers are restored from JSON response:
  • RetryMiddleware;
  • AjaxCrawlMiddleware;
  • ChunkedTransferMiddleware;
  • DecompressionMiddleware - not enabled by default;
  • HttpCompressionMiddleware - see set HttpCompressionMiddleware priority to 810 scrapy/scrapy#1895;
  • RedirectMiddleware;
  • RobotsTxtMiddleware;
  • RefererMiddleware
• check that examples work;
• allow to handle cookies transparently;
• allow to handle headers transparently;

See #44, #14, #12, #10, #8.

[codecov-io]

Current coverage is 89.70%

Merging #45 into master will increase coverage by +0.33% as of e352f77

@@            master    #45   diff @@
=====================================
  Files            6      9     +3
  Stmts          160    379   +219
  Branches         0     69    +69
  Methods          0      0       
=====================================
+ Hit            143    340   +197
- Partial          0     20    +20
- Missed          17     19     +2

Review entire Coverage Diff as of e352f77

Powered by Codecov. Updated on successful CI builds.

[lopuhin]

I think both session_id and new_session_id are currently (https://github.com/scrapy-plugins/scrapy-splash/pull/45/files#diff-93e5c0fca1f417cfa28d48c75408be45R59) searched for in request.meta['splash'], not in ```request.meta['splash']['args']`. Which is more correct, the docs or the code?

[kmike]

A good catch; the code is correct. args are arguments sent to Splash; new_session_id is a SplashMiddleware option.