Code

Custom Splash responses#45

Code

Merged

kmike merged 36 commits intomasterscrapy-plugins/scrapy-splash:masterfrom responsesscrapy-plugins/scrapy-splash:responsesCopy head branch name to clipboardApr 2, 2016

+1,074-63

Commits

Commits on Mar 29, 2016

• initial implementation of custom Splash responses

  kmike

  committed
  19d1170

  Copy full SHA for 19d1170

• small cleanup

  kmike

  committed
  ca52e1d

  Copy full SHA for ca52e1d

• SplashRequest: get _original_url from request meta; improve __repr__

  kmike

  committed
  82ccc1c

  Copy full SHA for 82ccc1c

• expose SplashRequest as scrapyjs.SplashRequest

  kmike

  committed
  637595b

  Copy full SHA for 637595b

• fixed creation of custom Splash response classes

  kmike

  committed
  eeb5972

  Copy full SHA for eeb5972

• fixed response caching and duplication detection

  kmike

  committed
  f25075e

  Copy full SHA for f25075e

• ignore .scrapy folder

  kmike

  committed
  67b8c5e

  Copy full SHA for 67b8c5e

• response class is fixed by middleware for cached responses

  kmike

  committed
  ddd515d

  Copy full SHA for ddd515d

• DOC typo fix

  kmike

  committed
  1ec775f

  Copy full SHA for 1ec775f

• DOC mention that SplashRequest handles URL fragments automatically

  kmike

  committed
  32126fd

  Copy full SHA for 32126fd

Commits on Mar 30, 2016

• add an option to return unprocessed responses

  kmike

  committed
  913aff2

  Copy full SHA for 913aff2

• TST enable branch coverage

  kmike

  committed
  74df321

  Copy full SHA for 74df321

• special handling of some JSON keys in Splash responses

  kmike

  committed
  e581c66

  Copy full SHA for e581c66

• TST test that caching works for SplashResponses

  kmike

  committed
  86427bd

  Copy full SHA for 86427bd

• TST add htmlcov to .gitignore

  kmike

  committed
  c3cd10d

  Copy full SHA for c3cd10d

• only fix Response class if it is not fixed yet

  kmike

  committed
  a5d7070

  Copy full SHA for a5d7070

• fix Content-Type header for magic responses built from 'html' key

  kmike

  committed
  f0519ab

  Copy full SHA for f0519ab

• extract headers_to_scrapy function

  kmike

  committed
  9b7ed67

  Copy full SHA for 9b7ed67

• set SplashResponse cookies from 'cookies' json

  kmike

  committed
  fee2662

  Copy full SHA for fee2662

• PY2 fixed cookie handing

  kmike

  committed
  c2d7153

  Copy full SHA for c2d7153

• SplashJsonResponse: extract magic response handing to a method

  kmike

  committed
  643ee3c

  Copy full SHA for 643ee3c

• DOC fixed SplashResponse for Splash servers which use HTTP compression

  kmike

  committed
  0af058b

  Copy full SHA for 0af058b

Commits on Mar 31, 2016

• make sure SplashRequest is never handled by AjaxCrawlMiddleware

  kmike

  committed
  373264a

  Copy full SHA for 373264a

• DOC readme and example improvements

  kmike

  committed
  b43ea3a

  Copy full SHA for b43ea3a

• DOC typo fix

  kmike

  committed
  3fc7d09

  Copy full SHA for 3fc7d09

• DOC fixed a typo in example

  kmike

  committed
  356f19f

  Copy full SHA for 356f19f

Commits on Apr 1, 2016

• Fixed repr of SplashRequest when if haven't reached SplashMiddleware yet

  kmike

  committed
  12e81b7

  Copy full SHA for 12e81b7

• Log content of Splash Bad Request errors. See GH-37.

  kmike

  committed
  69eb8f6

  Copy full SHA for 69eb8f6

• cookies handling overhaul

  kmike

  committed
  6b777b5

  Copy full SHA for 6b777b5

Commits on Apr 2, 2016

• response.cookiejar

  kmike

  committed
  5a54996

  Copy full SHA for 5a54996

• DOC improve cookie docs

  kmike

  committed
  10162df

  Copy full SHA for 10162df

• actually set response.cookiejar

  kmike

  committed
  67c1ae8

  Copy full SHA for 67c1ae8

• cleanup cookie handling: drop unneeded code, add more tests

  kmike

  committed
  6637779

  Copy full SHA for 6637779

• TST add a test form SplashRequest repr

  kmike

  committed
  5a8954c

  Copy full SHA for 5a8954c

• add support for SplashRequest 'cookies' argument

  kmike

  committed
  d394c6f

  Copy full SHA for d394c6f

• pass headers to Splash by default

  kmike

  committed
  92018c4

  Copy full SHA for 92018c4