Allow spider attr #15
Conversation
|
|
||
| slot_policy = splash_options.get('slot_policy', self.slot_policy) | ||
| self._set_download_slot(request, meta, slot_policy) | ||
|
|
||
| args = splash_options.setdefault('args', {}) | ||
| args.setdefault('url', request.url) | ||
| args['url'] = request.url |
There was a problem hiding this comment.
@kmike are you okay with this change. What was usecase of .setdefault() here? Does it have any sense to create request with one url and specify another url in meta?
There was a problem hiding this comment.
Maybe I was thinking of allowing user not to use 'url' argument at all. This argument is not required for Splash scripts - e.g. you can render a chunk of HTML using splash:set_content. But the meta syntax doesn't make much sense in this case; something in lines of #12 could be a better fit.
* removed process_html (render response in html feature) from this PR * removed remote request tests with extra spider
* refactors tests from functions to objects inheriting from unittest.TestCase * adds tests for enabling middleware with spider attribute
|
updated after discussions |
|
@pawelmhm tests are failing |
| @@ -6,6 +6,7 @@ | |||
| from scrapy.exceptions import NotConfigured | |||
|
|
|||
| from scrapy import log | |||
| from scrapy.http.response.html import HtmlResponse | |||
TODO
something else? |
|
I know it has been a rather long time, but it looks like this is still a valid pull request, and a nice once at that. It seems to me like it’s only missing renaming @pawelmhm Do you have plans to resume work on it at some point? |
|
Continued at #235 |
This is about #14 and
#11
opening this for discussion, needs tests and improvements, would be cool to get some feedback if it's in good direction