Avoid duplicate requests

.bumpversion.cfg

@@ -1,7 +1,9 @@
 [bumpversion]
-current_version = 0.9.0
+current_version = 0.13.0
 commit = True
 tag = True
 tag_name = {new_version}
 
-[bumpversion:file:setup.py]
+[bumpversion:file:docs/conf.py]
+
+[bumpversion:file:scrapy_zyte_api/__version__.py]

.github/workflows/test.yml

@@ -34,10 +34,20 @@ jobs:
         - python-version: '3.10'
         - python-version: '3.11'
 
+        - python-version: '3.8'
+          toxenv: pinned-provider
+        - python-version: '3.11'
+          toxenv: provider
+
+        - python-version: '3.7'
+          toxenv: pinned-extra
+        - python-version: '3.11'
+          toxenv: extra
+
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
@@ -57,12 +67,12 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ["3.11"]
-        tox-job: ["mypy", "flake8", "twine-check"]
+        tox-job: ["mypy", "linters", "twine-check", "docs"]
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies

.gitignore

@@ -8,3 +8,4 @@ docs/_build
 *.egg-info/
 __pycache__/
 /test-results/
+build/

.pre-commit-config.yaml

@@ -1,23 +1,16 @@
 repos:
--   repo: https://github.com/pre-commit/mirrors-isort
-    rev: v5.7.0
+-   repo: https://github.com/PyCQA/isort
+    rev: 5.12.0
     hooks:
     - id: isort
--   repo: https://github.com/ambv/black
-    rev: 22.3.0
+-   repo: https://github.com/psf/black
+    rev: 23.7.0
     hooks:
     - id: black
-      language_version: python3.8
-      additional_dependencies:
-      - click<8.1
 -   repo: https://github.com/pycqa/flake8
-    rev: 3.8.4
-    hooks:
-        - id: flake8
--   repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.991
+    rev: 6.0.0
     hooks:
-    - id: mypy
+    - id: flake8
       additional_dependencies:
-      - types-setuptools
-      args: [--check-untyped-defs, --ignore-missing-imports, --no-warn-no-return]
+        - flake8-docstrings
+        - flake8-print

.readthedocs.yml

@@ -0,0 +1,12 @@
+version: 2
+formats: all
+sphinx:
+  configuration: docs/conf.py
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"  # Keep in sync with .github/workflows/test.yml
+python:
+  install:
+    - requirements: docs/requirements.txt
+    - path: .

CHANGES.rst

@@ -1,6 +1,138 @@
 Changes
 =======
 
+0.13.0 (2023-12-13)
+-------------------
+
+* Updated requirement versions:
+
+  * andi >= 0.5.0
+  * scrapy-poet >= 0.18.0
+  * web-poet >= 0.15.1
+  * zyte-api >= 0.4.8
+
+* The spider is now closed and the finish reason is set to
+  ``"zyte_api_bad_key"`` or ``"zyte_api_suspended_account"`` when receiving
+  "Authentication Key Not Found" or "Account Suspended" responses from Zyte
+  API.
+
+* The spider is now closed and the finish reason is set to
+  ``"failed_forbidden_domain"`` when all start requests fail because they are
+  pointing to domains forbidden by Zyte API.
+
+* The spider is now closed and the finish reason is set to
+  ``"plugin_conflict"`` if both scrapy-zyte-smartproxy and the transparent mode
+  of scrapy-zyte-api are enabled.
+
+* The ``extractFrom`` extraction option can now be requested by annotating the
+  dependency with a ``scrapy_zyte_api.ExtractFrom`` member (e.g.
+  ``product: typing.Annotated[Product, ExtractFrom.httpResponseBody]``).
+
+* The ``Set-Cookie`` header is now removed from the response if the cookies
+  were returned by Zyte API (as ``"experimental.responseCookies"``).
+
+* The request fingerprinting was improved by refining which parts of the
+  request affect the fingerprint.
+
+* Zyte API Request IDs are now included in the error logs.
+
+* Split README.rst into multiple documentation files and publish them on
+  ReadTheDocs.
+
+* Improve the documentation for the ``ZYTE_API_MAX_REQUESTS`` setting.
+
+* Test and CI improvements.
+
+0.12.2 (2023-10-19)
+-------------------
+
+* Unused ``<data type>Options`` (e.g. ``productOptions``) are now dropped
+  from ``ZYTE_API_PROVIDER_PARAMS`` when sending the Zyte API request
+* When logging Zyte API requests, truncation now uses
+  "..." instead of Unicode ellipsis.
+
+0.12.1 (2023-09-29)
+-------------------
+
+* The new ``_ZYTE_API_USER_AGENT`` setting allows customizing the user agent 
+  string reported to Zyte API.
+
+  Note that this setting is only meant for libraries and frameworks built on 
+  top of scrapy-zyte-api, to report themselves to Zyte API, for client software 
+  tracking and monitoring purposes. The value of this setting is *not* the 
+  ``User-Agent`` header sent to upstream websites when using Zyte API.
+
+
+0.12.0 (2023-09-26)
+-------------------
+
+* A new ``ZYTE_API_PROVIDER_PARAMS`` setting allows setting Zyte API
+  parameters, like ``geolocation``, to be included in all Zyte API requests by
+  the scrapy-poet provider.
+
+* A new ``scrapy-zyte-api/request_args/<parameter>`` stat, counts the number of
+  requests containing a given Zyte API request parameter. For example,
+  ``scrapy-zyte-api/request_args/url`` counts the number of Zyte API requests
+  with the URL parameter set (which should be all of them).
+
+  Experimental is treated as a namespace, and its parameters are the ones
+  counted, i.e. there is no ``scrapy-zyte-api/request_args/experimental`` stat,
+  but there are stats like
+  ``scrapy-zyte-api/request_args/experimental.responseCookies``.
+
+
+0.11.1 (2023-08-25)
+-------------------
+
+* scrapy-zyte-api 0.11.0 accidentally increased the minimum required version of
+  scrapy-poet from 0.10.0 to 0.11.0. We have reverted that change and
+  implemented measures to prevent similar accidents in the future.
+
+* Automatic parameter mapping no longer warns about dropping the
+  ``Accept-Encoding`` header when the header value matches the Scrapy default.
+
+* The README now mentions additional changes that may be necessary when
+  switching Twisted reactors on existing projects.
+
+* The README now explains how status codes, from Zyte API or from wrapped
+  responses, are reflected in Scrapy stats.
+
+0.11.0 (2023-08-07)
+-------------------
+
+* Added a ``ZYTE_API_MAX_REQUESTS`` setting to limit the number of successful
+  Zyte API requests that a spider can send. Reaching the limit stops the
+  spider.
+
+* Setting ``requestCookies`` to ``[]`` in the ``zyte_api_automap`` request
+  metadata field now triggers a warning.
+
+0.10.0 (2023-07-14)
+-------------------
+
+* Added more data types to the scrapy-poet provider:
+
+  * ``zyte_common_items.ProductList``
+  * ``zyte_common_items.ProductNavigation``
+  * ``zyte_common_items.Article``
+  * ``zyte_common_items.ArticleList``
+  * ``zyte_common_items.ArticleNavigation``
+
+* Moved the new dependencies added in 0.9.0 and needed only for the scrapy-poet
+  provider (``scrapy-poet``, ``web-poet``, ``zyte-common-items``) into the new
+  optional feature ``[provider]``.
+
+* Improved result caching in the scrapy-poet provider.
+
+* Added a new setting, ``ZYTE_API_USE_ENV_PROXY``, which can be set to ``True``
+  to access Zyte API using a proxy configured in the local environment.
+
+* Fixed getting the Scrapy Cloud job ID.
+
+* Improved the documentation.
+
+* Improved the CI configuration.
+
 0.9.0 (2023-06-13)
 ------------------
 
@@ -71,7 +203,7 @@ Changes
     cookiejar of the request.
 
   * A new boolean setting, ``ZYTE_API_EXPERIMENTAL_COOKIES_ENABLED``, can be
-    set to ``True`` to enable automated mapping of cookies from a request
+    set to ``True`` to enable automatic mapping of cookies from a request
     cookiejar into the ``experimental.requestCookies`` Zyte API parameter.
 
 * ``ZyteAPITextResponse`` is now a subclass of ``HtmlResponse``, so that the
@@ -143,10 +275,10 @@ When upgrading, you should set the following in your Scrapy settings:
   be set to ``True`` to make all requests use Zyte API by default, with request
   parameters being automatically mapped to Zyte API parameters.
 * Add a Request meta key, ``zyte_api_automap``, that can be used to enable
-  automated request parameter mapping for specific requests, or to modify the
-  outcome of automated request parameter mapping for specific requests.
+  automatic request parameter mapping for specific requests, or to modify the
+  outcome of automatic request parameter mapping for specific requests.
 * Add a ``ZYTE_API_AUTOMAP_PARAMS`` setting, which is a counterpart for
-  ``ZYTE_API_DEFAULT_PARAMS`` that applies to requests where automated request
+  ``ZYTE_API_DEFAULT_PARAMS`` that applies to requests where automatic request
   parameter mapping is enabled.
 * Add the ``ZYTE_API_SKIP_HEADERS`` and ``ZYTE_API_BROWSER_HEADERS`` settings
   to control the automatic mapping of request headers.