Dependency Update Forcing Node.js v20 - Request for Awareness

[wojtekKrol]

Hello,

First, I'd like to express my appreciation for the excellent work done on the amplify crawlee library and its associated packages, including @crawlee/playwright, @crawlee/utils, and others. Your contributions are invaluable to the developer community.

I am writing to bring to your attention an issue related to the recent dependency updates, specifically concerning the got-scraping package. Currently, @crawlee/playwright depends on @crawlee/utils, which in turn depends on got-scraping with the version specified as "got-scraping": "^4.0.0".

The caret (^) character in the version specification allows for updates that do not change the leftmost non-zero digit, which means that version ^4.0.0 includes all minor and patch updates, including 4.x.x.

In got-scraping version 4.0.5, the got package dependency was updated from ^13.0.0 to ^14.2.1. This change is significant because:

• got version 13.0.0 requires Node.js >=16.
• got version 14.0.0 requires Node.js >=20.

As a result, this forces developers to use Node.js v20 when they update to the latest got-scraping version within the ^4.0.0 range. This requirement can be quite disruptive, especially for those who are not yet ready to upgrade their Node.js environment to version 20.

Additionally, it's important to note that got-scraping was updated from ^3.2.15 to ^4.0.0 in crawlee version 3.6.0, which means this issue affects all versions of crawlee released after 3.6.0.

This problem affects not only the amplify crawlee packages but also any other packages that use got-scraping with the caret (^) character in their version specification. The cascading effect of this dependency update can have a widespread impact on the developer community.

For those affected by this issue, there are workarounds available that support 16 >= Node.js:

• npm: Add the following to your package.json to override the latest got-scraping version:

  "overrides": {
    "got-scraping": "4.0.5" 
  }

• yarn: Add the following to your package.json to resolve the got-scraping version:

  "resolutions": {
    "got-scraping": "4.0.5"
  }

Could you please review this dependency update and consider the impact on the broader developer community? An interim solution could be pinning the got-scraping dependency to a version prior to 4.0.5 until a broader consensus or transition plan is in place for requiring Node.js v20.

Thank you for your understanding and consideration.

Best regards,

Wojciech

Below is an error thrown once we won't override:

npm install

npm error code EBADENGINE
npm error engine Unsupported engine
npm error engine Not compatible with your version of node/npm: [email protected]
npm error notsup Not compatible with your version of node/npm: [email protected]
npm error notsup Required: {"node":">=20"}
npm error notsup Actual:   {"npm":"10.7.0","node":"v18.20.3"}

[B4nan]

FYI this is just a constraint, it works just fine with node 16.

[wojtekKrol]

Yes it does works after adding overrides or am I missing something?

[B4nan]

it does work without them too, you just need to force install to ignore that constraint. got is still compatible with node 16, they just dropped the official support and made the engines check more strict, but codewise there is nothing that would require node 20 or node 18. our CI works just fine with node 16.

[wojtekKrol]

I understand now that the code itself doesn't strictly require Node.js v20 and that the constraint is more about official support. Unfortunately, using the --force flag isn't an option for our production pipelines due to best practices and policy constraints.

Thank you for the clarification.