Severe degradation of performance

[bghtrbb]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

Here I have 2 performance screen shots a 1 week and 1 hour closeup that is running in the cloud. As you can see there is a severe decline in performance the longer it run.

Explanation for the flat section....
The flat part you see at the end of the graph is from when the container kept shutting down due to issue #2085. Then the line ends when I shut it off.

But once I fixed that you can see the performance continues at the same poor performance. So the 1 hour close up is from the last hour after I got it back and running. 5+ minute stops.

At this point there are around 190,000 results.

1 Week

1 Hour Closeup

Code sample

No response

Package version

3.5.3

Node.js version

18.16.0

Operating system

Linux On ECS Fargate

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

Hi there, can you provide some reproduction for this? We can't help with screenshots, for performance issues, it's crucial to have a complete reproduction.

The crawlers are by default stateful, we are just finishing with new guides about deploying crawlee projects to AWS lambda and similar, so be sure to check those out, they will help you understand what needs to be done for stateless crawling: #2084