Memory overloaded: overestimated used memory

[galaczi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Issue description

I am running my script with about 60 concurrency. After a while I am getting this warning:

WARN PlaywrightCrawler:AutoscaledPool:Snapshotter: Memory is critically overloaded. Using 20290 MB of 22444 MB (90%). Consider increasing available memory.

However, running free -h shows this:

               total        used        free      shared  buff/cache   available
Mem:            31Gi        14Gi        10Gi       169Mi       6.1Gi        16Gi
Swap:          4.0Gi          0B       4.0Gi

So in fact memory is not overloaded. The issue prevents opening new requests.

INFO PlaywrightCrawler:AutoscaledPool: state {"currentConcurrency":55,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":false,"memInfo":{"isOverloaded":true,"limitRatio":0.2,"actualRatio":1},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}

Code sample

No response

Package version

^3.1.3

Node.js version

^3.1.3

Operating system

Ubuntu

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

I am not sure how relevant is the information from free, we need to measure what the JS engine sees, and things can get quite complicated based on the environment you are using.

How/where do you run the crawler, is it inside a docker image, apify platform, or something else? We will be happy to verify, but you haven't provided any reproduction really.

If you see this locally, you could try logging what happens in here.

[galaczi]

I am sorry I cannot provide a reproduction. I have thousands of lines of code and long running tasks.

I am running it on an Ubuntu server. Just tested it. With 12 Gigs of used memory out of 32 Gigs it reports:

INFO  PlaywrightCrawler:AutoscaledPool: state {"currentConcurrency":67,"desiredConcurrency":33,"systemStatus":{"isSystemIdle":false,"memInfo":{"isOverloaded":true,"limitRatio":0.2,"actualRatio":1},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}

Is it that the limitRatio is set to 0.2? I have set a higher ratio though, like this:

const config = Configuration.getGlobalConfig()
config.set('availableMemoryRatio', 0.95)

Maybe that's not respected for some reason?

[galaczi]

I also tried with crawlee.json:

{
  "maxUsedCpuRatio": 0.95,
  "memoryMbytes": 22000
}

After using about 10G of memory the actualRatio jumps from 0 to 1 and the memory is overloaded.

[galaczi]

Here is a reproduction:

https://github.com/galaczi/crawlee-memtest

At around 60 concurrency the memory flips to overloaded, even though about 6-8 GB of actual memory is used.

[3rd]

Hitting the same issue in Docker on a VPS but not on local 👀

[B4nan]

There are some improvements in v3.10 that could help (especially for running on a VPS instead of the apify platform), be sure to update if you are using older version.