WSL2 + Docker causes severe memory leaks in vmmem process, consuming all my machine's physical memory

[davidr-PA]

Version

Microsoft Windows [Version 10.0.19044.1889]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.10.102.1

Distro Version

Ubuntu-20.04

Other Software

Docker Desktop: 4.11.1

Repro Steps

You simply need to build docker images several times. It results in this:

Note: I don't even have any active Docker containers running. Everything's terminated.

It seems to be the same as this issue, reported 3 years ago and still unresolved: #4166

I never encountered this on MacOS, but it's slowing my development speed down greatly on Windows. It definitely gets worse the more times I build Docker images. It's resulting in build processes slowing down to somewhere between 1/10th and 1/50th the speed they run at when physical memory is available. It makes me need to restart my computer sometimes multiple times per day.

Expected Behavior

Memory gets released after processes terminate.

Actual Behavior

Memory does not get released, resulting in my computer using virtual memory and slowing to a crawl.
Resource usage (with no active containers running - all terminated):

Diagnostic Logs

No response

[labanv]

I have the same issue only by running a few fixed docker images. Within 2 days the memory is 100% used only stopping Docker Destop releases the memory. So I have switched back to Docker Desktop 4.11.0

[kkruit]

This is also happening to our team. Seems to have started for us last week. It appears to be maxing out our WSL memory allotted also, as the total calculated processes memory in task manager was not equal the total memory being used.

[Drol]

Same here on Windows 11 and Docker Desktop 4.11.1. Running asp.net containers from Visual Studio.

[julestop]

This is also happening to our team. Seems to have started for us last week. It appears to be maxing out our WSL memory allotted also, as the total calculated processes memory in task manager was not equal the total memory being used.

Same for me. Memory in task manager doesn't add up to total. Limiting the wslconfig file does limit the memory af the task, however, it's like the memory doesn't get returned and after 2-3 days or runtime my computer hits 100% memory usage and I have to reboot.

[LiamKarlMitchell]

Please fix.

[chrisb2244]

Also saw this, unsure if was related to unstarted containers - resource monitor showed ~6 instances of vmmem with 1GB committed and 0KB working set (and one or two with both committed and working, looking more functional).
Task manager showed 99% RAM used, but no really large consumers (just things like Chrome, JDK, Teams, one functioning(ish, given the RAM usage) vmmem, all in 100s MB).

Also running 4.11.1.

@labanv, did rolling back help you?

[Ador-able]

Same here on Windows server 2022 and Docker Desktop 4.11.1. After a few days the server memory takes up 100% the total calculated processes memory in task manager was not equal the total memory being used.

[CurtisVermeeren]

Same issue on Windows 10 Docker Desktop v4.11.1. Left my laptop for a bit and came back to 100% memory usage with 16GB and only one image running.

[davidr-PA]

This is also happening to our team. Seems to have started for us last week. It appears to be maxing out our WSL memory allotted also, as the total calculated processes memory in task manager was not equal the total memory being used.

Same for me. Memory in task manager doesn't add up to total. Limiting the wslconfig file does limit the memory af the task, however, it's like the memory doesn't get returned and after 2-3 days or runtime my computer hits 100% memory usage and I have to reboot.

Nice, I also limited memory allocation in wslconfig. It helped, but doesn't eliminate the issue (it just allows me to get away with building more docker containers, before my machine starts grinding to a halt as it dips into virtual memory).

[JamienAU]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

[labanv]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

for me it works, memory usage is still increasing (some leaks) but sirtainly not as fast as it would hit 100% in 4.11.1 it must be a combination bug of windows vmmem and docker desktop

[julestop]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

Are you back on 4.11.0 or 4.10.0? Looks like you and @labanv are on different versions. I went back to 4.11.0, and seems like my memory still creeps up, maybe more slowly though.

[labanv]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

Are you back on 4.11.0 or 4.10.0? Looks like you and @labanv are on different versions. I went back to 4.11.0, and seems like my memory still creeps up, maybe more slowly though.

I'm back at 4.11.0 but as I say it also increases memory usage but slowly.

[julestop]

I just rolled back from 4.11.0 to 4.10.0 and rebuilt everything, will report back.

[JamienAU]

Have been monitoring memory usage for 48 h now and it's looking good, RAM usage is at the expected level and hasn't creeped up.

[ben186]

Probably related and it will be fixed soon: docker/for-win#12877

[mrgreywater]

Also duplicates #8703

[jonny-circles]

Hi mrgreywater,

I have a fresh install of both WSL and Docker 4.12 and this issue still persists.

[davidr-PA]

@mrgreywater I re-installed with Docker Desktop 4.12, and I still seem to be encountering memory leaks when I build Docker images. Will continue to test as I will definitely be building more Docker images soon - and I will confirm if I see otherwise - but as of now this appears to still be active.

EDIT:
Docker version is:

docker --version
Docker version 20.10.17, build 100c701

[ivsokol]

for me process is based on:

• stopping docker desktop
• running wsl --shutdown
• manually restarting LxssManager service - this frees up all memory
• starting docker desktop
• do it again after couple of hours, when memory is again almost full

[thangnq1001]

Try these steps:

1. in C:\Users\<yourusername>, create .wslconfig file (skip if already there)
2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

• Check if the file is in UTF-8
• Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run wsl --shutdown then Restart-Service LxssManager
4. Run things again. Type wsl to go into wsl, then run free -mh and check total value, it should be 1GB as configured above

---

Updated: Some folks below (#8725 (comment)) had couple of additional configs in the file that worked for them (kernelCommandLine and autoMemoryReclaim).

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

[experimental]
autoMemoryReclaim=gradual

[albcunha]

Try these steps:

1. in C:\Users\<yourusername>, create .wslconfig file (skip if already there)
2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

Check if the file is in UTF-8. Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run Restart-Service LxssManager
4. Run things again. Inside wsl, run free -mh and check total value, it should be 1GB as configured above

Finally!!! This worked great!! Thanks a lot! It was driving me crazy!!!

[riccardogabellone]

Try these steps:

1. in `C:\Users\<yourusername>`, create `.wslconfig` file (skip if already there)

2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

Check if the file is in UTF-8. Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run `Restart-Service LxssManager`

4. Run things again. Inside wsl, run `free -mh` and check `total` value, it should be 1GB as configured above

Great man! Thank you!! Works like a charm for me in Windows 11, WSL2 + Docker

One more check I did:

• With no .wslconfig file enter the wsl e see cpus and ram in htop command, actual settings
• After wsl --shutdown, adding that configs and entering the wsl again, htop outputs wanted settings

Anyway, here is official doc

[chabad360]

This should not be the official recommended fix, a memory leak is a memory leak and it needs to be fixed.
I'm not interested in restarting whenever I want to do something more memory hungry.

[LiamKarlMitchell]

Well there are worse issues, when I copy files with Explorer sometimes it locks up all processes/ide etc and I can't even end task them.

[wonson]

Similar issue here.

I have a script with curl dl in a loop, multiple bg process.
I investigated the mem within linux with 'top', and the ram is perfectly fine.

But the ram for Vmmem in Win10 side keeps increasing

[jorgegarcia-ukg]

Same issue here. Running version 4.13.1 (90346).

[ifigueroap]

Same issue here

[themegabyte]

I guess 32Gb ram only cuts about 30 minutes of docker building. I should get 256GB before this is fixed since I tend to develop and test builds for more then 7 hours. Any deals around ya'all?

Edit: I figured my comment was not very helpful.

To add to some context, when I am running VSCODE and normal containers, this is not that much of a problem.

Its very noticeble when I am building containers for my staging builds. It eats way quickly.

Edit 2: I have moved to VMWARE for now. It has much better memory managmenet then WSL2.

[BastLast]

same issue there, I have to restart wsl 2 times a day...

Windows 10 + docker desktop 4.13.1

[huyz]

Like some others in this thread, my particular memory issue manifests principally in the Active column of the Page Table, as shown by RamMap—not Vmmem's Memory usage in in Task Manager.

After a while, I have to reboot.

Anyone know if this is the same issue or just a separate issue as the others?

[xfl12345]

I found a cause. That is an extension of VS code and the ID is ms-vscode-remote.remote-containers . After I disabled the extension and rebooted, there was finally no more memory leak.

[huyz]

Like some others in this thread, my particular memory issue manifests principally in the Active column of the Page Table, as shown by RamMap—not Vmmem's Memory usage in in Task Manager.

After a while, I have to reboot.

Anyone know if this is the same issue or just a separate issue as the others?

To answer my own question, the issue with a ballooning Page Table seems to be separate and is discussed in https://superuser.com/a/1841517/79987 (with potential culprit being AMD) and docker/for-win#13929

[ntauth]

Had to switch back to Hyper-V because because Docker + WSL2 is rendering my Razer Blade 15 unusable. Using VSCode devcontainers makes it worse.

[shanecranor]

There exist a simple approach to prove this bug: When I quit WSL2 using command wsl --shutdown then the memory is STILL full; so it is a bug on a whole system level, and not just WSL

I am having this exact issue. Also running AMD CPU and iGPU/dGPU, but disabling iGPU is not an option for me since I'm working on a laptop.

[huyz]

Also running AMD CPU and iGPU/dGPU, but disabling iGPU is not an option for me since I'm working on a laptop.

Did you try downgrading AMD Adrenaline to v23? Worked for me

[Eboubaker]

WSL: Windows Subsystem For Linux ❌
WSL: RAM eating simulator ✔

[TaggartMaher]

I had this problem so i got 96gb of ram and now the problem is gone.

[afor84]

The problem is still there after 2 years:
WSL2

Docker Desktop

MS Windows 10 Enterprise

[afor84]

I found a cause. That is an extension of VS code and the ID is ms-vscode-remote.remote-containers . After I disabled the extension and rebooted, there was finally no more memory leak.

Hi, I do not have Visual Studio installed, but I am experiencing the memory leak problem too. With VS code do you mean Visual Studio ? I am new with WSL. Right now I worked with Docker Engine in Linux.
Any feedback is appreciated. Thanks

[xfl12345]

I found a cause. That is an extension of VS code and the ID is ms-vscode-remote.remote-containers . After I disabled the extension and rebooted, there was finally no more memory leak.

Hi, I do not have Visual Studio installed, but I am experiencing the memory leak problem too. With VS code do you mean Visual Studio ? I am new with WSL. Right now I worked with Docker Engine in Linux. Any feedback is appreciated. Thanks

It doesn't matter.
"VS code" is "Visual Studio Code"
"VS code" is not equal to "Visual Studio"
They are different product.

[maou-shonen]

Hi folks, on the latest pre-release WSL versions we've made the 'dropCache' option on by default which should help address this use case. Please upgrade using wsl --update --pre-release and you'll get the autoMemoryReclaim=dropCache behaviour on by default and this should go a long way to fixing it! Thanks.

In my environment, even explicitly enabling it didn't improve the problem.😢

[GeorgeDeac]

It also happens to me, but here the weird part is that the leaked memory seems to be unmanaged cause it's not reported as being part of a process anywhere.

[Slyke]

It's becoming so bad that I need to reboot almost daily. I have 64gb of ram. 25% is used on start up. Within 24 hours, it's at 80% being used. This only happens after opening VSCode and using with WSL2.

[xfl12345]

It's becoming so bad that I need to reboot almost daily. I have 64gb of ram. 25% is used on start up. Within 24 hours, it's at 80% being used. This only happens after opening VSCode and using with WSL2.

1. Disable the extension of VS code which the ID is ms-vscode-remote.remote-containers .
2. Roll back Docker Desktop version to 4.33.0 (version 4.30.0 is also fine). (It is not recommended to directly install the old version without first uninstalling the currently installed one.)
3. Reboot.

And the previous version of Docker Desktop can be downloaded from the following link as below.
Docker Desktop Release Notes 4.33.0

Addons:

• Docker Desktop version 4.34.2 (167172) is OK

Enjoy :)

[afor84]

As written above (#8725 (comment)), I resolved by creating or editing the .wslconfig file.

[shanecranor]

In my case it was an AMD driver issue, and downgrading my drivers solved the problem without making any changes to vscode or docker. There was no task in task manager taking up the memory.

[andrsmllr]

@shanecranor Do you recall which driver you downgraded from which problematic version to which working version?

[huyz]

@shanecranor Do you recall which driver you downgraded from which problematic version to which working version?

See https://superuser.com/a/1841601/79987

[csckcac]

It's becoming so bad that I need to reboot almost daily. I have 64gb of ram. 25% is used on start up. Within 24 hours, it's at 80% being used. This only happens after opening VSCode and using with WSL2.

Try to edit C:\Users\<username>\.wslconfig, create if it is not found

# Settings apply across all Linux distros running on WSL 2
[wsl2]
# Limits VM memory to use no more than 4 GB, this can be set as whole numbers using GB or MB
memory=60GB
swap=40GB
kernelCommandLine="sysctl.vm.swappiness=10"

then run wsl --shutdown and reopen everything
you may tune the 60/40 value to suit your need
default 60 swappiness is not correct in WSL IMO.

[moracabanas]

It's becoming so bad that I need to reboot almost daily. I have 64gb of ram. 25% is used on start up. Within 24 hours, it's at 80% being used. This only happens after opening VSCode and using with WSL2.

Try to edit C:\Users\<username>\.wslconfig, create if it is not found

# Settings apply across all Linux distros running on WSL 2
[wsl2]
# Limits VM memory to use no more than 4 GB, this can be set as whole numbers using GB or MB
memory=60GB
swap=40GB
kernelCommandLine="sysctl.vm.swappiness=10"

then run wsl --shutdown and reopen everything you may tune the 60/40 value to suit your need default 60 swappiness is not correct in WSL IMO.

In my case if you hit the swap because your RAM is hoarded by WSL, then your performance is crushed and your Disk IO starts to get your NVME durability until you restart wsl with --shutdown

[Isoris]

Same here. I fixed it by doing an update to the latest Nvidia drivers (365.90 i my case) with clean folder install. Then I also uninstalled VS code in windows. I am not sure which of the two solved the problem.

Previously there was 3-5 instances of Nvidia container and 2-3 instances of Nvidia shield and Shadow. And telemetry was quite high in RAM usage.

Now it's around 1 running process of each. And the virtual machine of WSL has a lower RAM usage. In addition to that the GPU is almost at 0-3% in the background but before it was peaking everytime to 30-40% or more in the background..

[GeorgeDeac]

Can confirm if you have phantom leaked RAM memory usage, it is most likely a combination of the VSCode container extension and the docker version.

[thaynes43]

@GeorgeDeac I am experiencing this but don't see anything as "ms-vscode-remote.remote-containers" but I do have the Dev Containers extension which I am pretty fond of. Is that the same? I'm not sure how to find the exact name like that.

Also, even when I quit docker desktop and shut it down and do wsl --shutdown the memory is not reclaimed. Vmmem process is gone but the memory flatlines wherever the leak left off and no other process appears to be using any memory so whatever is holding it is out of scope for Task Manager to report on. Right now, I can only reboot to reclaim the memory.

I'd love to be able continue using dev containers without having to reboot every other day, but I might get a head start on setting up a dev environment elsewhere for now. I have no doubt that is the cause since I just started using them and it just started leaking like crazy (hit 100% after three days).

[Isoris]

I feel like I have some sort of malware. But I don't understand what's going on to be honmest. Also WSLg installed by itself and my vhdx was destroyed there was only written data when I opened it. I am not sure if it's because I tried to mount the image with the wrong distribution after doing the fdisk shrink to reclaim disk space and I used 24.04 instead of 22.01. can it destroy your vhdx to do that?. And also I noticed Ubuntu was installed by itself with WSLg. Is it because I clicked Ubuntu on the Microsoft store? It was a few week ago and I was very tired. But my computer has been behaving super strange lately. Many events to ask for user privilege escalation, many Nvidia shield containers opened and Nvidia shadow like 4-5 processes each. Then there was a perioe with many remote desktop icons popping up for like half a second? But I had Linux opened (WSL but not WSLg at that time) and also Microsoft word !? Then I noticed that some of my files were erased and there was binary instead. Like for instance a text file that I used. Half of the file had binary inside instead? So weird. I don't really understand what's going on but it may just be a software problem because at that time I opened my firewall and Rdp to comnect from my laptop to my computer but I thought I had set up an ssh key? Super weird stuff. Especially that I found a Putty RND file on my documents and idk how did this come here? I don't even gave Putty. Lately the phone link app and LINE a desktop app like messenger that I use for work opens itself at start-up and reopens itself sometimes. But I find that the most strange was when I had VS code opened + NVIDIA containers of telemetry. So I updated the graphic pilots drivers to 365.90 and it solved the issue with WSLg completely.

…

On Mon, Oct 21, 2024, 9:17 AM Tom Haynes ***@***.***> wrote: @GeorgeDeac <https://github.com/GeorgeDeac> I am experiencing this but don't see anything as "ms-vscode-remote.remote-containers" but I do have the Dev Containers extension which I am pretty fond of. Is that the same? I'm not sure how to find the exact name like that. Also, even when I quit docker desktop and shut it down and do wsl --shutdown the memory is not reclaimed. Vmmem process is gone but the memory flatlines wherever the leak left off and no other process appears to be using any memory so whatever is holding it is out of scope for Task Manager to report on. Right now, I can only reboot to reclaim the memory. I'd love to be able continue using dev containers without having to reboot every other day, but I might get a head start on setting up a dev environment elsewhere for now. I have no doubt that is the cause since I just started using them and it just started leaking like crazy (hit 100% after three days). — Reply to this email directly, view it on GitHub <#8725 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASYS5TFZZQSNZMYKSXAXKY3Z4RP2JAVCNFSM56T3BJS2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TENBSGU2DCOBZHE4Q> . You are receiving this because you commented.Message ID: ***@***.***>

[csckcac]

Docker Version: 4.34.3 (170107)
Engine: 27.2.0
Compose: v2.29.2-desktop.2
Credential Helper: v0.8.2
Kubernetes: v1.30.2

wsl -v
WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26091.1-240325-1447.ge-release
Windows version: 10.0.19044.2788

I hope this picture helps someone to debug.

the docker desktop will then be crashed, I don't know it is wsl2 or docker issue, so I will post here anyway.

My docker and WSL is updated recently, com.docker.backend.exe is able to release the memory
so I think autoMemoryReclaim is partially working.

[GeorgeDeac]

@GeorgeDeac I am experiencing this but don't see anything as "ms-vscode-remote.remote-containers" but I do have the Dev Containers extension which I am pretty fond of. Is that the same? I'm not sure how to find the exact name like that.

Also, even when I quit docker desktop and shut it down and do wsl --shutdown the memory is not reclaimed. Vmmem process is gone but the memory flatlines wherever the leak left off and no other process appears to be using any memory so whatever is holding it is out of scope for Task Manager to report on. Right now, I can only reboot to reclaim the memory.

I'd love to be able continue using dev containers without having to reboot every other day, but I might get a head start on setting up a dev environment elsewhere for now. I have no doubt that is the cause since I just started using them and it just started leaking like crazy (hit 100% after three days).

Yeah I removed that, haven't had leaks since then. Had only one more leak when I was building a container with triton and nvidia stuff in it. Idk what wrong lately with windows docker..

[Isoris]

Maybe you can restart your computer everyday.. as a quickfix. Le lun. 21 oct. 2024 à 21:22, George Deac ***@***.***> a écrit :

…

@GeorgeDeac <https://github.com/GeorgeDeac> I am experiencing this but don't see anything as "ms-vscode-remote.remote-containers" but I do have the Dev Containers extension which I am pretty fond of. Is that the same? I'm not sure how to find the exact name like that. Also, even when I quit docker desktop and shut it down and do wsl --shutdown the memory is not reclaimed. Vmmem process is gone but the memory flatlines wherever the leak left off and no other process appears to be using any memory so whatever is holding it is out of scope for Task Manager to report on. Right now, I can only reboot to reclaim the memory. I'd love to be able continue using dev containers without having to reboot every other day, but I might get a head start on setting up a dev environment elsewhere for now. I have no doubt that is the cause since I just started using them and it just started leaking like crazy (hit 100% after three days). Yeah I removed that, haven't had leaks since then. Had only one more leak when I was building a container with triton and nvidia stuff in it. Idk what wrong lately with windows docker.. — Reply to this email directly, view it on GitHub <#8725 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASYS5TH5HANFAHOSGBSPDNLZ4UEYVAVCNFSM56T3BJS2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TENBSGY4DEOJTG44Q> . You are receiving this because you commented.Message ID: ***@***.***>

[mustermr]

Restarting the WSLService (ending the wslservice.exe process) works for me. Still not great, and can't use it for anything like a production solution because of this.