file corruption virtioFS and gRPC bind mount 4.36.0

[ltd]

Description

File is corrupted on successive bind mounts

Reproduce

#virtiofs
mac> FILEPATH=$PWD/f
mac> echo "abc" > f
mac> docker run --rm -it -v /:/foo alpine cat /foo/host_mnt/$FILEPATH
abc
mac> echo "123abc" > f
mac> docker run --rm -it -v /:/foo alpine cat /foo/host_mnt/$FILEPATH
123a

#gRPC Fuse
mac> FILEPATH=$PWD/f
mac> echo "abc" > f
mac> docker run --rm -it -v /:/foo alpine cat /foo/host_mnt/$FILEPATH
abc
mac> echo "123abc" > f
mac> docker run --rm -it -v /:/foo alpine cat /foo/host_mnt/$FILEPATH
abc

Expected behavior

the file should contain 123abc inside the container

docker version

Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:38:18 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.36.0 (175267)
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:19 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/ltd/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     /Users/ltd/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     /Users/ltd/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/ltd/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/ltd/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/ltd/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/ltd/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/ltd/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/ltd/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ltd/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     /Users/ltd/.docker/cli-plugins/docker-scout

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 48
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.751GiB
 Name: docker-desktop
 ID: 9102edb3-68f1-4017-ad83-c4252c4fc753
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 93
  Goroutines: 134
  System Time: 2024-11-27T13:28:36.862796176Z
  EventsListeners: 11
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/ltd/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

9BC97F9B-3CBB-432A-8F9F-8228CACF15D1/20241127133547

Additional Info

No response

[christianseel]

I got the same issue with 4.36.0. I did a downgrade to 4.35.1 and I'm still able to reproduce the issue with the steps mentioned above.

Downgrading to 4.34.3 (170107) appears to be a successful workaround for now.

[ltd]

I got the same issue with 4.36.0. I did a downgrade to 4.35.1 and I'm still able to reproduce the issue with the steps mentioned above.

Downgrading to 4.34.3 (170107) appears to be a successful workaround for now.

Thanks, I reverted back to 4.32 my previous version, I just assumed this was introduced in 4.36. It's a show stopper for my workflow.

[djs55]

@ltd I notice you're bind-mounting the root of the internal helper VM and accessing files via the /host_mnt path. Normally we would expect the docker run to use host paths which sets up caching and invalidation, for example: (note the .)

% echo "abc" > f
% docker run --rm -it -v .:/foo alpine cat /foo/f
abc
% echo "abc123" > f
% docker run --rm -it -v .:/foo alpine cat /foo/f
abc123

Could you describe the use-case you have for the /host_mnt path?

[ltd]

@djs55 , I'm taking multiple paths on the command line as arguments, it's way more straight forward to access them under host_mnt, rather than have to make a mount for each one. At the time of running, I don't even know which args are paths, and the paths point to files or directories.

[djs55]

Thanks for the response.

One of the goals of Docker Desktop is to behave as similarly as possible to "native" dockerd on Linux, where docker run -v is ultra-quick because it's implemented by a Linux "bind mount" so container file accesses incur almost zero overhead. Unfortunately on Mac and Windows we have to use a helper Linux VM to run Linux containers because macOS and Windows don't support the Linux ABI natively. Since there are two separate kernels running on the same machine (macOS and Linux), we can't use a "bind mount" and have to use a remote filesystem to allow the Linux containers to access files on the host. Unfortunately every filesystem operation is therefore slower because of all the additional context switching between the Linux VM (the filesystem client) and the host (the filesystem server). To mitigate the performance loss we turn on as much caching as we can, to reduce the number of remote filesystem calls needed. Since we cache filesystem contents in the VM we also need to ensure the cache is invalidated when the file changes. The OS APIs for monitoring for file changes aren't capable of watching the whole disk -- there are too many changes always happening on a machine and the queues overflow. Therefore Docker Desktop only watches directories explicitly mentioned on the docker run -v command-line. The / directory is special and actually refers to the helper VM, as a convenience so it's possible to use docker run -v /var/run/docker.sock:/var/run/docker.sock to expose the Docker API itself to containers.

I think the phenomenon you observe can be explained by

1. to increase the performance (to get closer to "native" performance): we've increased the caching in recent versions
2. the internal path in the VM /host_mnt was exposed by accident: it's not visible on "native" dockerd, which is the reference implementation
3. by using the internal path /host_mnt in a docker run -v, the cache invalidation is bypassed so the container sees stale caches

The only solution I can think of is to make Docker aware of the "important" directories whose contents should be monitored for changes, and add these as docker run -v arguments. It's fine to use parent directories, for example docker run -v /Users/me/myprojects:/myprojects will also see changes in all child directories e.g. /Users/me/myprojects/a/b/c/1/2/3/...

We also recommend to avoid using docker run -v $HOME:/foo, as the Mac continually writes to paths such as $HOME/Library/Containers/com.apple.Safari/... which generates a lot of events, and therefore cache invalidations, which costs CPU.

If possible I recommend avoiding /host_mnt and using a list of docker run -v /Users/me/foo:/foo -v /Users/me/bar:/bar, as /host_mnt really an implementation detail which isn't portable.

[ltd]

@djs55, thanks for the explanation.

[COil]

Hello, just to confirm the issue. A downgrade to 4.34.3 fixed the problem.
It also prevents the code to be updated.
For example, if we add a die('foobar1'); in the main front controler of a PHP application, the debug is not displayed, the old code is used (as if it was cached somewhere) until the application container is restarted.