VirtioFS: File size in bind mounted volume does not match host

[lorenzodalaqua]

Description

This might have some similarity to issue #7494.

The development setup where I work uses a project volume shared by multiple containers. The volume is also bind mounted to a directory on the host using mount options:

docker volume create \
    --driver local \
    --opt type="none" \
    --opt o="bind" \
    --opt device="$HOST_DIRECTORY" \
    "$VOLUME_NAME"

This has worked fine in Docker Desktop on Mac, using VirtioFS (and the "Apple Virtualization Framework"), with Docker Desktop up until version 4.32. The volume is synced any host file changes due to being bind mounted.

When upgrading to version 4.33 we started having issues with files being "truncated" when accessed by containers when there are changes to the file on the host. The contents are updated on the container, but do not fully match the host.

For example, a file with contents: ABCDEFG on the host, after adding some characters to it, (e.g. making the contents 123ABCDEFG), when accessed by the container through the volume, will be truncated (e.g. 123ABCD), with the amount of characters added being absent at the end of the file.

By checking with ls on both the container and host, I was able to determine the file size is not updated correctly on the volume, but the contents are.

Looking at this comment on the issue I referenced above, I suspect this might also be related to VirtioFS cache invalidation not being triggered. If I run this sequence of commands from inside a container mounting the volume:

cat file.txt
# Get corrupted/truncated contents.

ls
cat file.txt
# File content is no longer truncated and matches the host.

Anyway, I was able to reproduce it consistently with the following script:

Reproduce

VOLUME_NAME="testing-volume"
DIRECTORY="directory"
FILE="test.txt"

docker volume create --driver local --opt type="none" --opt o="bind" --opt device="$(pwd)" "$VOLUME_NAME"
mkdir $DIRECTORY
echo "123456789" > $DIRECTORY/$FILE
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine cat $DIRECTORY/$FILE
# Output is "123456789"

echo "ABC123456789" > $DIRECTORY/$FILE
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine cat $DIRECTORY/$FILE
# Output is "ABC1234567%" (% representing no end of line character on my terminal, as the contents are truncated)

docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine ls -la $DIRECTORY/$FILE
# Output for the file is: "-rw-r--r--    1 root     root            10 Dec 10 14:03 directory/test.txt", notice 10 characters when it's now 13
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine ls -la $DIRECTORY
# Output for the file is: "-rw-r--r--    1 root     root            13 Dec 10 14:04 test.txt", size is updated

docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine cat $DIRECTORY/$FILE
# Output is "ABC123456789", the correct contents on the host

Expected behavior

The file in the bind mounted volume should match the file on the host on every change.

docker version

Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:38:18 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.36.0 (175267)
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:19 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     /Users/lorenzo/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 15
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.653GiB
 Name: docker-desktop
 ID: 1e75072f-7d8f-47c3-917a-43dc08d31755
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/lorenzo/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

4EB0271E-BBC1-4A04-87BB-E7F66581011B/20241210141531

Additional Info

This behavior happened only with VirtioFS, the same setup when run with the "osxfs (Legacy)" file sharing implementation had no issues.

This might be of note: when I attempted running the same reproduction script above with the "Docker VMM (Beta)" Virtual Machine Manager, the file is never updated on the volume after changes on the host, it was not corrupted/truncated, it is just never updated.

[CharlesParent]

Hello,
I experience the same issue using Docker Desktop 4.37.0 (178034)
I confirm that switching from VirtioFS to osxfs (Legacy) solved the issue regarding file sharing

On top of what @lorenzodalaqua said, I encountered a strange issue where files where not truncated, but changes in files done on my IDE was not directly reflected in the docker container (I have a flask app with gunicorn that should reload on file changes, but no reload was happening after modifying the code).
Stranger, I discovered that, if I am in the container and I do a ls in the folder in which code changed, reload was happening at this moment.
It might indeed be related to VirtioFS cache invalidation not being triggered

[wkirby]

Checking in with the same problem. MacOS Sequoia 15.1.1, 2021 M1 Max. I suspected this might be something related to inotify and/or atomic save; the issue is persistent across Sublime Text and VSCode, but intermittent when files are edited with vim.

🤷

docker info

Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/wyatt/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     /Users/wyatt/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     /Users/wyatt/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/wyatt/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/wyatt/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/wyatt/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/wyatt/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/wyatt/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/wyatt/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/wyatt/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     /Users/wyatt/.docker/cli-plugins/docker-scout

Server:
 Containers: 29
  Running: 20
  Paused: 0
  Stopped: 9
 Images: 24
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 16.07GiB
 Name: docker-desktop
 ID: a42c5b6e-dfc2-4245-9d34-b5b43ff43dd7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/wyatt/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

docker version

Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:38:18 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.36.0 (175267)
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:19 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0