"dotnet build" fails to run reliably when run from within a Dokany filesystem

[hach-que]

Environment

• Windows version: Windows 11 (10.0.22000)
• Processor architecture: x64
• Dokany version: 2.0.6.1000
• Library type (Dokany/FUSE): Dokany

Check List

• I checked my issue doesn't exist yet
• My issue is valid with mirror default sample and not specific to my user-mode driver implementation
• I can always reproduce the issue with the provided description below.
• I have updated Dokany to the latest version and have reboot my computer after.
• (there are no bug fixes in Git since 2.0.6.1000, so installing a snapshot felt redundant)

Description

I have been trying to write a filesystem for the past week or so where the use case involved running "dotnet build" from within the filesystem; that is, both dotnet.exe, the .NET SDK and the project to build are all within the Dokany filesystem. Unfortunately over the course of doing this I started encountering subtle bugs where the loaded .NET assemblies would be corrupt or otherwise invalid. I then tried porting my efforts to WinFSP with no luck.

To isolate the issue, I started putting together a test suite that would test virtual filesystems using their official samples. Both Dokany and WinFSP fail in different ways; Dokany's failure case seems to manifest with incorrect data reads, but typically only the first time the data is read.

The test suite I have written is available at https://src.redpoint.games/redpointgames/msbuild-fstest and can be used to reproduce the issues. I've been able to reproduce the issues in a Windows 11 VM (so they're not environment specific), using both the C++ and .NET mirror samples. If you need instructions on how to set up a VM to reproduce the issues, see here (though you will want to install the Dokany driver instead of the WinFSP driver).

The latest results of the test suite are available online here: https://msbuild-fstest.redpointgames.dev/

[Liryna]

Hi @hach-que ,

Dokany's failure case seems to manifest with incorrect data reads, but typically only the first time the data is read.

How have you been able to determine it was a read issue ? Running procmon when it fails could help narrow the faulty workflow (with the sample logs).

Is it also flaky with the memfs sample ? (so far I got 10/10 with the mirror tests)
If your VM has a small number of Threads, does allocating more improve the situation ?

[hach-que]

@Liryna I can replicate the issue on my own Windows machine, 4 build servers and a fresh virtual machine (the latter to eliminate any environment issues). The reason I suspect it's a read issue is the type of errors that dotnet build is emitting to the console - things like the loaded assembly is invalid, or that the executing program performed an invalid instruction (if the executable code loaded into memory from reading the file via Dokan wasn't intact, this is how it could manifest).

I wasn't able to get the memfs sample working at all - I simply get [2022-12-12 04:01:48.058] [error] dokan_memfs failure: Driver something wrong.

(Of note, #1110 looks rather suspicious; I'm not sure of the specifics with .NET loading assemblies into memory, but if it does use memory-mapped I/O, then the subtle corruption in #1110 would likely exhibit the behaviours I'm seeing)

[hach-que]

This comment (dotnet/runtime#62391 (comment)) suggests that the .NET runtime does use memory-mapped I/O for assemblies. Fixing #1110 first might be the best course of action.

[boAtLimbic]

@hach-que I'm also experiencing similar issue when compiling over NFS. I see you also are experiencing issues with WinSCP, which I have yet to probe.

I'm trying to compile over NFS on centralised storage with the Unreal Engine. I would like to collaborate, if you're interested.

[Vinny636]

It's interesting that the .NET filesystems work but the native C++ ones do not. I wonder if the additional latency involved with marshalling between native and .NET is allowing your tests to pass?

Sorry that I'm commenting on an issue that's more than a year old, but I'm embarking on a filesystem project as well and I'm curious about failures when dealing with memory-mapped files.

[hach-que]

I ended up moving over to WinFSP, which has been far more reliable for writing Windows filesystems in user space. A similar issue that affected WinFSP was fixed within a few months with a pretty good response from the developer.

I'm going to close this issue as it doesn't look like it's going to be actioned in Dokany and I don't plan on switching away from WinFSP.

[Liryna]

Glad you were able to find a solution that works for you! Dokan is reliable and there are projects and enterprises that have been using it for years at large scale that proves it. This seems to be a niche issue that had a low priority in my TODO sorry. If anyone would like to take a look, I would be happy to give a hand.