EPERM error on Windows when multiple processes try to write the same file

[asapach]

When write-file-atomic is used by multiple processes (e.g. in worker-farm) it can run into a race condition where the same file is being written/renamed concurrently. On Windows it causes the following error:

Error: EPERM: operation not permitted, rename '...' -> '...'

#22 seems to address this issue for async writes within the same process, but the issue still exists for parallel sync/async writes.

Here's a repro that demonstrates the problem: https://github.com/asapach/write-atomic-issue

Original issue: jestjs/jest#4444

[jeanlauliac]

Serializing writes does not solve the issue, because one of the main benefit of write-file-atomic is to work across multiple parallel processes. If rename is not an atomic operation at the OS level, then the rename strategy is not viable anymore.

Perhaps there are other elements we can leverage on Windows to make atomic writes. For example, the low-level rename API might have a flag to make it atomic.

Alternatively, we can mitigate the issue by catching EPERM errors, and trying again for the same file a few milliseconds later. This approach could quickly get out of control I'm afraid, as writes could get delayed by a long time and processes constantly interrupting each others writes.

[asapach]

Upstream fix: isaacs/node-graceful-fs#119

[mekwall]

@jeanlauliac Renames on Windows are not atomic and we would probably need to implement it ourselves. Node uses MoveFileEx function for renaming on Windows and there's a flag called MOVEFILE_WRITE_THROUGH which node doesn't currently use, which might make it better. Just a theory though as I have no real insight into the Windows File API.

[Knagis]

I made the following change locally to solve the problem (using jest, had to restart many times until everything was properly cached). Not ideal but helped to mitigate the immediate problem.

    try {
      fs.renameSync(tmpfile, filename);
    } catch (err) {
      if (process.platform !== "win32" || err.errno !== -4048 || !fs.existsSync(filename)) {
        throw err;
      }
    }

[iarna]

I intend to loop back on this one soonish and see if the repro is still an issue with some of the latest fixes (particularly those that fix filehandle leaks, that I expect exacerbated this on Windows). (If you find yourself looking at this issue and I haven't gotten back to it yet, I would be eternally grateful if you gave a go a reproing it yourself!)

[ehmicky]

@iarna I've just tried the demo repo it with Node 12.1.0 and the latest version of all dependencies, and this is still an issue.

For information, this sometimes create issues when using nyc together with ava on Windows.