Reduce risk of loosing/currupting data

[liquidnya]

If the software or the PC crashes the queue might have been in the middle of writing data to the file system, which could lead to a corrupted file.

To prevent this, the queue could write the data to a shadow file, wait for a file system sync, and then atomically rename the new file to the actual file name.

A solution to the problem could be to use this algorithm: https://stackoverflow.com/a/7957634
But with the following changes:

• wait for a file system sync after writing the file and before renaming the files
• make sure to use an atomic file rename
• use locks around the algorithm, like for example use locko: https://www.npmjs.com/package/locko

[liquidnya]

I am going to implement this like suggested here for a different project: simonlast/node-persist#94 (comment)
Using the libraries https://github.com/isaacs/node-graceful-fs and https://github.com/npm/write-file-atomic.

[skybloo]

Another option that would probably be simpler to implement would be using the database as a pseudo application directory as in this article, which would provide atomic updates more or less for free, and avoid having to go through the whole file to read/write state. The shadowing would probably be somewhat more performant overall, just requires more work and file system stuff.

[liquidnya]

I think the suggestion to use a database with a pseudo application directory is very good, since it will greatly improve performance.
Changing the write from just a regular write to an atomic write was actually quite easy by using a library that does exactly that, which is currently implemented by #22. The most work was to move all the code in a new source file, and to convert the old save file format to the new one.

[liquidnya]

Fixed by #22.
This could be further improved with the following:

• Rewrite the queue code that accesses wait time to use the object stored in the save file instead of using 2 lists.
• Use writeFileAtomic instead of writeFileAtomicSync
  I have this branch that implements this already: https://github.com/liquidnya/fluid-queso-queue/tree/async-save
  However the tests don't work correctly, since simulating time does not work well with using async IO in the async timer.
  The tests are passing, but only because the test timer interval is set to 1_000 ms instead of 60_000 ms.
• Implement a pseudo application directory using a database (Reduce risk of loosing/currupting data #12 (comment))

However, the underlying issue of loosing data is solved, and all these improvements are performance improvements only.