File IO race condition when restarting pigeons with MPI

[dominic-chang]

Hi, I recently tried to restart a pigeons run that I had going previously by firing up multiple processes that were each running.

pigeons(pt.exec_folder,
    on = Pigeons.MPIProcesses(
        n_mpi_processes = 20,
        walltime="10-00:00:00",
        n_threads = 24,
        dependencies = [
            Pigeons, 
            preamble_path
        ],
        mpiexec_args=`--mpi=pmi2`
    )
)

This resulted in the following error

ERROR: LoadError: IOError: rm("results/latest"): not a directory (ENOTDIR)
Stacktrace:
 [1] uv_error
   @ ./libuv.jl:100 [inlined]
 [2] rm(path::String; force::Bool, recursive::Bool)
   @ Base.Filesystem ./file.jl:307
 ...

I assume it's because each process is trying to remove the results/latest folder all at the same time. I'm not sure if this is a bug, or if I used the API incorrectly.

[alexandrebouchard]

Thanks for reporting! It seems to occur with "rm("results/latest", force = true)" which according to Julia doc one have thought to be OK (" If force=true is passed, a non-existing path is not treated as error.") but maybe because of multithreading as you said it might not behave as advertise. I will push a fix momentarily....