Pool Files on Levante

[ackerlar]

Describe the problem you are facing
It seems that some pool files are still missing on Levante:

WARNING: File not found: /pool/data/AWICM/FESOM2/MESHES_FESOM2.1/core2/tarfilesT63/input/echam6/T63CORE2_jan_surf.nc
2022-06-02 15:22:47.828132
WARNING: File not found: /pool/data/AWICM/FESOM2/MESHES_FESOM2.1/core2/tarfilesT63/input/echam6/T63CORE2_VGRATCLIM.nc
2022-06-02 15:22:48.194048
WARNING: File not found: /pool/data/AWICM/FESOM2/MESHES_FESOM2.1/core2/tarfilesT63/input/echam6/T63CORE2_VLTCLIM.nc

Did I miss anything or is there a plan to synchronize the pool directories of the different machines?

[mandresm]

Hi @ackerlar,

I am converting this into a Discussion, because I believe it is not an ESM-Tools problem, but a pool problem.

There are inconsistencies between the DKRZ pools (mistral on the past and now in levante) and the ones in ollie:

In ollie the directory built by default for ESM-Tools /work/ollie/pool/FESOM/meshes_default/core/tarfilesT63 exists.

In levante/mistral the folder /pool/data/AWICM/FESOM2/MESHES_FESOM2.1/core2/tarfilesT63 never existed, I could never find a tarfilesT63 in /pool/data/AWICM/FESOM2. There is one here /pool/data/AWICM/FESOM1/MESHES/core/tarfilesT63 but it's for FESOM1 so I have no clue what would that do.

I've seen people using always their own copies of the tarfilesT63 folder, specified from the runscript so all tells me that this is one of the inconsistencies across the pools. Eventually, someone will need to take care of making the pools consistent... I also don't know who to contact about the tarfilesT63 because I don't know who is responsible. Maybe this is the thread to start the discussion.

@tsemmler05, @chrisdane are any of you involved in the pools administration? What are the tartfilesT63 and where should they be in the DKRZ/ollie pools?

@pgierz and @denizural might be interested in following this conversation.

[ackerlar]

Thanks @mandresm! Yes, on mistral I also mostly used forcing and init files from some user directories. But that contradicts the idea of the pool directories.
Wouldn't it be possible to run an automatic synchronization regularly, like once a month or so?

[pgierz]

Yes. Someone tell me which version "the truth" is and I'll set something up

[ackerlar]

I just compared the files on Ollie for FESOM1 and FESOM2 from here: /work/ollie/pool/FESOM/meshes_default/core/tarfilesT63/input and here /work/ollie/lackerma/glacial-inception/nonstandard_pool_files/fesom/mesh_CORE2_finaltopo_mean/tarfilesT63/input
The first one being the pool directory, the second one being the files that I used for my AWI-ESM-2 experiments. They are all identical.

So, I guess it is safe to use FESOM1 pool files for FESOM2 simulations. Could someone maybe verify this? Then, I guess we would not need two redundant pool directories for FESOM1 and 2.

[tsemmler05]

Just wanted to note that I would be the person responsible for the content of the /pool directory on levante. Yes, I agree that we need to synchronise the files between ollie and levante. And if there are files missing in the different distributions then I need to know this to have a complete version of the files available. I am surprised to hear that FESOM1 and FESOM2 mesh files should be identical? At least for the CORE mesh there are some different grid points as far as I remember.

[pgierz]

@tsemmler05 this I also find rather surprising.

I will very certainly regret this but: I'm now in the HPC group, and "making pools clean" sounds like a job for the HPC people to release the scientists a bit in order to do more research. I could start taking care of the pools. As I already have sufficient permissions on AWI's machines, I suppose the important element is just to talk to the right people to get write access on the Dkrz pool. Tido, could you point me in the right direction? (Unless of course some one screams no no I want to do this)

[ackerlar]

Maybe '/work/ollie/pool/FESOM/meshes_default' contains fesom2 meshes? But as there is an explicit fesom2 directory, I assumed it contains fesom1 meshes. Or it is the other way around... Anyway, I will double check it. Thanks!

[tsemmler05]

@pgierz: Thanks very much for this offer! I was in touch with Hendryk Bockelmann from DKRZ. In principle as it stands now the whole a270 group should have write permission to the /pool directory - at least it was like this on mistral.

[mbutzin]

I had the same problem today and found that the new pool directory on Levante seems to be
/work/ab0246/pool/
I am not sure if this is the best solution because "ab0246" is not really obvious to users from other projects.