You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the catalog builder is given shortcut links rather than the path to the real pp directory, the path currently reflects that shortcut link. I think it should be changed to reflect the true path to the file, for the purposes of dmget recognizing if the file is a DMF file and bringing it online if necessary, which it won't do if its a link. I also have some applications where I am gathering additional metadata by backing out of the file directory from the path to the ascii directory, and this workflow doesn't work if I am giving it shortcuts since the ascii file doesn't exist.
In case you are wondering, I use the shortcut links to combine output from several runs with potentially different experiment names or file structures together but create my own file structure so the catalog knows they are part of the same run. I know of several other users who do similar practices. For example, I was previously using intel22 compilers and had to switch to intel23 mid-run after the gaea update so the file structure is different and the catalog builder doesn't correctly combine the files when doing to_dataset_dict.
Of course I could code my way around this - or make shortcuts to the ascii files as well, but I think this is a simple change that may catch quite a few potential issues in the future like dmget not working as expected. There is a simple solution which is that when the catalog is built, instead of just inserting the path to the file to the path column, you can create a path object from Pathlib like so and then use the resolve attribute to get the true path:
When the catalog builder is given shortcut links rather than the path to the real pp directory, the path currently reflects that shortcut link. I think it should be changed to reflect the true path to the file, for the purposes of dmget recognizing if the file is a DMF file and bringing it online if necessary, which it won't do if its a link. I also have some applications where I am gathering additional metadata by backing out of the file directory from the path to the ascii directory, and this workflow doesn't work if I am giving it shortcuts since the ascii file doesn't exist.
In case you are wondering, I use the shortcut links to combine output from several runs with potentially different experiment names or file structures together but create my own file structure so the catalog knows they are part of the same run. I know of several other users who do similar practices. For example, I was previously using intel22 compilers and had to switch to intel23 mid-run after the gaea update so the file structure is different and the catalog builder doesn't correctly combine the files when doing to_dataset_dict.
Of course I could code my way around this - or make shortcuts to the ascii files as well, but I think this is a simple change that may catch quite a few potential issues in the future like dmget not working as expected. There is a simple solution which is that when the catalog is built, instead of just inserting the path to the file to the path column, you can create a path object from Pathlib like so and then use the resolve attribute to get the true path:
Then you can simply use
str(filepath)
to get back the string to put into the catalog. This shouldn't mess with paths that aren't shortcuts anyway.The text was updated successfully, but these errors were encountered: