Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

path column should reflect true path if given a shortcut #92

Open
apreucil opened this issue Nov 21, 2024 · 0 comments
Open

path column should reflect true path if given a shortcut #92

apreucil opened this issue Nov 21, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@apreucil
Copy link

When the catalog builder is given shortcut links rather than the path to the real pp directory, the path currently reflects that shortcut link. I think it should be changed to reflect the true path to the file, for the purposes of dmget recognizing if the file is a DMF file and bringing it online if necessary, which it won't do if its a link. I also have some applications where I am gathering additional metadata by backing out of the file directory from the path to the ascii directory, and this workflow doesn't work if I am giving it shortcuts since the ascii file doesn't exist.

In case you are wondering, I use the shortcut links to combine output from several runs with potentially different experiment names or file structures together but create my own file structure so the catalog knows they are part of the same run. I know of several other users who do similar practices. For example, I was previously using intel22 compilers and had to switch to intel23 mid-run after the gaea update so the file structure is different and the catalog builder doesn't correctly combine the files when doing to_dataset_dict.

Of course I could code my way around this - or make shortcuts to the ascii files as well, but I think this is a simple change that may catch quite a few potential issues in the future like dmget not working as expected. There is a simple solution which is that when the catalog is built, instead of just inserting the path to the file to the path column, you can create a path object from Pathlib like so and then use the resolve attribute to get the true path:

file = pathlib.Path('path/to/shortcut_link_file.nc')
filepath = file.resolve()

Then you can simply use str(filepath) to get back the string to put into the catalog. This shouldn't mess with paths that aren't shortcuts anyway.

@apreucil apreucil added the enhancement New feature or request label Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants