Make crdb-seed more robust to missing files
#4193
Comments
jordanhendricks
added
Testing & Analysis
|
It looks like this is due to macOS having a So we need to probably move the crdb-seed dir out of |
|
I see that the tmp cleaner only seems to run daily and cleans stuff that hasn't been accessed in 3 days. Is the assumption that the seed was built by a test run several days earlier, the tmp cleaner ran in the meantime, then when |
I think that's the case, yeah. Would definitely be curious about the logs. (I think it's not the symlink, it's the target that got removed if I understood the error correctly) |
|
I don't see anything in there. Worth noting that the seed in question was in fact built 5 days before I ran into this. |
|
Dang. I see much more verbose stuff there: Given how sparse your output is, I'm not sure we can say this wasn't the cause, even though the file isn't listed there. Assuming this is the explanation: what if we instead store the seed as a tarball instead of a directory tree? Then the tmp cleaner will either wipe it all or none. Either way, when we unpack it, it should be exactly what we built. If we have to stick with a tree, it'd be nice if the script could somehow validate that the tree is intact. This does seem a little hard since we don't know what the structure of it should be. We could store a manifest into it and check that when we go to use it...which is how I got to the tarball idea. My worry about moving it to "target" is just that it could still become corrupted there, plus this feels more like a tmpfile than a build artifact. But it would probably address the immediate problem. |
|
Hmm the tarball idea is definitely interesting and adds atomicity. |
|
Part of the issue here also was that if you access a file via a symlink, the atime of the symlink changes (assuming not-noatime) but the atime of the underlying file does change. The Apple file cleaner works based on atimes, not mtimes, and note that:
|
Several changes: 1. In #4193, @david-crespo observed some missing files in the crdb-seed generated directory. My suspicion is that that is due to the `/tmp` cleaner that runs on macOS. @davepacheco suggested using a tarball to get atomicity (either the file exists or it doesn't), and it ended up being pretty simple to do that at the end. 2. Teach nexus-test-utils to ensure that the seed tarball exists, fixing `omicron-dev run-all` and anything else that uses nexus-test-utils (and avoiding a dependency on the environment). 3. Move `crdb-seed` to `dev-tools` (thanks Dave for pointing it out!) 4. Add a way to invalidate the cache with `CRDB_SEED_INVALIDATE=1` in the environment. 5. Add a readme for `crdb-seed`. Fixes #4206. Hopefully addresses #4193.
|
#4208 should hopefully address this. I'm going to close this for now -- let's reopen if we run into this again. |
Trying to run tests on 6bc5e60 after previously successfully running them on (95% sure) the last commit from #4154, I got the following error:
The file it's mad about is a symlink that points to a nonexistent file.
Manually deleting
/var/folders/lk/vsx86g0545g3g2tsmvs50py40000gn/T/crdb-base/c57f4ced1447ac7b5e0722b644563fcf94b4998591ad01a8c2cabf9ec1af380eallowed it to recreate the directory from scratch, fixing the issue.The text was updated successfully, but these errors were encountered: