Squashfs volumes are not mountable thru enroot on HyperPod

[cfregly]

In an attempt to improve performance, the user has copied the SquashFS image (which contains their dataset) onto FSx - and trying to mount the SquashFS image into the docker container with --container-mounts.

However, enroot/pyxis could not access the mounted SquashFS files as enroot/pyxis does not support SquashFS mounts, it seems.

[cfregly]

This is a known issue with enroot/pyxis described here: NVIDIA/enroot#180 (comment)

Some options are the following:
1/ Extract the dataset onto FSx, but this creates many small files which may not be optimal.

2/ Extract the dataset onto each worker node's local NVMe disk for better performance

3/ Use squashfuse and autofs per NVIDIA/enroot#180 (comment)