smoke tests support storage mount only

[zpoint]

Feature for #4423

About #4317

• Add the smoke test:
  • test_aws_storage_mounts_with_stop_only_mount
• Implement a command generator function to improve code reusability.
• Modify the test_storage_mounting.yaml.j2 template to support an additional parameter, only_allow_mount, which specifies if only MOUNT is allowed and disables COPY.

Test:

• pytest tests/test_smoke.py::test_aws_storage_mounts_with_stop --aws
• pytest tests/test_smoke.py::test_aws_storage_mounts_with_stop_only_allow_mount --aws
• pytest tests/test_smoke.py::test_gcp_storage_mounts_with_stop --gcp

All pass

• pytest tests/test_smoke.py::test_azure_storage_mounts_with_stop --azure

All fail on master, success on 0.7.1. These two test cases fail on the master branch but pass on release/0.7.1. We can proceed with 0.7.1 but need to take a closer look at the master branch(The master branch fails with or without this PR; it is not affected.)

E 12-06 14:43:31 subprocess_utils.py:141] blobfuse2 already installed. Proceeding...
E 12-06 14:43:31 subprocess_utils.py:141] Mounting  to /mount_private_mount with blobfuse2...
E 12-06 14:43:31 subprocess_utils.py:141] *** blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. ***
E 12-06 14:43:31 subprocess_utils.py:141] Visit https://aka.ms/blobfuse2warnings#220 to see the list of vulnerabilities associated with your current version [2.2.0]
E 12-06 14:43:31 subprocess_utils.py:141] Error: failed to initialize new pipeline [config error in file_cache [temp directory not empty]]
E 12-06 14:43:31 subprocess_utils.py:141] 
Traceback (most recent call last):
  File "/Users/zepingguo/miniconda3/envs/sky/bin/sky", line 8, in <module>
    sys.exit(cli())
  File "/Users/zepingguo/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/zepingguo/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/zepingguo/Desktop/skypilot/sky/utils/common_utils.py", line 366, in _record
    return f(*args, **kwargs)
  File "/Users/zepingguo/Desktop/skypilot/sky/cli.py", line 838, in invoke
    return super().invoke(ctx)
  File "/Users/zepingguo/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/zepingguo/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/zepingguo/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/zepingguo/Desktop/skypilot/sky/utils/common_utils.py", line 386, in _record
    return f(*args, **kwargs)
  File "/Users/zepingguo/Desktop/skypilot/sky/cli.py", line 2576, in start
    core.start(name,
  File "/Users/zepingguo/Desktop/skypilot/sky/utils/common_utils.py", line 386, in _record
    return f(*args, **kwargs)
  File "/Users/zepingguo/Desktop/skypilot/sky/core.py", line 379, in start
    return _start(cluster_name,
  File "/Users/zepingguo/Desktop/skypilot/sky/core.py", line 314, in _start
    backend.sync_file_mounts(handle=handle,
  File "/Users/zepingguo/Desktop/skypilot/sky/utils/common_utils.py", line 386, in _record
    return f(*args, **kwargs)
  File "/Users/zepingguo/Desktop/skypilot/sky/utils/common_utils.py", line 366, in _record
    return f(*args, **kwargs)
  File "/Users/zepingguo/Desktop/skypilot/sky/backends/backend.py", line 76, in sync_file_mounts
    return self._sync_file_mounts(handle, all_file_mounts, storage_mounts)
  File "/Users/zepingguo/Desktop/skypilot/sky/backends/cloud_vm_ray_backend.py", line 3117, in _sync_file_mounts
    self._execute_storage_mounts(handle, storage_mounts)
  File "/Users/zepingguo/Desktop/skypilot/sky/backends/cloud_vm_ray_backend.py", line 4653, in _execute_storage_mounts
    raise exceptions.CommandError(
sky.exceptions.CommandError: Command to mount failed with return code 1.
Failed to run command before rsync ~/tmp-workdir -> /mount_private_mount. Ensure that the network is stable, then retry. echo '
#!/usr/bin/env bash
set -e

{ [ "$(whoami)" == "root" ] && function sudo() { "$@"; } || true; }

MOUNT_PATH=/mount_private_mount
MOUNT_BINARY=blobfuse2

# Check if path is already mounted
if grep -q $MOUNT_PATH /proc/mounts ; then
    echo "Path already mounted - unmounting..."
    fusermount -uz "$MOUNT_PATH"
    echo "Successfully unmounted $MOUNT_PATH."
fi

# Install MOUNT_BINARY if not already installed
if [ -x "$(command -v blobfuse2)" ]; then
  echo "$MOUNT_BINARY already installed. Proceeding..."
else
  echo "Installing $MOUNT_BINARY..."
  sudo apt-get update; sudo apt-get install -y -o Dpkg::Options::="--force-confdef" fuse3 libfuse3-dev && wget -nc https://github.com/Azure/azure-storage-fuse/releases/download/blobfuse2-2.2.0/blobfuse2-2.2.0-Debian-11.0.x86_64.deb -O /tmp/blobfuse2.deb && sudo dpkg --install /tmp/blobfuse2.deb && mkdir -p ~/.sky/blobfuse2_cache;
fi

# Check if mount path exists
if [ ! -d "$MOUNT_PATH" ]; then
  echo "Mount path $MOUNT_PATH does not exist. Creating..."
  sudo mkdir -p $MOUNT_PATH
  sudo chmod 777 $MOUNT_PATH
else
  # Check if mount path contains files
  if [ "$(ls -A $MOUNT_PATH)" ]; then
    echo "Mount path $MOUNT_PATH is not empty. Please mount to another path or remove it first."
    exit 42
  fi
fi
echo "Mounting $SOURCE_BUCKET to $MOUNT_PATH with $MOUNT_BINARY..."
AZURE_STORAGE_ACCOUNT=xxx AZURE_STORAGE_ACCESS_KEY=xxx blobfuse2 /mount_private_mount --allow-other --no-symlinks -o umask=022 -o default_permissions --tmp-path ~/.sky/blobfuse2_cache/sky635694a6141885a3_sky-test-1733466958 --container-name sky-test-1733466958
echo "Mounting done."
' > ~/.sky/mount_97651.sh && chmod +x ~/.sky/mount_97651.sh && bash ~/.sky/mount_97651.sh && rm ~/.sky/mount_97651.sh See logs in ~/sky_logs/sky-2024-12-06-14-40-59-370163/storage_mounts.log
D 12-06 14:43:32 cloud_vm_ray_backend.py:3914] Provisioner version: ProvisionerVersion.SKYPILOT using new provisioner for teardown.
D 12-06 14:43:38 metadata_utils.py:115] Remove metadata of cluster t-azure-storage-mount-b8-ee-94a6.
D 12-06 14:43:38 metadata_utils.py:115] Remove metadata of cluster t-azure-storage-mount-b8-ee.


Mounting  to /mount_private_mount with blobfuse2...
*** blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. ***
Visit https://aka.ms/blobfuse2warnings#220 to see the list of vulnerabilities associated with your current version [2.2.0]
Error: failed to initialize new pipeline [config error in file_cache [temp directory not empty]]
^C

Test the modified command work as expected

(sky) ➜  skypilot git:(dev/zeping/smoke_test_support_storage_mount_only) ✗ (output="[]"; [ "$output" = "[]" ] && exit 1 || exit 0)
(sky) ➜  skypilot git:(dev/zeping/smoke_test_support_storage_mount_only) ✗ echo $?                                                
1
(sky) ➜  skypilot git:(dev/zeping/smoke_test_support_storage_mount_only) ✗ (output="[ss]"; [ "$output" = "[]" ] && exit 1 || exit 0)
(sky) ➜  skypilot git:(dev/zeping/smoke_test_support_storage_mount_only) ✗ echo $?                                                  
0

About #4411

It seems the smoke test can already test it, no need to add

pytest tests/test_smoke.py::test_use_spot --azure

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[Michaelvll]

Thanks @zpoint! Mostly looks good to me.

[Michaelvll]

LGTM. Thanks @zpoint!