[Release] Release 0.7.1

[zpoint]

Based on releases/0.7.0, cherry-picks all commits from 0.7.1, with minor changes only to smoke_tests.py to ensure more smoke tests pass and Buildkite works.

Smoke tests:

Use buildkite CI to run the following tests:

• pytest tests/test_smoke.py --aws
• pytest tests/test_smoke.py --gcp
• pytest tests/test_smoke.py --azure
• pytest tests/test_smoke.py --kubernetes

All passes except the failures:

pytest tests/test_smoke.py::test_tpu_vm_pod --gcp ---- setup fail, env error, fixed by other PR on master
pytest tests/test_smoke.py::test_tpu_vm --gcp ---- setup fail, env error, fixed by other PR on master
pytest tests/test_smoke.py::TestStorageWithCredentials::test_gcs_regions --azure --- credential issue, Permission denied by location policies.
pytest tests/test_smoke.py::test_gcp_force_enable_external_ips --gcp --- ssh fail on provision, even on master
pytest tests/test_smoke.py::test_managed_jobs_storage --azure --- ? Fail to provision ?
pytest tests/test_smoke.py::test_azure_best_tier_failover --azure --- ResourcesUnavailableError
pytest tests/test_smoke.py::test_file_mounts --azure ---Failed to run command before rsync ?
pytest tests/test_smoke.py::test_skyserve_new_autoscaler_update --azure --- Got FAILED_INITIAL_DELAY instead of FAILED
pytest tests/test_smoke.py::test_azure_disk_tier --azure --- ResourcesUnavailableError
pytest tests/test_smoke.py::test_kubernetes_context_failover --kubernetes --- Resource limit, no h100
pytest tests/test_smoke.py::TestStorageWithCredentials::test_gcs_regions --aws --- Permission denied by location policies

You can view by clicking the failure from buildkite:

Manual tests:

• locally build docs, open docs/build/index.html, scroll over “CLI Reference” (ideally, every page) to see if there are missing sections (we once caught the CLI page completely missing due to an import error; and once it has weird blockquotes displayed)
• Check sky -v
• backward_compatibility_tests.sh run against 0.7.0 on aws, run by buildkite
• Run manual stress tests (see subsection below)
  • following script

    sky jobs launch --gpus A100:8 --cloud aws echo hi -y
    # Check we are properly failing over the zones:
    sky jobs logs --controller
    

  • following script (Fail due to resource unavaliable)

    sky launch -c dbg --cloud aws --num-nodes 16 --gpus T4 --down --use-spot 
    sky down dbg
    

  • sky launch --num-nodes=75 -c dbg --cpus 2+ --use-spot --down --cloud aws -y
  • many jobs

# Launching many jobs on a cluster
sky launch -c test-many-jobs --cloud aws --cpus 16 --region us-east-1
python3 -c "
import subprocess
from multiprocessing.pool import ThreadPool

def run_task(task):
    print(f'Running task {task}')
    subprocess.run(f'sky exec test-many-jobs -d \"echo hi {task}; sleep 60\"', shell=True)

pool = ThreadPool(8)
pool.map(run_task, range(1000))
"
# Test the job queue on cluster is correct
sky queue test-many-jobs

• sky show-gpus manual tests

• Run a 24-hour+ spot job and ensure it doesn’t OOM
  sky spot launch -n test-oom --cloud aws --cpus 2 sleep 1000000000000000

[Michaelvll]

For this kind of hot fixes, we may want to include it in master and cherry pick it?

[zpoint]

It's due to a merge conflict. The master branch's value is FAILED_DRIVER, which does not exist in version 0.7.1 but is correct in the master branch.