[Dependency] Drop Python 3.6 support, switch to Python>=3.7 and fix a…

…zure launching issue (#1952)

* use write files instead

* fix azure timeout issue

* upgrade python version

format

* Update docs to 3.7

.github/workflows/format.yml

@@ -13,9 +13,7 @@ on:
       - 'releases/**'
 jobs:
   format:
-    # Need to specify 20.04, because ubuntu-latest does not work with
-    # python 3.6: https://github.com/actions/setup-python/issues/355#issuecomment-1335042510
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version: ["3.8"]

.github/workflows/mypy-generic.yml

@@ -15,8 +15,6 @@ on:
       - 'releases/**'
 jobs:
   mypy:
-    # Need to specify 20.04, because ubuntu-latest does not work with
-    # python 3.6: https://github.com/actions/setup-python/issues/355#issuecomment-1335042510
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       - run: 'echo "No mypy to run"'

.github/workflows/mypy.yml

@@ -13,9 +13,7 @@ on:
       - 'releases/**'
 jobs:
   mypy:
-    # Need to specify 20.04, because ubuntu-latest does not work with
-    # python 3.6: https://github.com/actions/setup-python/issues/355#issuecomment-1335042510
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version: ["3.8"]

.github/workflows/pylint.yml

@@ -14,12 +14,10 @@ on:
 
 jobs:
   pylint:
-    # Need to specify 20.04, because ubuntu-latest does not work with
-    # python 3.6: https://github.com/actions/setup-python/issues/355#issuecomment-1335042510
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.6"]
+        python-version: ["3.8"]
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}

.github/workflows/pytest-generic.yml

@@ -14,8 +14,6 @@ on:
       - 'releases/**'
 jobs:
   python-test:
-    # Need to specify 20.04, because ubuntu-latest does not work with
-    # python 3.6: https://github.com/actions/setup-python/issues/355#issuecomment-1335042510
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       - run: 'echo "No tests to run"'

.github/workflows/pytest.yml

@@ -14,7 +14,7 @@ jobs:
   python-test:
     strategy:
       matrix:
-        python-version: [3.6]
+        python-version: [3.8]
         test-path:
           - tests/test_cli.py
           - tests/test_config.py
@@ -26,9 +26,7 @@ jobs:
           - tests/test_storage.py
           - tests/test_wheels.py
           - tests/test_spot.py
-    # Need to specify 20.04, because ubuntu-latest does not work with
-    # python 3.6: https://github.com/actions/setup-python/issues/355#issuecomment-1335042510
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
         uses: actions/checkout@v2

CONTRIBUTING.md

@@ -16,7 +16,7 @@ We use GitHub to track issues and features. For new contributors, we recommend l
 
 ### Installing SkyPilot for development
 ```bash
-# SkyPilot requires python >= 3.6.
+# SkyPilot requires python >= 3.7.
 # You can just install the dependencies for
 # certain clouds, e.g., ".[aws,azure,gcp,lambda]"
 pip install -e ".[all]"

docs/source/getting-started/installation.rst

@@ -5,7 +5,7 @@ Install SkyPilot using pip:
 
 .. code-block:: console
 
-  $ # SkyPilot requires python >= 3.6. For Apple Silicon, use >= 3.8.
+  $ # SkyPilot requires python >= 3.7. For Apple Silicon, use >= 3.8.
   $ # Recommended: use a new conda env to avoid package conflicts.
   $ conda create -y -n sky python=3.8
   $ conda activate sky

docs/source/reference/local/setup.rst

@@ -22,7 +22,7 @@ To install Ray and SkyPilot for all users, run the following commands on all loc
 
    $ pip3 install ray[default]==2.0.1
 
-   $ # SkyPilot requires python >= 3.6.
+   $ # SkyPilot requires python >= 3.7.
    $ pip3 install skypilot
 
 

sky/design_docs/onprem-design.md

@@ -10,7 +10,7 @@
 
 ## Installing Ray and SkyPilot 
 - Admin installs Ray==2.0.1 and SkyPilot globally on all machines. It is assumed that the admin regularly keeps SkyPilot updated on the cluster.
-- Python >= 3.6 for all users.
+- Python >= 3.7 for all users.
 - When a regular user runs `sky launch`, a local version of SkyPilot will be installed on the machine for each user. The local installation of Ray is specified in `sky/templates/local-ray.yml.j2`.
 
 ## Registering clusters as a regular user

sky/setup_files/setup.py

@@ -90,7 +90,7 @@ def parse_readme(readme: str) -> str:
     'rich',
     'tabulate',
     'typing-extensions',
-    'filelock',  # TODO(mraheja): Enforce >=3.6.0 when python version is >= 3.7
+    'filelock>=3.6.0',
     # This is used by ray. The latest 1.44.0 will generate an error
     # `Fork support is only compatible with the epoll1 and poll
     # polling strategies`
@@ -118,8 +118,10 @@ def parse_readme(readme: str) -> str:
     # TODO(zongheng): azure-cli is huge and takes a long time to install.
     # Tracked in: https://github.com/Azure/azure-cli/issues/7387
     # azure-identity is needed in node_provider.
+    # We need azure-identity>=1.13.0 to enable the customization of the
+    # timeout of AzureCliCredential.
     'azure': [
-        'azure-cli>=2.31.0', 'azure-core', 'azure-identity',
+        'azure-cli>=2.31.0', 'azure-core', 'azure-identity>=1.13.0',
         'azure-mgmt-network'
     ],
     'gcp': ['google-api-python-client', 'google-cloud-storage'],
@@ -157,15 +159,14 @@ def parse_readme(readme: str) -> str:
     long_description=long_description,
     long_description_content_type='text/markdown',
     setup_requires=['wheel'],
-    requires_python='>=3.6',
+    requires_python='>=3.7',
     install_requires=install_requires,
     extras_require=extras_require,
     entry_points={
         'console_scripts': ['sky = sky.cli:cli'],
     },
     include_package_data=True,
     classifiers=[
-        'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: 3.9',

sky/skylet/providers/azure/config.py

@@ -48,7 +48,12 @@ def _configure_resource_group(config):
     subscription_id = config["provider"].get("subscription_id")
     if subscription_id is None:
         subscription_id = get_cli_profile().get_subscription_id()
-    resource_client = ResourceManagementClient(AzureCliCredential(), subscription_id)
+    # Increase the timeout to fix the Azure get-access-token (used by ray azure
+    # node_provider) timeout issue.
+    # Tracked in https://github.com/Azure/azure-cli/issues/20404#issuecomment-1249575110
+    resource_client = ResourceManagementClient(
+        AzureCliCredential(process_timeout=30), subscription_id
+    )
     config["provider"]["subscription_id"] = subscription_id
     logger.info("Using subscription id: %s", subscription_id)
 

sky/skylet/providers/azure/node_provider.py

@@ -68,7 +68,10 @@ def __init__(self, provider_config, cluster_name):
         subscription_id = provider_config["subscription_id"]
         self.cache_stopped_nodes = provider_config.get("cache_stopped_nodes", True)
         # Sky only supports Azure CLI credential for now.
-        credential = AzureCliCredential()
+        # Increase the timeout to fix the Azure get-access-token (used by ray azure
+        # node_provider) timeout issue.
+        # Tracked in https://github.com/Azure/azure-cli/issues/20404#issuecomment-1249575110
+        credential = AzureCliCredential(process_timeout=30)
         self.compute_client = ComputeManagementClient(credential, subscription_id)
         self.network_client = NetworkManagementClient(credential, subscription_id)
         self.resource_client = ResourceManagementClient(credential, subscription_id)

sky/skylet/ray_patches/__init__.py

@@ -84,16 +84,3 @@ def patch() -> None:
 
     from ray.autoscaler._private import updater
     _run_patch(updater.__file__, _to_absolute('updater.py.patch'))
-
-    # Fix the Azure get-access-token (used by ray azure node_provider) timeout issue,
-    # by increasing the timeout.
-    # Tracked in https://github.com/Azure/azure-cli/issues/20404#issuecomment-1249575110
-    # Only patch it if azure cli is installed.
-    try:
-        import azure
-        from azure.identity._credentials import azure_cli
-        version = pkg_resources.get_distribution('azure-cli').version
-        _run_patch(azure_cli.__file__, _to_absolute('azure_cli.py.patch'),
-                   version)
-    except ImportError:
-        pass