ci: run the smoke tests on a schedule (canonical#1387)

We have some basic smoke tests, which are currently the only tests we
regularly run against an actual Juju model. The tests deploy a basic
charm and verify that it goes to `active` status (which happens in an
`install` handler). However, these are currently only run if someone
manually runs `tox -e smoke`. This PR adds running the tests on a
schedule (currently monthly, in the days before we will likely do a
release).

This tests:
* Juju 3.5 (latest stable 3.x - I suggest we *change* this to 3.6 when
that's released, rather than *adding* 3.6). The smoke tests do not
currently work with Juju 2.9 (I'm not sure if it's worth adding that at
this point), but if that changes, it should only require a matrix change
in the workload. The smoke tests use python-libjuju, which doesn't yet
support Juju 4.0, but when that changes, this should also only require a
matrix change.
 * Charmcraft 2.x and 3.x.
 * LXD and MicroK8s.

The smoke test is changed to use a plain `subprocess.run` to pack the
charm, rather than use pytest-operator for this. The `pytest-operator`
approach doesn't work out-of-the-box in the workflow, and it seems
cleaner to just do the pack ourselves.

There's an [example run on my
fork](https://github.com/tonyandrewmeyer/operator/actions/runs/11006199263/job/30560942709).

This could be optimised in the future to store the packed charms and
share them across Juju versions and cloud types, but it doesn't seem
worth doing that for now.

Fixes canonical#1322

.github/workflows/smoke.yaml

@@ -0,0 +1,68 @@
+name: ops Smoke Tests
+
+on:
+  workflow_dispatch:
+    schedule:
+      - cron:  '0 7 25 * *'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        # pylibjuju does not currently support Juju 4.x
+        # The smoke tests do not yet work on Juju 2.9.
+        juju-version: ['3.5']
+        charmcraft-version: ['2.x', '3.x']
+        cloud: ['lxd', 'microk8s']
+
+    env:
+      JUJU_VERSION: "${{ matrix.juju-version }}"
+
+    steps:
+      # LXD is required for charmcraft to pack, even if it's not used as the
+      # Juju cloud.
+      - name: Set up LXD
+        uses: canonical/setup-lxd@8fb85546a934dfb994becf81341dd387ffe6aabb
+        with:
+          channel: 5.0/stable
+
+      - name: Set up Microk8s
+        if: matrix.cloud == 'microk8s'
+        uses: balchua/[email protected]
+        with:
+          channel: '1.26-strict/stable'
+          devMode: 'true'
+          addons: '["dns", "hostpath-storage"]'
+
+      - name: Set up Juju (classic)
+        if: matrix.juju-version == '2.9'
+        run: sudo snap install juju --classic --channel=${{ matrix.juju-version }}
+
+      - name: Set up Juju
+        if: matrix.juju-version != '2.9'
+        run: sudo snap install juju --channel=${{ matrix.juju-version }}
+
+      - name: Bootstrap Juju controller (k8s)
+        if: matrix.cloud == 'microk8s'
+        run: sg snap_microk8s -c 'juju bootstrap microk8s'
+
+      - name: Bootstrap Juju controller (lxd)
+        if: matrix.cloud == 'lxd'
+        run: juju bootstrap localhost
+
+      - name: Install charmcraft
+        run: sudo snap install charmcraft --channel=${{ matrix.charmcraft-version }} --classic
+
+      - name: Checkout the repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3
+        uses: actions/setup-python@v5
+
+      - name: Install tox
+        run: pip install tox~=4.2
+
+      - name: Run smoke tests
+        run: tox -e smoke

test/smoke/test_smoke.py

@@ -14,7 +14,11 @@
 #
 # Learn more about testing at: https://juju.is/docs/sdk/testing
 
+import grp
 import logging
+import os
+import pathlib
+import subprocess
 
 import pytest
 from pytest_operator.plugin import OpsTest
@@ -45,6 +49,33 @@
 """
 
 
+def pack(charm_dir: pathlib.Path):
+    """Pack the charm.
+
+    The pytest-operator plugin has a pack method, but it doesn't work out of the
+    box in GitHub actions, and there isn't really any reason that it should be
+    part of the plugin, so we just have a simple subprocess here.
+    """
+    cmd = ['charmcraft', 'pack', '--verbose']
+    # We need to use `sudo` in the GitHub actions environment, just as in
+    # the pack test. `sg lxd -c` should work, but does not - perhaps because of
+    # the way we are installing LXD?
+    if 'lxd' not in {grp.getgrgid(g).gr_name for g in os.getgroups()}:
+        cmd.insert(0, 'sudo')
+
+    logger.info('Building charm with %r', cmd)
+    subprocess.run(cmd, cwd=charm_dir, check=True)
+    logger.info('Built charm')
+
+    # Move the packed charm to the charm directory.
+    dest_name = None
+    for charm in charm_dir.glob('*.charm'):
+        dest_name = charm_dir / charm.name
+        charm.rename(dest_name)
+    # With the way we use charmcraft, we know that there will only be one.
+    return dest_name.absolute()
+
+
 @pytest.mark.parametrize(
     'base,charmcraft_version,name',
     (
@@ -55,13 +86,23 @@
 )
 async def test_smoke(ops_test: OpsTest, base: str, charmcraft_version: int, name: str):
     """Verify that we can build and deploy charms from supported bases."""
+    available_charmcraft_version = (
+        subprocess.run(['charmcraft', 'version'], check=True, capture_output=True)  # noqa: S607
+        .stdout.decode()
+        .strip()
+        .rsplit()[-1]
+        .split('.')
+    )
+    if int(available_charmcraft_version[0]) < charmcraft_version:
+        pytest.skip(f'charmcraft version {available_charmcraft_version} is too old for this test')
+        return
     charmcraft_yaml = {
         2: CHARMCRAFT2_YAML,
         3: CHARMCRAFT3_YAML,
     }[charmcraft_version].format(base=base)
     with open('./test/charms/test_smoke/charmcraft.yaml', 'w') as outf:
         outf.write(charmcraft_yaml)
-    charm = await ops_test.build_charm('./test/charms/test_smoke/')
+    charm = pack(pathlib.Path('./test/charms/test_smoke/'))
 
     app = await ops_test.model.deploy(
         charm, base=f'ubuntu@{base}', application_name=f'{name}-smoke'

tox.ini

@@ -140,6 +140,7 @@ description = Run a smoke test against a Juju controller.
 allowlist_externals = juju
                       charmcraft
                       bash
+passenv = JUJU_VERSION
 deps =
     build
     coverage[toml]~=7.0
@@ -151,6 +152,8 @@ commands =
     python -m build --sdist --outdir={toxinidir}/test/charms/test_smoke/
     # Inject the tarball into the smoke test charm's requirements.
     bash -c 'echo "./$(ls -1 ./test/charms/test_smoke/ | grep tar.gz)" > ./test/charms/test_smoke/requirements.txt'
+    # If a specific Juju version is set, then make sure we are using that version of pylibjuju.
+    bash -c 'if [ -n "$JUJU_VERSION" ]; then pip install "juju ~= $JUJU_VERSION"; fi'
 
     # Run our smoke tests (this will build the charm, then run the tests).
     pytest -v --tb native --log-cli-level=INFO -s {posargs} {toxinidir}/test/smoke/