Skip to content

Commit

Permalink
breaking: Switch to ST124 "shorthand notation" syntax in charmcraft.y…
Browse files Browse the repository at this point in the history
…aml (#258)

Enables building & releasing multi-base charms with 24.04 in a single
charmcraft.yaml and git branch

Integration testing is not supported on multiple bases—it is currently
only supported on 22.04

## Breaking changes
- ST124 "shorthand notation" syntax required in charmcraft.yaml
- `pytest-operator-cache` (which overrides `ops_test.build_charm` from
pytest-operator) now assumes 22.04 base

## Deprecation notice
- `pytest-operator-cache` is deprecated & will be removed in a future
release
  • Loading branch information
carlcsaposs-canonical authored Jan 10, 2025
1 parent eb08b36 commit 92d0a9f
Show file tree
Hide file tree
Showing 15 changed files with 292 additions and 193 deletions.
16 changes: 16 additions & 0 deletions .github/workflows/build_charm.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,19 @@ with:
cache: true
```
remember to add your charm's branch(es) to charmcraftcache by running `ccc add` or by [opening an issue](https://github.com/canonical/charmcraftcache-hub/issues/new?assignees=&labels=add-charm&projects=&template=add_charm_branch.yaml&title=Add+charm+branch).

### Required charmcraft.yaml syntax
Only [ST124 - Multi-base platforms in craft tools](https://docs.google.com/document/d/1QVHxZumruKVZ3yJ2C74qWhvs-ye5I9S6avMBDHs2YcQ/edit) "shorthand notation" syntax is supported

#### Example
```yaml
platforms:
[email protected]:amd64:
[email protected]:arm64:
[email protected]:amd64:
[email protected]:arm64:
```

Under the charmcraft.yaml `platforms` key, `build-on` and `build-for` syntax are not supported

The `base` and `bases` charmcraft.yaml keys are not supported
33 changes: 15 additions & 18 deletions .github/workflows/build_charm.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,11 @@ on:
outputs:
artifact-prefix:
description: Charm packages are uploaded to GitHub artifacts beginning with this prefix
value: ${{ jobs.collect-bases.outputs.artifact-prefix-with-inputs }}
value: ${{ jobs.collect-platforms.outputs.artifact-prefix-with-inputs }}

jobs:
collect-bases:
name: Collect bases for charm | ${{ inputs.path-to-charm-directory }}
collect-platforms:
name: Collect platforms for charm | ${{ inputs.path-to-charm-directory }}
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
Expand All @@ -70,25 +70,25 @@ jobs:
run: pipx install git+https://github.com/canonical/data-platform-workflows@'${{ steps.workflow-version.outputs.sha }}'#subdirectory=python/cli
- name: Checkout
uses: actions/checkout@v4
- name: Collect charm bases to build from charmcraft.yaml
- name: Collect charm platforms to build from charmcraft.yaml
id: collect
run: collect-charm-bases --directory='${{ inputs.path-to-charm-directory }}' --cache='${{ inputs.cache }}'
run: collect-charm-platforms --directory='${{ inputs.path-to-charm-directory }}' --cache='${{ inputs.cache }}'
outputs:
bases: ${{ steps.collect.outputs.bases }}
platforms: ${{ steps.collect.outputs.platforms }}
artifact-prefix-with-inputs: ${{ inputs.artifact-prefix || steps.collect.outputs.default_prefix }}

build:
strategy:
matrix:
base: ${{ fromJSON(needs.collect-bases.outputs.bases) }}
name: 'Build charm | base #${{ matrix.base.id }}'
platform: ${{ fromJSON(needs.collect-platforms.outputs.platforms) }}
name: 'Build charm | ${{ matrix.platform.name }}'
needs:
- collect-bases
runs-on: ${{ matrix.base.runner }}
- collect-platforms
runs-on: ${{ matrix.platform.runner }}
timeout-minutes: 120
steps:
- name: (GitHub-hosted ARM runner) Install libpq-dev
if: ${{ matrix.base.runner == 'Ubuntu_ARM64_4C_16G_02' }}
if: ${{ matrix.platform.runner == 'Ubuntu_ARM64_4C_16G_02' }}
# Needed for `charmcraftcache` to resolve dependencies (for postgresql charms with psycopg2)
run: |
sudo apt-get update
Expand Down Expand Up @@ -139,12 +139,9 @@ jobs:
run: |
if '${{ inputs.cache }}'
then
sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- charmcraftcache pack -v --bases-index='${{ matrix.base.id }}'
sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- charmcraftcache pack -v --platform='${{ matrix.platform.name }}'
else
# Workaround for https://github.com/canonical/charmcraft/issues/1389 on charmcraft 2
touch requirements.txt
sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- charmcraft pack -v --bases-index='${{ matrix.base.id }}'
sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- charmcraft pack -v --platform='${{ matrix.platform.name }}'
fi
env:
# Used by charmcraftcache (to avoid GitHub API rate limit)
Expand All @@ -153,14 +150,14 @@ jobs:
if: ${{ failure() && steps.pack.outcome == 'failure' }}
uses: actions/upload-artifact@v4
with:
name: logs-charmcraft-build-${{ needs.collect-bases.outputs.artifact-prefix-with-inputs }}-base-${{ matrix.base.id }}
name: logs-charmcraft-build-${{ needs.collect-platforms.outputs.artifact-prefix-with-inputs }}-platform-${{ matrix.platform.name_in_artifact }}
path: ~/.local/state/charmcraft/log/
if-no-files-found: error
- run: touch .empty
- name: Upload charm package
uses: actions/upload-artifact@v4
with:
name: ${{ needs.collect-bases.outputs.artifact-prefix-with-inputs }}-base-${{ matrix.base.id }}
name: ${{ needs.collect-platforms.outputs.artifact-prefix-with-inputs }}-platform-${{ matrix.platform.name_in_artifact }}
# .empty file required to preserve directory structure
# See https://github.com/actions/upload-artifact/issues/344#issuecomment-1379232156
path: |
Expand Down
14 changes: 13 additions & 1 deletion .github/workflows/build_rock.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,16 @@ jobs:
build:
name: Build rock
uses: canonical/data-platform-workflows/.github/workflows/[email protected]
```
```
### Supported `platforms` syntax in rockcraft.yaml
Only "shorthand notation" is supported

Example rockcraft.yaml
```yaml
platforms:
amd64:
arm64:
```

`build-on` and `build-for` are not supported
22 changes: 11 additions & 11 deletions .github/workflows/build_rock.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ on:
outputs:
artifact-prefix:
description: Rock packages are uploaded to GitHub artifacts beginning with this prefix
value: ${{ jobs.collect-bases.outputs.artifact-prefix-with-inputs }}
value: ${{ jobs.collect-platforms.outputs.artifact-prefix-with-inputs }}

jobs:
collect-bases:
collect-platforms:
name: Collect platforms for rock | ${{ inputs.path-to-rock-directory }}
runs-on: ubuntu-latest
timeout-minutes: 5
Expand All @@ -49,19 +49,19 @@ jobs:
uses: actions/checkout@v4
- name: Collect rock platforms to build from rockcraft.yaml
id: collect
run: collect-rock-bases --directory='${{ inputs.path-to-rock-directory }}'
run: collect-rock-platforms --directory='${{ inputs.path-to-rock-directory }}'
outputs:
bases: ${{ steps.collect.outputs.bases }}
platforms: ${{ steps.collect.outputs.platforms }}
artifact-prefix-with-inputs: ${{ inputs.artifact-prefix || steps.collect.outputs.default_prefix }}

build:
strategy:
matrix:
base: ${{ fromJSON(needs.collect-bases.outputs.bases) }}
name: 'Build rock | ${{ matrix.base.id }}'
platform: ${{ fromJSON(needs.collect-platforms.outputs.platforms) }}
name: 'Build rock | ${{ matrix.platform.name }}'
needs:
- collect-bases
runs-on: ${{ matrix.base.runner }}
- collect-platforms
runs-on: ${{ matrix.platform.runner }}
timeout-minutes: 15
steps:
- name: Get workflow version
Expand Down Expand Up @@ -93,19 +93,19 @@ jobs:
- name: Pack rock
id: pack
working-directory: ${{ inputs.path-to-rock-directory }}
run: sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- rockcraft pack -v --platform='${{ matrix.base.id }}'
run: sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- rockcraft pack -v --platform='${{ matrix.platform.name }}'
- name: Upload rockcraft logs
if: ${{ failure() && steps.pack.outcome == 'failure' }}
uses: actions/upload-artifact@v4
with:
name: logs-rockcraft-build-${{ inputs.artifact-prefix }}-architecture-${{ matrix.base.id }}
name: logs-rockcraft-build-${{ inputs.artifact-prefix }}-platform-${{ matrix.platform.name }}
path: ~/.local/state/rockcraft/log/
if-no-files-found: error
- run: touch .empty
- name: Upload rock package
uses: actions/upload-artifact@v4
with:
name: ${{ needs.collect-bases.outputs.artifact-prefix-with-inputs }}-architecture-${{ matrix.base.id }}
name: ${{ needs.collect-platforms.outputs.artifact-prefix-with-inputs }}-platform-${{ matrix.platform.name }}
# .empty file required to preserve directory structure
# See https://github.com/actions/upload-artifact/issues/344#issuecomment-1379232156
path: |
Expand Down
33 changes: 32 additions & 1 deletion .github/workflows/build_snap.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,35 @@ jobs:
build:
name: Build snap
uses: canonical/data-platform-workflows/.github/workflows/[email protected]
```
```
### Supported `platforms` and `architectures` syntax in snapcraft.yaml
See https://snapcraft.io/docs/architectures#how-to-create-a-snap-for-a-specific-architecture

#### core24
Only `platforms` is supported. `architectures` is not supported

Only "shorthand notation" is supported

Example snapcraft.yaml
```yaml
platforms:
amd64:
arm64:
```

`build-on` and `build-for` are not supported

#### core22
Only `architectures` is supported. `platforms` is not supported

`architectures` must be a list of dictionaries. Each dictionary in the list must contain a `build-on` key

Example snapcraft.yaml
```yaml
architectures:
- build-on: [amd64]
build-for: [amd64]
- build-on: [arm64]
build-for: [arm64]
```
26 changes: 13 additions & 13 deletions .github/workflows/build_snap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ on:
outputs:
artifact-prefix:
description: Snap packages are uploaded to GitHub artifacts beginning with this prefix
value: ${{ jobs.collect-bases.outputs.artifact-prefix-with-inputs }}
value: ${{ jobs.collect-platforms.outputs.artifact-prefix-with-inputs }}

jobs:
collect-bases:
name: Collect architectures for snap | ${{ inputs.path-to-snap-project-directory }}
collect-platforms:
name: Collect platforms for snap | ${{ inputs.path-to-snap-project-directory }}
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
Expand All @@ -50,21 +50,21 @@ jobs:
run: pipx install git+https://github.com/canonical/data-platform-workflows@'${{ steps.workflow-version.outputs.sha }}'#subdirectory=python/cli
- name: Checkout
uses: actions/checkout@v4
- name: Collect snap architectures to build from snapcraft.yaml
- name: Collect snap platforms to build from snapcraft.yaml
id: collect
run: collect-snap-bases --directory='${{ inputs.path-to-snap-project-directory }}'
run: collect-snap-platforms --directory='${{ inputs.path-to-snap-project-directory }}'
outputs:
bases: ${{ steps.collect.outputs.bases }}
platforms: ${{ steps.collect.outputs.platforms }}
artifact-prefix-with-inputs: ${{ inputs.artifact-prefix || steps.collect.outputs.default_prefix }}

build:
strategy:
matrix:
base: ${{ fromJSON(needs.collect-bases.outputs.bases) }}
name: 'Build snap | ${{ matrix.base.id }}'
platform: ${{ fromJSON(needs.collect-platforms.outputs.platforms) }}
name: 'Build snap | ${{ matrix.platform.name }}'
needs:
- collect-bases
runs-on: ${{ matrix.base.runner }}
- collect-platforms
runs-on: ${{ matrix.platform.runner }}
timeout-minutes: 30
steps:
- name: Get workflow version
Expand Down Expand Up @@ -96,19 +96,19 @@ jobs:
- name: Pack snap
id: pack
working-directory: ${{ inputs.path-to-snap-project-directory }}
run: sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- snapcraft pack -v --build-for='${{ matrix.base.id }}'
run: sudo --user "$USER" --preserve-env --preserve-env=PATH -- env -- snapcraft pack -v --build-for='${{ matrix.platform.name }}'
- name: Upload snapcraft logs
if: ${{ failure() && steps.pack.outcome == 'failure' }}
uses: actions/upload-artifact@v4
with:
name: logs-snapcraft-build-${{ inputs.artifact-prefix }}-architecture-${{ matrix.base.id }}
name: logs-snapcraft-build-${{ inputs.artifact-prefix }}-platform-${{ matrix.platform.name }}
path: ~/.local/state/snapcraft/log/
if-no-files-found: error
- run: touch .empty
- name: Upload snap package
uses: actions/upload-artifact@v4
with:
name: ${{ needs.collect-bases.outputs.artifact-prefix-with-inputs }}-architecture-${{ matrix.base.id }}
name: ${{ needs.collect-platforms.outputs.artifact-prefix-with-inputs }}-platform-${{ matrix.platform.name }}
# .empty file required to preserve directory structure
# See https://github.com/actions/upload-artifact/issues/344#issuecomment-1379232156
path: |
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/integration_test_charm.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
Workflow file: [integration_test_charm.yaml](integration_test_charm.yaml)

> [!WARNING]
> The `pytest-operator-cache` plugin is **deprecated**. Follow the migration instructions here: [pytest_operator_cache/deprecation_notice.md](../../python/pytest_plugins/pytest_operator_cache/deprecation_notice.md)
## Usage
### Step 1: Create your workflow
```yaml
Expand Down Expand Up @@ -27,7 +30,6 @@ jobs:
#### Step A
Add
```toml
pytest-operator-cache = {git = "https://github.com/canonical/data-platform-workflows", tag = "v0.0.0", subdirectory = "python/pytest_plugins/pytest_operator_cache"}
pytest-operator-groups = {git = "https://github.com/canonical/data-platform-workflows", tag = "v0.0.0", subdirectory = "python/pytest_plugins/pytest_operator_groups"}
```
to your integration test dependencies in `pyproject.toml`.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Copied from https://github.com/canonical/charmcraftcache/blob/main/charmcraftcache/_platforms.py
import pathlib

import yaml

_SYNTAX_DOCS = "https://github.com/canonical/data-platform-workflows/blob/main/.github/workflows/build_charm.md#required-charmcraftyaml-syntax"


class Platform(str):
"""Platform in charmcraft.yaml 'platforms' (e.g. '[email protected]:amd64')"""

def __new__(cls, value: str, /):
try:
_, architecture = value.split(":")
except ValueError:
raise ValueError(
"Invalid ST124 shorthand notation in charmcraft.yaml 'platforms': "
f"{repr(value)}\n\nMore info: {_SYNTAX_DOCS}"
)
instance: Platform = super().__new__(cls, value)
instance.architecture = architecture
return instance


def get(charmcraft_yaml: pathlib.Path, /):
"""Get platforms from charmcraft.yaml"""
charmcraft_yaml_data = yaml.safe_load(charmcraft_yaml.read_text())
for key in ("base", "bases"):
if key in charmcraft_yaml_data:
raise ValueError(
f"'{key}' key in charmcraft.yaml not supported. Use 'platforms' key instead.\n\n"
f"More info: {_SYNTAX_DOCS}"
)
yaml_platforms = charmcraft_yaml_data.get("platforms")
if not yaml_platforms:
raise ValueError(
f"'platforms' key in charmcraft.yaml required\n\nMore info: {_SYNTAX_DOCS}"
)
if not isinstance(yaml_platforms, dict):
raise TypeError(
"Expected charmcraft.yaml 'platforms' with type 'dict', got "
f"{repr(type(yaml_platforms).__name__)}: {repr(yaml_platforms)}\n\n"
f"More info: {_SYNTAX_DOCS}"
)
for value in yaml_platforms.values():
if value is not None:
raise ValueError(
"Shorthand notation required ('build-on' and 'build-for' not supported) in "
f"charmcraft.yaml 'platforms'.\n\nMore info: {_SYNTAX_DOCS}"
)
# Validate format of keys in `yaml_platforms`
platforms = [Platform(platform) for platform in yaml_platforms]

return platforms
Loading

0 comments on commit 92d0a9f

Please sign in to comment.