Skip to content

Latest commit

 

History

History
477 lines (358 loc) · 19.2 KB

HACKING.md

File metadata and controls

477 lines (358 loc) · 19.2 KB
Raw

Setting up a Dev Environment

To work in the framework itself you will need Python >= 3.8. Linting, testing, and docs automation is performed using tox, which you should install. For improved performance on the tests, ensure that you have PyYAML installed with the correct extensions:

apt-get install libyaml-dev
pip install --force-reinstall --no-cache-dir pyyaml

Testing

The following are likely to be useful during development:

# Run linting and unit tests
tox

# Run tests, specifying whole suite or specific files
tox -e unit
tox -e unit test/test_charm.py

# Format the code using Ruff
tox -e fmt

# Compile the requirements.txt file for docs
tox -e docs-deps

# Generate a local copy of the Sphinx docs in docs/_build
tox -e docs

# run only tests matching a certain pattern
tox -e unit -- -k <pattern>

For more in depth debugging, you can enter any of tox's created virtualenvs provided they have been run at least once and do fun things - e.g. run pytest directly:

# Enter the linting virtualenv
source .tox/lint/bin/activate

...

# Enter the unit testing virtualenv and run tests
source .tox/unit/bin/activate
pytest
...

Pebble Tests

The framework has some tests that interact with a real/live Pebble server. To run these tests, you must have pebble installed and available in your path. If you have the Go toolchain installed, you can run go install github.com/canonical/pebble/cmd/pebble@master. This will install pebble to $GOBIN if it is set or $HOME/go/bin otherwise. Add $GOBIN to your path (e.g. export PATH=$PATH:$GOBIN or export PATH=$PATH:$HOME/go/bin in your .bashrc) and you are ready to run the real Pebble tests:

tox -e pebble

To do this even more manually, you could start the Pebble server yourself:

export PEBBLE=$HOME/pebble
export RUN_REAL_PEBBLE_TESTS=1
pebble run --create-dirs --http=:4000 &>pebble.log &

# Then
tox -e unit -- test/test_real_pebble.py
# or
source .tox/unit/bin/activate
pytest -v test/test_real_pebble.py

Using an ops branch in a charm

When making changes to ops, you'll commonly want to try those changes out in a charm.

From a Git branch

If your changes are in a Git branch, you can simply replace your ops version in requirements.txt (or pyproject.toml) with a reference to the branch, like:

#ops ~= 2.9
git+https://github.com/{your-username}/operator@{your-branch-name}

git is not normally available when charmcraft is packing the charm, so you'll need to also tell charmcraft that it's required for the build, by adding something like this to your charmcraft.yaml:

parts:
  charm:
    build-packages:
      - git

From local code

If your changes are only on your local device, you can inject your local ops into the charm after it has packed, and before you deploy it, by unzipping the .charm file and replacing the ops folder in the virtualenv. This small script will handle that for you:

#!/usr/bin/env bash

if [ "$#" -lt 2 ]
then
	echo "Inject local copy of Python Operator Framework source into charm"
	echo
    echo "usage: inject-ops.sh file.charm /path/to/ops/dir" >&2
    exit 1
fi

if [ ! -f "$2/framework.py" ]; then
    echo "$2/framework.py not found; arg 2 should be path to 'ops' directory"
    exit 1
fi

set -ex

mkdir inject-ops-tmp
unzip -q $1 -d inject-ops-tmp
rm -rf inject-ops-tmp/venv/ops
cp -r $2 inject-ops-tmp/venv/ops
cd inject-ops-tmp
zip -q -r ../inject-ops-new.charm .
cd ..
rm -rf inject-ops-tmp
rm $1
mv inject-ops-new.charm $1

Using a Juju branch

If your ops change relies on a change in a Juju branch, you'll need to deploy your charm to a controller using that version of Juju. For example, with microk8s:

  1. Build Juju and its dependencies
  2. Run make microk8s-operator-update
  3. Run GOBIN=/path/to/your/juju/_build/linux_amd64/bin:$GOBIN /path/to/your/juju bootstrap
  4. Add a model and deploy your charm as normal

Regression testing against existing charms

We rely on automation to update charm pins of a bunch of charms that use the operator framework. The script can be run locally too.

Contributing

Changes are proposed as pull requests on GitHub.

For coding style, we follow PEP 8 as well as a team Python style guide.

Pull requests should have a short title that follows the conventional commit style using one of these types:

  • chore
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • test

At present, we only add a scope in these cases:

  • If the PR is limited to changes in ops/_private/harness.py, also include the scope (harness)
  • If the PR is limited to changes in testing/, also include the scope (testing)

For example:

  • feat: add the ability to observe change-updated events
  • fix!: correct the type hinting for config data
  • docs(harness): clarify the types of exceptions that Harness.add_user_secret may raise
  • ci(testing): adjust the workflow that publishes ops-scenario

Note that the commit messages to the PR's branch do not need to follow the conventional commit format, as these will be squashed into a single commit to main using the PR title as the commit message.

Copyright

The format for copyright notices is documented in the LICENSE.txt. New files should begin with a copyright line with the current year (e.g. Copyright 2024 Canonical Ltd.) and include the full boilerplate (see APPENDIX of LICENSE.txt). The copyright information in existing files does not need to be updated when those files are modified -- only the initial creation year is required.

Documentation

In general, new functionality should always be accompanied by user-focused documentation that is posted to https://juju.is/docs/sdk. The content for this site is written and hosted on https://discourse.charmhub.io/c/doc. New documentation should get a new topic/post on this Discourse forum and then should be linked into the main docs navigation page(s) as appropriate. The ops library's SDK page content is pulled from the corresponding Discourse topic. Each page on juju.is has a link at the bottom that takes you to the corresponding Discourse page where docs can be commented on and edited (if you have earned those privileges).

Currently we don't publish separate versions of documentation for separate releases. Instead, new features should be sign-posted (for example, as done for File and directory existence in 1.4) with Markdown like this:

[note status="version"]1.4[/note]

next to the relevant content (e.g. headings, etc.).

The ops library's API reference is automatically built and published to ops.readthedocs.io. Please be complete with docstrings and keep them informative for users. The published docs are always for the in-development (main branch) of ops, and do not include any notes indicating changes or additions across ops versions - we encourage all charmers to promptly upgrade to the latest version of ops, and to refer to the release notes and changelog for learning about changes.

We do note when features behave differently when using different Juju versions. Use the .. jujuadded:: x.y directive to indicate that the feature is only available when using version x.y (or higher) of Juju, ..jujuchanged:: x.y when the feature's behaviour in ops changes, and ..jujuremoved:: x.y when the feature will be available in ops but not in that version (or later) of Juju. Unmarked features are assumed to work and be available in the current LTS version of Juju.

During the release process, changes also get a new entry in CHANGES.md. These are grouped into the same groupings as commit messages (feature, fix, documentation, performance, etc). The only exceptions are changes that are not visible to the built releases, such as CI workflow changes, or are implicit, such as bumping the ops version number. Each entry should be a short, single line, bullet point, and should reference the GitHub PR that introduced the change (as plain text, not a link).

As noted above, you can generate a local copy of the API reference docs with tox:

tox -e docs
open docs/_build/html/index.html

If dependencies are updated in pyproject.toml, you can run the following command before generating docs to recompile the requirements.txt file used for docs:

tox -e docs-deps

How to Pull in Style Changes

The documentation uses Canonical styling which is customised on top of the Furo Sphinx theme. The easiest way to pull in Canonical style changes is by using the Canonical documentation starter pack, see docs and repository.

TL;DR:

  • Clone the starter pack repository to a local directory: git clone [email protected]:canonical/sphinx-docs-starter-pack.
  • Copy the folder .sphinx under the starter pack repo to the operator repo docs/.sphinx.

How to Customise Configurations

There are two configuration files: docs/conf.py and docs/custom_conf.py, copied and customised from the starter pack repo.

To customise, change the file docs/custom_conf.py only, and theoretically, we should not change docs/conf.py (however, some changes are made to docs/conf.py, such as adding autodoc, PATH, fixing issues, etc.)

How to Pull in Dependency Changes

The Canonical documentation starter pack uses Make to build the documentation, which will run the script docs/.sphinx/build_requirements.py and generate a requirement file requirements.txt under docs/.sphinx/.

To pull in new dependency changes from the starter pack, change to the starter pack repository directory, and build with the following command. This will create a virtual environment, generate a dependency file, install the software dependencies, and build the documentation:

make html

Then, compare the generated file .sphinx/requirements.txtand the project.optional-dependencies.docs section of pyproject.toml and adjust the pyproject.toml file accordingly.

Dependencies

The Python dependencies of ops are kept as minimal as possible, to avoid bloat and to minimise conflict with the charm's dependencies. The dependencies are listed in pyproject.toml in the project.dependencies section.

Dev Tools

Formatting and Checking

Test environments are managed with tox and executed with pytest, with coverage measured by coverage. Static type checking is done using pyright, and extends the Python 3.8 type hinting support through the typing_extensions package.

Formatting uses Ruff.

All tool configuration is kept in project.toml. The list of dependencies can be found in the relevant tox.ini environment deps field.

Building

The build backend is setuptools, and the build frontend is build.

Publishing a Release

To make a release of the ops and/or ops-scenario packages, do the following:

  1. Check if there's a chore: update charm pins auto-generated PR in the queue. If it looks good, merge it and check that tests still pass. If needed, you can re-trigger the Update Charm Pins workflow manually to ensure latest charms and ops get tested.
  2. Visit the releases page on GitHub.
  3. Click "Draft a new release"
  4. The "Release Title" is the full version numbers of ops and/or ops-scenario, in the form ops <major>.<minor>.<patch> and ops-scenario <major>.<minor>.<patch> and a brief summary of the main changes in the release. For example: 2.3.12 Bug fixes for the Juju foobar feature when using Python 3.12
  5. Have the release create a new tag, in the form <major>.<minor>.<patch> for ops and scenario-<major>.<minor>.<patch> for ops-scenario. If releasing both packages, use the ops tag.
  6. If the last release was for both ops and ops-scenario, leave the previous tag choice on auto. If the last release was for only one package, change the previous tag to be the last time the same package(s) were being released.
  7. Use the "Generate Release Notes" button to get a copy of the changes into the notes field. The 'Release Documentation' section below details the form that the release notes and changelog should take.
  8. For ops, change version.py's version to the appropriate string. For ops-scenario, change the version in testing/pyproject.toml. Both packages use semantic versioning, and adjust independently (that is: ops 2.18 doesn't imply ops-scenario 2.18, or any other number).
  9. Add, commit, and push, and open a PR to get the changelogs and version bumps into main (and get it merged).
  10. Save the release notes as a draft, and have someone else in the Charm-Tech team proofread the release notes.
  11. If the release includes both ops and ops-scenario packages, then push a new tag in the form scenario-<major>.<minor>.<patch>. This is done by executing git tag scenario-x.y.z, then git push upstream tag scenario-x.y.z locally (assuming you have configured canonical/operator as a remote named upstream).
  12. When you are ready, click "Publish". GitHub will create the additional tag.

Pushing the tags will trigger automatic builds for the Python packages and publish them to PyPI (ops and ops-scenario) (authorisation is handled via a Trusted Publisher relationship). Note that it sometimes take a bit of time for the new releases to show up.

See .github/workflows/publish-ops.yaml and .github/workflows/publish-ops-scenario.yaml for details. (Note that the versions in the YAML refer to versions of the GitHub actions, not the versions of the ops library.)

You can troubleshoot errors on the Actions Tab.

  1. Announce the release on Discourse and Matrix.

  2. Open a PR to change the version strings to the expected next version, with ".dev0" appended (for example, if 3.14.1 is the next expected version, use '3.14.1.dev0').

Release Documentation

We produce several pieces of documentation for ops and ops-scenario releases, each serving a separate purpose and covering a different level.

Avoid using the word "Scenario", preferring "unit testing API" or "state transition testing". Users should install ops-scenario with pip install ops[testing] rather than using the ops-scenario package name directly.

git log

git log is used to see every change since a previous release. Obviously, no special work needs to be done so that this is available. A link to the GitHub view of the log will be included at the end of the GitHub release notes when the "Generate Release Notes" button is used, in the form:

**Full Changelog**: https://github.com/canonical/operator/compare/2.17.0...2.18.0

These changes include both ops and ops-scenario. If someone needs to see changes only for one of the packages, then the /testing/ folder can be filtered in/out.

CHANGES.md

A changelog is kept in version control that simply lists the changes in each release, other than chores. The changelog for ops is at the top level, in CHANGES.md, and the changelog for ops-scenario is in the /testing folder, CHANGES.md. There will be overlap between the two files, as many PRs will include changes to common infrastructure, or will adjust both ops and also the testing API in ops-scenario.

Adding the changes is done in preparation for a release. Use the "Generate Release Notes" button in the GitHub releases page, and copy the text to the CHANGES.md files.

  • Group the changes by the commit type (feat, fix, and so on) and use full names ("Features", not "feat", "Fixes", not "fix") for group headings.
  • Remove any chores.
  • Remove any bullets that do not apply to the package. For instance, if a bullet only affects ops[testing], don't include it in CHANGES.md when doing an ops release. The bullet should go in testing/CHANGES.md instead. If ops[testing] is not being released yet, put the bullet in a placeholder section at top of testing/CHANGES.md.
  • Strip the commit type prefix from the bullet point, and capitalise the first word.
  • Strip the username (who did each commit) if the author is a member of the Charm Tech team.
  • Replace the link to the pull request with the PR number in parentheses.
  • Where appropriate, collapse multiple tightly related bullet points into a single point that refers to multiple commits.
  • Where appropriate, add backticks for code formatting.

For example: the PR

* docs: clarify where StoredState is stored by @benhoyt in https://github.com/canonical/operator/pull/2006

is added to the "Documentation" section as:

* Clarify where StoredState is stored (#2006)

GitHub Release Notes

The GitHub release notes include the list of changes found in the changelogs, but:

  • If both ops and ops-scenario packages are being released, include all the changes in the same set of release notes. If only one package is being released, remove any bullets that apply only to the other package.
  • The links to the PRs are left in full.
  • Add a section above the list of changes that briefly outlines any key changes in the release.

Discourse Release Announcement

Post to the framework category with a subject matching the GitHub release title.

The post should resemble this:

The Charm Tech team has just released version x.y.z of ops!

It’s available from PyPI by using `pip install ops`, and `pip install ops[testing]`,
which will pick up the latest version. Upgrade by running `pip install --upgrade ops`.

The main improvements in this release are ...

Read more in the [full release notes on GitHub](link to the GitHub release).

In the post, outline the key improvements both in ops and ops-scenario - the point here is to encourage people to check out the full notes and to upgrade promptly, so ensure that you entice them with the best that the new versions have to offer.