Allow path-only datasets in manifest schema

Summary

Allow a manifest's datasets to be specified as paths, allowing metadata to be gathered dynamically from these datasets instead of hard-coded in a manifest serialisation.

Contents (#101)

IMPORTANT: There are 2 breaking changes.

Enhancements

• BREAKING CHANGE: Make Twine.available_strands a set instead of a tuple
• Update manifest schema to allow path-only datasets
• Raise error if a dataset specified in a manifest strand of the twine is missing from the corresponding manifest
• Add Twine.available_manifest_strands property

Refactoring

• BREAKING CHANGE: Move dataset files tag checking into octue-sdk-python at octue.runner.Runner._validate_dataset_file_tags

Refactoring

• Move manifest and twine migrations into migrations subpackage

Use latest JSON metaschema

Summary

Use the latest JSON metaschema and switch to HTTPS to access it.

Contents (#100)

Enhancements

• Use latest JSON metaschema 2020-12 and access it via HTTPS

Fixes

• Fix labels ref in manifest schema

Dependencies

• Upgrade to jsonschema==4.4.0

Remove credentials environment variable name from children schema

Summary

Remove the credentials environment variable name from the children schema. The default GOOGLE_APPLICATION_CREDENTIALS is the only name now used in octue-sdk-python.

Contents (#98)

Enhancements

• Remove the credentials environment variable name from the children schema

Simplify twine and manifest schemas

Summary

Simplify specification of datasets in twines and manifests.

Contents (#95)

IMPORTANT: There are 2 breaking changes.

Enhancements

• BREAKING CHANGE: Require "datasets" field to be a dictionary in manifest.json files
• BREAKING CHANGE: Require "datasets" field in the "manifest" strands of twine.json to be a dictionary
• Raise error if JSON objects contain duplicate keys

Fixes

• Use correct method to transform manifests during validation

Skip pull request description update if skip indicator in body

Contents (#94)

Operations

• Skip pull request description update if skip indicator in body

Add docstring linting and PEP8 naming pre-commit checks

Contents (#93)

Operations

• Add and apply pydocstyle docstring linting to pre-commit
• Add and apply pep8-naming flake8 extension to pre-commit

Add ability to validate monitors

Contents (#92)

New features

• Add ability to validate monitor updates

Operations

• Use latest conventional commits pre-commit version

Quality Checklist

• New features are fully tested

Update CI/CD workflows

Contents

Operations

• Unify skip commands in CI/CD workflows
• Update usage of release notes compiler in workflow
• Fix mkver.conf file so it keeps versions below 1.0.0 (i.e. keeps the package in beta)

Use new PR description workflow

Contents

Operations

• Use new PR description workflow

Use conventional-commits repo for CI checks

Contents

Operations

• Use conventional-commits repo for CI checks