Skip to content

Commit

Permalink
Merge branch 'main' into dpe-4816-missing-jinja
Browse files Browse the repository at this point in the history
  • Loading branch information
@dragomirp
dragomirp committed Jul 5, 2024
2 parents 35e3e86 + def028c commit 15b5253
Show file tree
Hide file tree
Showing 72 changed files with 3,793 additions and 878 deletions.
19 changes: 19 additions & 0 deletions .github/workflows/sync_docs.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Copyright 2024 Canonical Ltd.
# See LICENSE file for licensing details.
name: Sync docs from Discourse

on:
workflow_dispatch:
schedule:
- cron: 00 01 * * *

jobs:
sync-docs:
name: Sync docs from Discourse
uses: canonical/data-platform-workflows/.github/workflows/_sync_docs.yaml@main
secrets:
discourse-api-user: ${{ secrets.DISCOURSE_API_USERNAME }}
discourse-api-key: ${{ secrets.DISCOURSE_API_KEY }}
permissions:
contents: write # Needed to push branch & tag
pull-requests: write # Needed to create PR
127 changes: 127 additions & 0 deletions docs/explanation/e-architecture.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Architecture

[PostgreSQL](https://www.postgresql.org/) is one of the most popular open source database. The “[Charmed PostgreSQL](https://charmhub.io/postgresql)” is a Juju-based operator to deploy and support PostgreSQL from [day 0 to day 2](https://codilime.com/blog/day-0-day-1-day-2-the-software-lifecycle-in-the-cloud-age/), it is based on the [PostgreSQL Community Edition](https://www.postgresql.org/community/) using the [Patroni](https://github.com/zalando/patroni) to manage PostgreSQL cluster based on [PostgreSQL synchronous replication](https://patroni.readthedocs.io/en/latest/replication_modes.html#postgresql-synchronous-replication).

![image|690x423, 100%](upload://fqMd5JlHeegw0PlUjhWKRu858Nc.png)

<a name="hld"></a>
## HLD (High Level Design)

The charm design leverages on the SNAP “[charmed-postgresql](https://snapcraft.io/charmed-postgresql)” which is deployed by Juju on the specified VM/MAAS/bare-metal machine based on Ubuntu Jammy/22.04. SNAP allows to run PostgreSQL service(s) in a secure and isolated environment ([strict confinement](https://ubuntu.com/blog/demystifying-snap-confinement)). The installed SNAP:
```
> juju ssh postgresql/0 snap list charmed-postgresql
Name Version Rev Tracking Publisher Notes
charmed-postgresql 14.9 70 latest/stable dataplatformbot held
```

The SNAP ships the following components:

* PostgreSQL (based on Ubuntu APT package "[postgresql](https://packages.ubuntu.com/jammy/postgresql)")
* PgBouncer (based on Canonical [backport](https://launchpad.net/~data-platform/+archive/ubuntu/pgbouncer))
* Patroni (based on Canonical [backport](https://launchpad.net/~data-platform/+archive/ubuntu/patroni))
* pgBackRest (based on Canonical [backport](https://launchpad.net/~data-platform/+archive/ubuntu/pgbackrest))
* Prometheus PostgreSQL Exporter (based on Canonical [backport](https://launchpad.net/~data-platform/+archive/ubuntu/postgres-exporter))
* Prometheus PgBouncer Exporter (based on Canonical [backport](https://launchpad.net/~data-platform/+archive/ubuntu/pgbouncer-exporter))
* Prometheus Grafana dashboards and Loki alert rules are part of the charm revision (and missing in SNAP).

Versions of all the components above are carefully chosen to fit functionality of each other.

The Charmed PostgreSQL unit consisting of a several services which are enabled/activated accordingly to the setup:

```
> juju ssh postgresql/0 snap services charmed-postgresql
Service Startup Current Notes
charmed-postgresql.patroni enabled active -
charmed-postgresql.pgbackrest-service enabled active -
charmed-postgresql.prometheus-postgres-exporter enabled active -
```

The `patroni` snap service is a main PostgreSQL instance which is normally up and running right after the charm deployment.

The `pgbackrest` snap service is a backup framework for PostgreSQL. It is disabled if [Backup](/t/9683) is not configured.

The `prometheus-postgres-exporter` service is activated after the relation with [COS Monitoring](/t/10600) only.

> **:information_source: Note:** it is possible to star/stop/restart snap services manually but it is NOT recommended to avoid a split brain with a charm state machine! Do it with a caution!!!
> **:warning: Important:** all snap resources must be executed under the special user `snapd_daemon` only!
The snap "charmed-postgresql" also ships list of tools used by charm:
* `charmed-postgresql.psql` (alias `psq`) - is PostgreSQL interactive terminal.
* `charmed-postgresql.patronictl` - a tool to monitor and manage Patroni.
* `charmed-postgresql.pgbackrest` - a tool to backup/restore PostgreSQL DB.

<a name="integrations"></a>
## Integrations

### PgBouncer

[PgBouncer](http://www.pgbouncer.org/) is a lightweight connection pooler for PostgreSQL that provides transparent routing between your application and back-end PostgreSQL Servers. The "[PgBouncer](https://charmhub.io/pgbouncer)" is an independent charm "Charmed PostgreSQL" can be related with.

### TLS Certificates Operator

[TLS Certificates](https://charmhub.io/tls-certificates-operator) charm responsible for distributing certificates through relationship. Certificates are provided by the operator through Juju configs. For the playground deployments, the [self-signed operator](https://charmhub.io/self-signed-certificates) is available as well.

### S3 Integrator

[S3 Integrator](https://charmhub.io/s3-integrator) is an integrator charm for providing S3 credentials to Charmed PostgreSQL which seek to access shared S3 data. Store the credentials centrally in the integrator charm and relate consumer charms as needed.

### Data Integrator

[Data Integrator](https://charmhub.io/data-integrator) charm is a solution to request DB credentials for non-native Juju applications. Not all applications implement a data_interfaces relation but allow setting credentials via config. Also, some of the applications are run outside of juju. This integrator charm allows receiving credentials which can be passed into application config directly without implementing juju-native relation.

### PostgreSQL Test App

The charm "[PostgreSQL Test App](https://charmhub.io/postgresql-test-app)" is a Canonical test application to validate the charm installation / functionality and perform the basic performance tests.

### Grafana

Grafana is an open-source visualization tools that allows to query, visualize, alert on, and visualize metrics from mixed datasources in configurable dashboards for observability. This charms is shipped with its own Grafana dashboard and supports integration with the [Grafana Operator](https://charmhub.io/grafana-k8s) to simplify observability. Please follow [COS Monitoring](/t/10600) setup.

### Loki

Loki is an open-source fully-featured logging system. This charms is shipped with support for the [Loki Operator](https://charmhub.io/loki-k8s) to collect the generated logs. Please follow [COS Monitoring](/t/10600) setup.

### Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit with a dimensional data model, flexible query language, efficient time series database and modern alerting approach. This charm is shipped with a Prometheus exporters, alerts and support for integrating with the [Prometheus Operator](https://charmhub.io/prometheus-k8s) to automatically scrape the targets. Please follow [COS Monitoring](/t/10600) setup.

<a name="lld"></a>
## LLD (Low Level Design)

Please check the charm state machines displayed on [workflow diagrams](/t/9305). The low-level logic is mostly common for both VM and K8s charms.

<!--- TODO: Describe all possible installations? Cross-model/controller? --->

### Juju Events

Accordingly to the [Juju SDK](https://juju.is/docs/sdk/event): “an event is a data structure that encapsulates part of the execution context of a charm”.

For this charm, the following events are observed:

1. [on_install](https://juju.is/docs/sdk/install-event): install the snap "charmed-postgresql" and perform basic preparations to bootstrap the cluster on the first leader (or join the already configured cluster).
2. [leader-elected](https://juju.is/docs/sdk/leader-elected-event): generate all the secrets to bootstrap the cluster.
3. [leader-settings-changed](https://juju.is/docs/sdk/leader-settings-changed-event): Handle the leader settings changed event.
4. [start](https://juju.is/docs/sdk/start-event): Init/setting up the cluster node.
5. [config_changed](https://juju.is/docs/sdk/config-changed-event): usually fired in response to a configuration change using the GUI or CLI. Create and set default cluster and cluster-set names in the peer relation databag (on the leader only).
6. [update-status](https://juju.is/docs/sdk/update-status-event): Takes care of workload health checks.
<!--- 7. database_storage_detaching: TODO: ops? event?
8. TODO: any other events?
--->

### Charm Code Overview

The "[src/charm.py](https://github.com/canonical/postgresql-operator/blob/main/src/charm.py)" is the default entry point for a charm and has the `PostgresqlOperatorCharm` Python class which inherits from CharmBase.

CharmBase is the base class from which all Charms are formed, defined by [Ops](https://juju.is/docs/sdk/ops) (Python framework for developing charms). See more information in [Charm](https://juju.is/docs/sdk/constructs#heading--charm).

The `__init__` method guarantees that the charm observes all events relevant to its operation and handles them.

The VM and K8s charm flavors shares the codebase via [charm libraries](https://juju.is/docs/sdk/libraries) in [lib/charms/postgresql_k8s/v0/](https://github.com/canonical/postgresql-k8s-operator/blob/main/lib/charms/postgresql_k8s/v0/postgresql.py) (of K8s flavor of the charm!):
```
> charmcraft list-lib postgresql-k8s
Library name API Patch
postgresql 0 12
postgresql_tls 0 7
```
6 changes: 4 additions & 2 deletions docs/explanation/e-interfaces.md → docs/explanation/e-interfaces-endpoints.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ juju deploy postgresql -n 3 --channel 14
juju deploy mycharm

# Relate PostgreSQL with your application
juju relate postgresql-k8s:database mycharm:database
juju relate postgresql:database mycharm:database

# Check established relation (using postgresql_client interface):
juju status --relations
Expand All @@ -30,9 +30,11 @@ juju status --relations
# > postgresql:database mycharm:database postgresql_client regular
```

Find all details about default and extra DB user roles in "[Charm Users explanations](/t/10798)".

## Legacy interfaces

**Note:** Legacy relations are deprecated and will be discontinued on future releases. Usage should be avoided.
**Note:** Legacy relations are deprecated and will be discontinued on future releases. Usage should be avoided. Check the legacy interfaces implementation limitations in the "[Legacy charm](/t/10690)" document.

### Legacy `pgsql` interface (`db` and `db-admin` endpoints):

Expand Down
43 changes: 43 additions & 0 deletions docs/explanation/e-juju-details.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Juju tech details

[Juju](https://juju.is/) is an open source orchestration engine for software operators that enables the deployment, integration and lifecycle management of applications at any scale, on any infrastructure using charms.

This [charm](https://charmhub.io/postgresql) is an operator - business logic encapsulated in reusable software packages that automate every aspect of an application's life. Charms are shared via [CharmHub](https://charmhub.io/).

See also:

* [Juju Documentation](https://juju.is/docs/juju) and [Blog](https://ubuntu.com/blog/tag/juju)
* [Charm SDK](https://juju.is/docs/sdk)

## Breaking changes between Juju 2.9.x and 3.x

As this charm documentation is written for Juju 3.x, users of 2.9.x will encounter noteworthy changes when following the instructions. This section explains those changes.

Breaking changes have been introduced in the Juju client between versions 2.9.x and 3.x. These are caused by the renaming and re-purposing of several commands - functionality and command options remain unchanged.

In the context of this guide, the pertinent changes are shown here:

|2.9.x|3.x|
| --- | --- |
|**add-relation**|**integrate**|
|**relate**|**integrate**|
|**run**|**exec**|
|**run-action --wait**|**run**|

See the [Juju 3.0 release notes](https://juju.is/docs/juju/roadmap#heading--juju-3-0-0---22-oct-2022) for the comprehensive list of changes.

The response is to therefore substitute the documented command with the equivalent 2.9.x command. For example:

### Juju 3.x:
```shell
juju integrate postgresql:database postgresql-test-app

juju run postgresql/leader get-password
```
### Juju 2.9.x:
```shell
juju relate postgresql:database postgresql-test-app

juju run-action --wait postgresql/leader get-password
```
> :tipping_hand_man: [The document based on OpenStack guide.](https://docs.openstack.org/charm-guide/latest/project/support-notes.html#breaking-changes-between-juju-2-9-x-and-3-x)
84 changes: 84 additions & 0 deletions docs/explanation/e-legacy-charm.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
## Charm types "legacy" vs "modern"
There are [two types of charms](https://juju.is/docs/sdk/charm-taxonomy#heading--charm-types-by-generation) stored under the same charm name `postgresql`:

1. [Reactive](https://juju.is/docs/sdk/charm-taxonomy#heading--reactive) charm in the channel `latest/stable` (called `legacy`)
2. [Ops-based](https://juju.is/docs/sdk/ops) charm in the channel `14/stable` (called `modern`)

The legacy charm provided endpoints `db` and `db-admin` (for the interface `pgsql`). The modern charm provides old endpoints as well + new endpoint `database` (for the interface `postgresql_client`). Read more details about the available [endpoints/interfaces](https://charmhub.io/postgresql/docs/e-interfaces).

**Note**: Please choose one endpoint to use. No need to relate all of them simultaneously!

## The default track "latest" vs "14"

The [default track](https://docs.openstack.org/charm-guide/yoga/project/charm-delivery.html) has been switched from the `latest` to `14`. It is [to ensure](https://discourse.charmhub.io/t/request-switch-default-track-from-latest-to-14-for-postgresql-k8s-charms/10314) all new deployments use a modern codebase. We strongly advise against using the latest track due to its implicit nature. In doing so, a future charm upgrade may result in a PostgreSQL version incompatible with an integrated application. Track 14 guarantees PostgreSQL 14 deployment only. The track `latest` will be closed after all applications migrated from Reactive to Ops-based charm.

<a name="howtomigrate"></a>
## How to migrate from "legacy" to "modern" charm

The "modern" charm provides temporary support for the legacy interfaces:

* **quick try**: relate the current application with new charm using endpoint `db` (set the channel to `14/stable`). No extra changes necessary:

```
postgresql:
charm: postgresql
channel: 14/stable
```

* **proper migration**: migrate the application to the new interface "[postgresql_client](https://github.com/canonical/charm-relation-interfaces)". The application will connect PostgreSQL using "[data_interfaces](https://charmhub.io/data-platform-libs/libraries/data_interfaces)" library from "[data-platform-libs](https://github.com/canonical/data-platform-libs/)" via endpoint `database`.

**Warning**: NO in-place upgrades possible! Reactive charm cannot be upgraded to Operator-framework-based one. To move DB data, the second/modern DB application must be launched nearby and data should be copied from "legacy" application to the "modern" one. Canonical Data Platform team is preparing copy&paste guide right now. Please [contact us](https://chat.charmhub.io/charmhub/channels/data-platform) if you need migration instructions.

**Note**: the `trust` option must be enabled if [ Role Based Access Control (RBAC)](https://kubernetes.io/docs/concepts/security/rbac-good-practices/) is in use on your Kubernetes.

## How to deploy old "legacy" postgresql charm

Deploy the charm using the channel `latest/stable`:

```
postgresql:
charm: postgresql
channel: latest/stable
```

**Note**: remove Charm store prefix `cs:` from the bundle. Otherwise the modern charm will be chosen by Juju (due to the default track pointing to `14/stable` and not `latest/stable`). The common error message is: `cannot deploy application "postgresql": unknown option "..."`.

## Config options supported by modern charm

The legacy charm config options were not moved to the modern charm due to no need. The modern charm applies the best possible configuration automatically.

Feel free to [contact us](https://chat.charmhub.io/charmhub/channels/data-platform) about the DB tuning/config options.

## Extensions supported by modern charm

The legacy charm provided plugins/extensions enabling through the relation (interface `pgsql`). It is NOT supported by the modern charm (neither `pgsql` nor `postgresql_client` interfaces). Please enable the necessary extensions using appropriate `plugin_*_enable` [config option](https://charmhub.io/postgresql/configure) of the modern charm. After enabling the modern charm, it will provide plugins support for both `pgsql` and `postgresql_client` interfaces.

Please find the list of supported PostgreSQL [Extensions](/t/10946) by modern charm.

Feel free to [contact us](/t/11863) with a list of required extensions.

## Roles supported by modern charm

In the legacy charm, the user could request roles by setting the `roles` field to a comma separated list of desired roles. It is NOT supported by the modern charm implementation of the legacy `pgsql` interface. The same functionality is provided via the modern `postgresql_client` using "[extra-user-roles](/t/10798#extra-user-roles)".

For more information about migrating the new interface, see [this guide](/t/11865).

## Supported PostgreSQL versions by modern charm

At the moment, the modern charms support PostgreSQL 14 (based on Jammy/22.04 series) only.

Please [contact us](https://chat.charmhub.io/charmhub/channels/data-platform) if you need different versions/series.

## Supported architectures

Currently, the charm supports architecture `amd64` (all revisions) and `arm64` (from revision 396+).

See the technical details in [Supported architectures](/t/11743).

## How to report issues and contact authors

The "legacy charm" (from `latest/stable`) is stored on [Launchpad](https://git.launchpad.net/postgresql-charm/), here is the link to report all [legacy charm issues](https://bugs.launchpad.net/postgresql-charm).

The "modern charm" (from `14/stable`) is stored on [GitHub](https://github.com/canonical/postgresql-operator), here is the link to report [modern charm issues](https://github.com/canonical/postgresql-operator/issues/new/choose).

Do you have questions? [Contact us](/t/11863)!
Loading

0 comments on commit 15b5253

Please sign in to comment.