feat: upgrade metering stack to 2024.1 (rackerlabs#398)

* feat: add postgres operator and cluster * feat: update postgres operator to 1.12.2 * fix: bump shared buffers max conns and vol size * fix: add cluster to pg system ns * fix: use new postgres-cluster hostname * fix: move cluster to openstack namespace The service is being looked for here * fix: disable deprecated ha queues * fix: explicitly set durable queues false * fix: disable amqp durable queues for cinder * fix: make admin endpoint keystone-api keystone is not a valid endpoint with a DNS entry but keystone-api is * feat: begin to add custom gnocchi container * feat: add gh workflow for building gnocchi * fix: convert repo name to lowercase * feat: replace gnocchi api image with custom one * fix: apache2 needs this to start * fix: install apache2-mod-wsgi after py3 * fix: need the gnocchi user to exist * fix: update path to gnocchi module * fix: push events don't inherit default inputs * fix: only build on push if container changed * fix: specify required drivers during install * fix: pass default push_version to build-args * fix: add apt dependencies for source builds * fix: dir is created automatically now * fix: memcache is required by keystone * fix: pymemcache is also needed * fix: redirect apache2 errorlog to stderr * feat: update remaining image tags * fix: db-init expects python * fix: pymysql is needed by db-init * fix: sqlalchemy>2 has breaking changes e.g. `AttributeError: 'Engine' object has no attribute 'execute'` * fix: re-enable db-init job * fix: refactor image tag name and build version opt * fix: normalize image tag name * fix: reduce gnocchi image footprint * feat: let changes to main branch trigger build Scoped to only changes in the release and/or container file of course. * feat: begin to add custom ceilometer image * fix: update ceilometer tags to custom image * fix: add additional requirements for ceilometer * fix: update event definitions to match 2024.1 * fix: update gnocchi_resources to match 2024.1 * fix: update meters to match 2024.1 * fix: replace remaining images with 2024.1 * fix: remove duplicate dictionary from ipmi * fix: change event publisher to gnocchi * feat: add aio overlay for postgres * docs: add new postgres operator install info Using an operator is preferred here. * docs: update gnocchi installation info Refactor verbiage, add architecture diagram, and improve page layout. * fix: change repo for gnocchi and ceilometer images * fix: remove empty line from end of svg
LukeRepko · Aug 9, 2024 · bbffd28 · bbffd28
1 parent cb4c453
commit bbffd28
Show file tree

Hide file tree

Showing 17 changed files with 582 additions and 210 deletions.
diff --git a/.github/workflows/release-ceilometer.yaml b/.github/workflows/release-ceilometer.yaml
@@ -0,0 +1,81 @@
+#
+name: Create and publish a Ceilometer image
+
+on:
+  push:
+    paths:
+    - '.github/workflows/release-ceilometer.yaml'
+    - 'Containerfiles/CeilometerRXT-Containerfile'
+    branches:
+      - development
+      - main
+  workflow_dispatch:
+    inputs:
+      imageTag:
+        description: 'Set tag for the image'
+        required: true
+        default: '2024.1-ubuntu_jammy'
+        type: choice
+        options:
+          - 2024.1-ubuntu_jammy
+      buildVersion:
+        description: 'Set version to build'
+        required: true
+        default: 'stable/2024.1'
+        type: choice
+        options:
+          - stable/2024.1
+          - master
+
+# Defines two custom environment variables for the workflow. These are used for the Container registry domain, and a name for the Docker image that this workflow builds.
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository  }}
+  DEF_TAG_NAME: 2024.1-ubuntu_jammy
+  DEF_BUILD_VERSION: stable/2024.1
+
+# There is a single job in this workflow. It's configured to run on the latest available version of Ubuntu.
+jobs:
+  build-and-push-image:
+    runs-on: ubuntu-latest
+    # Sets the permissions granted to the `GITHUB_TOKEN` for the actions in this job.
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      # Uses the `docker/login-action` action to log in to the Container registry registry using the account and password that will publish the packages. Once published, the packages are scoped to the account defined here.
+      - name: Log in to the Container registry
+        uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      # ghcr only allows lowercase repository names
+      - name: lowercase repo name
+        run: |
+          echo "IMAGE_NAME=${GITHUB_REPOSITORY,,}" >>${GITHUB_ENV}
+      # This step uses [docker/metadata-action](https://github.com/docker/metadata-action#about) to extract tags and labels that will be applied to the specified image. The `id` "meta" allows the output of this step to be referenced in a subsequent step. The `images` value provides the base name for the tags and labels.
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+      # This step uses the `docker/build-push-action` action to build the image, based on your repository's `Dockerfile`. If the build succeeds, it pushes the image to GitHub Packages.
+      # It uses the `context` parameter to define the build's context as the set of files located in the specified path. For more information, see "[Usage](https://github.com/docker/build-push-action#usage)" in the README of the `docker/build-push-action` repository.
+      # It uses the `tags` and `labels` parameters to tag and label the image with the output from the "meta" step.
+      - name: Dynamically set MY_DATE environment variable
+        run: echo "MY_DATE=$(date +%s)" >> $GITHUB_ENV
+      - name: Build and push Docker image
+        uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
+        with:
+          context: .
+          file: Containerfiles/CeilometerRXT-Containerfile
+          push: true
+          tags: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/ceilometer:${{ github.event.inputs.imageTag || env.DEF_TAG_NAME }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/ceilometer:${{ github.event.inputs.imageTag || env.DEF_TAG_NAME }}-${{ env.MY_DATE }}
+          labels: ${{ steps.meta.outputs.labels }}
+          build-args: |
+            VERSION=${{ github.event.inputs.buildVersion || env.DEF_BUILD_VERSION}}
diff --git a/.github/workflows/release-gnocchi.yaml b/.github/workflows/release-gnocchi.yaml
@@ -0,0 +1,82 @@
+#
+name: Create and publish a Gnocchi image
+
+on:
+  push:
+    paths:
+    - '.github/workflows/release-gnocchi.yaml'
+    - 'Containerfiles/GnocchiRXT-Containerfile'
+    branches:
+      - development
+      - main
+  workflow_dispatch:
+    inputs:
+      imageTag:
+        description: 'Set tag for the image'
+        required: true
+        default: '2024.1-ubuntu_jammy'
+        type: choice
+        options:
+          - 2024.1-ubuntu_jammy
+      buildVersion:
+        description: 'Set version to build'
+        required: true
+        default: '4.6.2'
+        type: choice
+        options:
+          - 4.6.2
+          - stable/4.6
+          - master
+
+# Defines two custom environment variables for the workflow. These are used for the Container registry domain, and a name for the Docker image that this workflow builds.
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository  }}
+  DEF_TAG_NAME: 2024.1-ubuntu_jammy
+  DEF_BUILD_VERSION: 4.6.2
+
+# There is a single job in this workflow. It's configured to run on the latest available version of Ubuntu.
+jobs:
+  build-and-push-image:
+    runs-on: ubuntu-latest
+    # Sets the permissions granted to the `GITHUB_TOKEN` for the actions in this job.
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      # Uses the `docker/login-action` action to log in to the Container registry registry using the account and password that will publish the packages. Once published, the packages are scoped to the account defined here.
+      - name: Log in to the Container registry
+        uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      # ghcr only allows lowercase repository names
+      - name: lowercase repo name
+        run: |
+          echo "IMAGE_NAME=${GITHUB_REPOSITORY,,}" >>${GITHUB_ENV}
+      # This step uses [docker/metadata-action](https://github.com/docker/metadata-action#about) to extract tags and labels that will be applied to the specified image. The `id` "meta" allows the output of this step to be referenced in a subsequent step. The `images` value provides the base name for the tags and labels.
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+      # This step uses the `docker/build-push-action` action to build the image, based on your repository's `Dockerfile`. If the build succeeds, it pushes the image to GitHub Packages.
+      # It uses the `context` parameter to define the build's context as the set of files located in the specified path. For more information, see "[Usage](https://github.com/docker/build-push-action#usage)" in the README of the `docker/build-push-action` repository.
+      # It uses the `tags` and `labels` parameters to tag and label the image with the output from the "meta" step.
+      - name: Dynamically set MY_DATE environment variable
+        run: echo "MY_DATE=$(date +%s)" >> $GITHUB_ENV
+      - name: Build and push Docker image
+        uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
+        with:
+          context: .
+          file: Containerfiles/GnocchiRXT-Containerfile
+          push: true
+          tags: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/gnocchi:${{ github.event.inputs.imageTag || env.DEF_TAG_NAME }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/gnocchi:${{ github.event.inputs.imageTag || env.DEF_TAG_NAME }}-${{ env.MY_DATE }}
+          labels: ${{ steps.meta.outputs.labels }}
+          build-args: |
+            VERSION=${{ github.event.inputs.buildVersion || env.DEF_BUILD_VERSION}}
diff --git a/Containerfiles/CeilometerRXT-Containerfile b/Containerfiles/CeilometerRXT-Containerfile
@@ -0,0 +1,26 @@
+FROM ubuntu:22.04
+ARG VERSION=""
+ENV CEILOMETER_VERSION ${VERSION:-master}
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt update && \
+    apt install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y
+RUN apt update && \
+    apt install -y \
+    python3.10 \
+    python3.10-dev \
+    python3-memcache \
+    git \
+    wget \
+    curl \
+    locales \
+    build-essential \
+    pkg-config \
+    libvirt-dev
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10 && \
+    pip install \
+    "ceilometer @ git+https://opendev.org/openstack/ceilometer.git@${CEILOMETER_VERSION}" \
+    "libvirt-python"
+RUN ln -s /usr/bin/python3 /usr/bin/python && \
+    adduser ceilometer --disabled-password && \
+    apt clean && rm -rf /var/lib/apt/lists/*
diff --git a/Containerfiles/GnocchiRXT-Containerfile b/Containerfiles/GnocchiRXT-Containerfile
@@ -0,0 +1,39 @@
+FROM ubuntu:22.04
+ARG VERSION=""
+ENV GNOCCHI_VERSION ${VERSION:-master}
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt update && \
+    apt install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y
+RUN apt update && \
+    apt install -y \
+    python3.10 \
+    python3.10-dev \
+    python3-memcache \
+    git \
+    wget \
+    curl \
+    apache2 \
+    locales \
+    libapache2-mod-wsgi-py3 \
+    build-essential \
+    libffi-dev \
+    libpq-dev \
+    postgresql \
+    memcached \
+    librados-dev \
+    liberasurecode-dev \
+    python3-rados \
+    ceph \
+    libsnappy-dev \
+    libprotobuf-dev
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10 && \
+    pip install \
+    "gnocchi[postgresql,ceph,keystone] @ git+https://github.com/gnocchixyz/gnocchi.git@${GNOCCHI_VERSION}" \
+    "pymemcache" \
+    "pymysql" \
+    "sqlalchemy>=1.4.24,<2"
+RUN ln -s /usr/bin/python3 /usr/bin/python && \
+    adduser gnocchi --disabled-password && \
+    sed -i 's@^ErrorLog.*@ErrorLog /dev/stderr@' /etc/apache2/apache2.conf && \
+    apt clean && rm -rf /var/lib/apt/lists/*