Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Gnocchi helm chart to Genestack #109

Merged
merged 10 commits into from
Mar 7, 2024

Conversation

LukeRepko
Copy link
Contributor

@LukeRepko LukeRepko commented Mar 5, 2024

This PR introduces helm configs for Gnocchi (OpenStack Metrics API) and its dependency, PostgreSQL. Gnocchi is a dependency of Ceilometer, which we will be adding to Genestack soon.

Note that PostgreSQL is being used as the indexer storage back-end in favor of our existing MariaDB operator implementation for several reasons.

  1. Gnocchi cites PostgreSQL as the preferred driver (though they don't state why exactly).
  2. I could not get the MySQL driver working with the helm chart, despite several attempts. It seems like it may be possible, but instructions are unclear, and there may be a bit of overhaul with the chart required to get it to work as expected. MySQL is supported by Gnocchi though, so it's within the realm of possibility.

Iff we go with PostgreSQL long term, we will want to investigate the upstream operator. The existing chart works and is being maintained to some extent, but the operator no doubt has more active contributors and will likely suit Genestack better.

@LukeRepko LukeRepko marked this pull request as draft March 5, 2024 14:42
@LukeRepko
Copy link
Contributor Author

Not ready for merge just yet, will be following up with one or two more commits to add documentation where appropriate regarding installing postgresql and gnocchi.

@LukeRepko LukeRepko force-pushed the gnocchi branch 5 times, most recently from 408224c to 35ad775 Compare March 5, 2024 17:39
@LukeRepko
Copy link
Contributor Author

LukeRepko commented Mar 5, 2024

Getting a 403 from the Gnocchi API pod over its service port 8041 -- looks like apache is denying requests due to...

[Tue Mar 05 20:19:45.567830 2024] [authz_core:error] [pid 8:tid 140141779719936] [client 10.233.64.50:40838] AH01630: client denied by server configuration: /var/lib/kolla/venv/lib/python2.7

That path doesn't exist because.... we have python3.x now. There is a place to update the apache configuration in the gnocchi-helm-overrides.yaml -- doing so now.

@LukeRepko
Copy link
Contributor Author

Past the previous issue, now getting a 500 response. There's a traceback in the gnocchi api logs related to there not being a section for gnocchi+keystone within /etc/gnocchi/api-paste.ini (which is derived from the helm-config-overrides). There's an updated api-paste in gnocchi/rest/ -- investigating.

@BjoernT BjoernT changed the title Introduce Gnocchi helm chart to Genestack [WIP] Introduce Gnocchi helm chart to Genestack Mar 6, 2024
@LukeRepko
Copy link
Contributor Author

Getting a 200 finally:

curl -D - -sH "x-auth-token: $token" http://gnocchi-api.openstack.svc.cluster.local:8041/healthcheck
HTTP/1.1 200 OK
Date: Wed, 06 Mar 2024 01:24:12 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Length: 0
Vary: Accept-Encoding
Content-Type: text/plain; charset=UTF-8

Note: Looking at the version response, the build is 4.4.4. The gnocchi_api image being used is quay.io/openstack.kolla/ubuntu-source-gnocchi-api:yoga - built only about a month ago. Gnocchi is up to 4.6.1, so this is pretty far behind.

ubuntu@openstack-flex-node-0:~$ curl -sH "x-auth-token: $token" http://gnocchi-api.openstack.svc.cluster.local:8041/ | jq
{
  "build": "4.4.4",
  "versions": [
    {
      "status": "CURRENT",
      "links": [
        {
          "rel": "self",
          "href": "http://gnocchi-api.openstack.svc.cluster.local:8041/v1/"
        }
      ],
      "id": "v1.0",
      "updated": "2015-03-19"
    }
  ]
}```

Copy link
Contributor

@cloudnull cloudnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple nits, and one general question.

Additionally, it looks like we're following openstack-helm's deployment of postgres, could we use the upstream postgres operator? https://postgres-operator.readthedocs.io/en/latest/

docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
docs/deploy-required-infrastructure.md Show resolved Hide resolved
LukeRepko added 10 commits March 6, 2024 09:41
This chart adds PostgreSQL. It's only planned use case at the moment is
for Gnocchi indexer storage. A few important notes about this chart are
below for consideration.

 - MySQL should be a supported driver for the Gnocchi indexer but I was
   unable to get that working. Considering Gnocchi cites preferring
   PostgreSQL as an indexer, and it's the default in the chart, opting
   for that at the moment in lieu of fighting the chart to do something
   it might need major overhauls to accomplish.

 - There is a an upstream operator for PG we should look into mid to
   long-term. This chart is only needed to make Gnocchi go, so that we
   can integrate Ceilometer.
This chart begins to onboard Gnocchi, the Openstack Metrics API. For now
we are using the PostgreSQL driver for indexer storage, as it's unclear
how to change it over to MySQL. Moreover, Gnocchi recommends using
PostgreSQL anyway.

Should it be determined later that we can safely use our existing
MariaDB implementation in Genestack with no major drawbacks, we should
migrate over.

This chart is not well maintained, requires updated images, and several
patches to make it work with Genestack.
Most in the helm chart were many years old; py2 was being used.
Needed to alter the storage-init script to correct now deprecated syntax
for ceph auth capabilities for a given user. The original script uses
`profile` in lieu of `allow` in several places.
There's a dependency on a mount related to this secret in several
places. It's use is as a fallback value if another is not provided.
Provide this keyring secret to avoid helm chart deployment failure.
We upgraded to python3 by pulling in a newer image, consequently this
path needed to be updated.
These may need further customization depending on the environment
gnocchi will be run. Aside from the pipeline:main section, this is the
default configuration in the latest version of gnocchi.
@LukeRepko LukeRepko requested a review from cloudnull March 6, 2024 16:39
@LukeRepko LukeRepko marked this pull request as ready for review March 6, 2024 16:45
@LukeRepko LukeRepko changed the title [WIP] Introduce Gnocchi helm chart to Genestack Introduce Gnocchi helm chart to Genestack Mar 6, 2024
Copy link
Contributor

@cloudnull cloudnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cloudnull cloudnull merged commit 2a28c8a into rackerlabs:main Mar 7, 2024
3 checks passed
@LukeRepko LukeRepko deleted the gnocchi branch September 5, 2024 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants