docs: craft a gitops focused installation #18

cardoe · 2024-03-05T22:58:18Z

Craft a GitOps focused installation process which users can use to easily stand up a dev environment.

skrobul

Aside from comments above, there is a bunch of things that did not fully converge on my installation:

❯ k -n argocd get applications.argoproj.io                   
NAME                          SYNC STATUS   HEALTH STATUS
app-of-apps                   Synced        Healthy
argo-events                   Unknown       Healthy
argo-workflows                Synced        Degraded
cert-manager                  Synced        Healthy
dexidp                        Synced        Degraded
ironic                        OutOfSync     Degraded
keystone                      OutOfSync     Degraded
mariadb                       Synced        Healthy
mariadb-operator              Synced        Healthy
memcached                     Synced        Healthy
messaging-topology-operator   Synced        Healthy
nautobot                      Synced        Progressing
postgres-db                   Synced        Healthy
postgres-operator             Synced        Healthy
rabbitmq-cluster              Synced        Healthy
rabbitmq-operator             Synced        Healthy

The ironic/keystone is most likely because of missing node labelling.
dex is because of missing azure secret
for argo-events the sso file is missing in /components/12-argo-events/sso, so kustomize fails:

 Message:               Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build <path to cached source>/components/12-argo-events --enable-helm` failed exit status 1: Error: loading KV pairs: file sources: [sso]: evalsymlink failure on '<path to cached source>/components/12-argo-events/sso' : lstat <path to cached source>/components/12-argo-events/sso: no such file or directory

docs/gitops-install.md

skrobul · 2024-04-03T17:42:22Z

scripts/gitops-secrets-gen.sh

+  labels:
+    argocd.argoproj.io/secret-type: repo-creds
+data:
+  sshPrivateKey: $(cat "${UC_DEPLOY_SSH_FILE}" | base64)


Suggested change

sshPrivateKey: $(cat "${UC_DEPLOY_SSH_FILE}" | base64)

sshPrivateKey: $(openssl rsa -in ${UC_DEPLOY_SSH_FILE} | base64 | tr -d '\n')

So openssl rsa -in doesn't work on my SSH key.

Interesting - can you post header of your SSH key? is it text or binary?

-----BEGIN OPENSSH PRIVATE KEY-----

oh okay, mine starts with:

-----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED DEK-Info: AES-128-CBC,5BD

which is PKCS#1, yours seems to be the OpenSSH's standard. There are 4-5 formats out there, so looks like we probably should ssh-keygen for the conversion and decryption instead.

The command to remove the passphrase would be:

ssh-keygen -p -N "" -f /tmp/testkey

Unfortunately it modifies the keyfile in-place, so we need to operate on a copy that we have to remove from the disk after it's used.

cardoe · 2024-04-03T21:28:29Z

So to reply to your top-level comments since I can't reply inline...

ironic/keystone are failing because technically when you use this branch right now you need to manually change targetRevision: HEAD to targetRevision: gitops-install in their generated files. Once this branch is merged that goes away. But ironic will STILL be in a bad place because OpenStack Helm requires you to pass the ethernet device in since the bind mount /dev and /sys. I haven't even started to address that portion in this branch and how to make that configurable. I'm planning on coming to that as a follow on.
I think dex needs to live in this repo without a hard dependency on the azure secret and instead allow us to define a temp user.
I have no idea what the argo-events sso secret is and that was one of the questions I had on the argo-events PR that was merged in. At least my system was broken in the same way on the main branch.

skrobul · 2024-04-04T10:12:02Z

I have no idea what the argo-events sso secret is and that was one of the questions I had on the argo-events PR that was merged in. At least my system was broken in the same way on the main branch.

sso is a configmap generated here, sourced from components/11-argo-workflows/sso. This in turn uses argo-sso secret, which is supposed to be generated by scripts/easy-secrets-gen.sh here but it is not for this deployment.

edit: did a clean test run and found the problem - #28 should fix it.

skrobul · 2024-04-04T10:17:56Z

I think dex needs to live in this repo without a hard dependency on the azure secret and instead allow us to define a temp user.

Agreed. Logged as PUC-224

Each component and operator includes some bits to create the namespace it lives in. For ArgoCD, we don't need a component to create the namespace but instead use it's functionality to create the namespaces.

Create the nautobot pieces in one application.

Allow the understack project to deploy into the dex namespace.

This secret is actually in a different format and managed by jobs and cronjobs that rotate out the value in keystone. The fact that we create this is wrong and the value is wrong. This will result in jobs failing to rotate values.

We don't want to enable OpenStack Helm's helm.sh/hooks because they set them as post-install,post-upgrade which in ArgoCD maps to PostSync. However the deployments and statefulsets in OpenStack Helm depend on the jobs to complete, which they can't if they set to deploy in PostSync

We're not using the ingress controller that OpenStack Helm has for their deployment so we need to set the "use_external_ingress_controller" flag.

Add parameters to component deployments to pull secrets and values from a deployment repo that customizes the configuration of these components.

cardoe · 2024-04-04T22:44:26Z

I think I've addressed all your feedback.

skrobul

just realised I forgot to include about missing argo-sso secret, which causes the secret-argo-sso-* files being generated in the wrong folder and as a result not included in the deploy repo.

also, this does not deploy the ingress anymore - which is a requirement for the argo-workflows and Nautobot to work. Do you plan to take care of this in separate PR?

scripts/easy-secrets-gen.sh

Added the ability to write out the data into a target directory. Added the generation of the OpenStack Helm values file by the script. Allowed skipping of the usage of kubeseal when generating the secrets.

Write up a GitOps focused installation of Understack along with some helper scripts to assist the user with crafting their secrets and their initial layout of their data.

skrobul

Approving with an understanding that:

ingress controller is not being deployed with this. It results in Dex not being reachable for authentication, so the argo-workflows and nautobot cannot be used
this will be tracked as separate story

cardoe · 2024-04-08T14:20:34Z

Approving with an understanding that:

ingress controller is not being deployed with this. It results in Dex not being reachable for authentication, so the argo-workflows and nautobot cannot be used

this will be tracked as separate story

Absolutely agree.

cardoe force-pushed the gitops-install branch from 0d04ae9 to 0382753 Compare March 8, 2024 02:46

cardoe force-pushed the gitops-install branch 20 times, most recently from 556884b to 670c244 Compare March 27, 2024 18:28

cardoe force-pushed the gitops-install branch 2 times, most recently from 5cec310 to 6a8a530 Compare April 1, 2024 14:58

cardoe marked this pull request as ready for review April 1, 2024 14:59

cardoe force-pushed the gitops-install branch 6 times, most recently from e6835b2 to ba389c2 Compare April 1, 2024 17:17

cardoe force-pushed the gitops-install branch from df41517 to 9705dad Compare April 1, 2024 18:30

andrrax approved these changes Apr 1, 2024

View reviewed changes

cardoe force-pushed the gitops-install branch 2 times, most recently from 4cfbfe0 to f613e11 Compare April 1, 2024 22:12

skrobul reviewed Apr 3, 2024

View reviewed changes

cardoe force-pushed the gitops-install branch 2 times, most recently from 41e26a4 to 460ab3f Compare April 3, 2024 22:33

cardoe added 8 commits April 4, 2024 08:27

feat: change namespace creation from a component

2c7f99b

Each component and operator includes some bits to create the namespace it lives in. For ArgoCD, we don't need a component to create the namespace but instead use it's functionality to create the namespaces.

fix: create nautobot pieces together

a3b7767

Create the nautobot pieces in one application.

fix: allow deploys into the dex namespace

2a650d9

Allow the understack project to deploy into the dex namespace.

fix: don't create keystone-credential-keys

f188c7d

This secret is actually in a different format and managed by jobs and cronjobs that rotate out the value in keystone. The fact that we create this is wrong and the value is wrong. This will result in jobs failing to rotate values.

fix: configure OpenStack Helm to use our ingress

af6d1a7

We're not using the ingress controller that OpenStack Helm has for their deployment so we need to set the "use_external_ingress_controller" flag.

feat: parameterize component deployments

af08b3b

Add parameters to component deployments to pull secrets and values from a deployment repo that customizes the configuration of these components.

feat: add parameterizes install of cert-manager

fbf00c0

cardoe force-pushed the gitops-install branch from 460ab3f to 5874465 Compare April 4, 2024 22:40

This comment was marked as outdated.

Sign in to view

skrobul requested changes Apr 8, 2024

View reviewed changes

scripts/easy-secrets-gen.sh Show resolved Hide resolved

cardoe added 2 commits April 8, 2024 08:32

feat: make secrets gen script more flexible

1894e59

Added the ability to write out the data into a target directory. Added the generation of the OpenStack Helm values file by the script. Allowed skipping of the usage of kubeseal when generating the secrets.

feat: craft a gitops focused installation

d54b5c4

Write up a GitOps focused installation of Understack along with some helper scripts to assist the user with crafting their secrets and their initial layout of their data.

cardoe force-pushed the gitops-install branch from 5874465 to d54b5c4 Compare April 8, 2024 13:32

cardoe requested a review from skrobul April 8, 2024 13:33

skrobul approved these changes Apr 8, 2024

View reviewed changes

cardoe merged commit 0b8e57f into main Apr 8, 2024
7 checks passed

cardoe deleted the gitops-install branch April 8, 2024 14:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: craft a gitops focused installation #18

docs: craft a gitops focused installation #18

cardoe commented Mar 5, 2024

skrobul left a comment

skrobul Apr 3, 2024

cardoe Apr 3, 2024

skrobul Apr 4, 2024

cardoe Apr 4, 2024

skrobul Apr 8, 2024

cardoe commented Apr 3, 2024

skrobul commented Apr 4, 2024 •

edited

Loading

skrobul commented Apr 4, 2024

cardoe commented Apr 4, 2024

This comment was marked as outdated.

skrobul left a comment •

edited

Loading

skrobul left a comment

cardoe commented Apr 8, 2024

	sshPrivateKey: $(cat "${UC_DEPLOY_SSH_FILE}" \| base64)
	sshPrivateKey: $(openssl rsa -in ${UC_DEPLOY_SSH_FILE} \| base64 \| tr -d '\n')

docs: craft a gitops focused installation #18

docs: craft a gitops focused installation #18

Conversation

cardoe commented Mar 5, 2024

skrobul left a comment

Choose a reason for hiding this comment

skrobul Apr 3, 2024

Choose a reason for hiding this comment

cardoe Apr 3, 2024

Choose a reason for hiding this comment

skrobul Apr 4, 2024

Choose a reason for hiding this comment

cardoe Apr 4, 2024

Choose a reason for hiding this comment

skrobul Apr 8, 2024

Choose a reason for hiding this comment

cardoe commented Apr 3, 2024

skrobul commented Apr 4, 2024 • edited Loading

skrobul commented Apr 4, 2024

cardoe commented Apr 4, 2024

This comment was marked as outdated.

skrobul left a comment • edited Loading

Choose a reason for hiding this comment

skrobul left a comment

Choose a reason for hiding this comment

cardoe commented Apr 8, 2024

skrobul commented Apr 4, 2024 •

edited

Loading

skrobul left a comment •

edited

Loading