Update Pepr finalizer code to reflect deletion status and failure

[mjnagel]

With the upstream Pepr changes in defenseunicorns/pepr#1321 we are now able to handle our finalizer a bit better. The upstream changes allow us to return false from the finalizer function to skip removal of the finalizer from the resource. In particular this will allow us to:

• We can perform a state update to the Package CR to indicate that the CR is in a Deleting state
• Catch errors on deletion (i.e. keycloak client cleanup) and retry or put the CR into a Failed Deletion state?
• Handle issues like Deleting Package before status is Ready does not properly clean up clients #963 by waiting to run the finalizer until the Package CR is in a ready or Failed state?

Definition of done:

• When a package is deleted, it is updated with a status of Deleting while the finalizer runs
• When an error/failure is hit during finalizing, the Package is marked as DeletionFailed with an event noting the error and manual cleanup steps for the user (ex: delete client from keycloak manually)

[eddiezane]

Ran into a snag I think is related to this. I was following the dev setup in the readme.

Use npx pepr deploy per the readme will use the latest version of pepr which appears to be incompatible.

$ npx pepr deploy
? This will remove and redeploy the module. Continue? › (y/N)(node:606301) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
✔ This will remove and redeploy the module. Continue? … yes
File /home/eddiezane/Codez/defenseunicorns/uds-core/src/pepr/docs-gen/main.ts is not formatted correctly
File /home/eddiezane/Codez/defenseunicorns/uds-core/src/pepr/operator/controllers/keycloak/authservice/config.ts is not formatted correctly
File /home/eddiezane/Codez/defenseunicorns/uds-core/src/pepr/operator/controllers/network/generators/kubeAPI.ts is not formatted correctly
File /home/eddiezane/Codez/defenseunicorns/uds-core/src/pepr/operator/controllers/utils.ts is not formatted correctly
Formatting errors were found. The build will continue, but you may want to run `npx pepr format` to address any issues.
Command failed: /home/eddiezane/Codez/defenseunicorns/uds-core/node_modules/.bin/tsc --project /home/eddiezane/Codez/defenseunicorns/uds-core/tsconfig.json --outdir dist
src/pepr/operator/index.ts(76,4): error TS2339: Property 'Finalize' does not exist on type 'void'.
src/pepr/policies/index.ts(31,7): error TS2353: Object literal may only specify known properties, and 'resyncFailureMax' does not exist in type 'WatchCfg'.

It worked when I used the intended version.

npm ci
node_modules/.bin/pepr deploy

[mjnagel]

@eddiezane I'm not sure that's related to this issue (the Finalizer changes mentioned here were merged in Pepr several months back and were backwards compatible) but I suspect you're correct on the cause of your problem being that npx pepr uses the latest version. Curiously I'm not able to hit that same issue, although my deploy has this error at a different point:

Error deploying module: {
  data: {
    kind: 'Status',
    apiVersion: 'v1',
    metadata: {},
    status: 'Failure',
    message: 'Apply failed with 3 conflicts: conflicts with "zarf" using admissionregistration.k8s.io/v1:\n' +
      '- .webhooks[name="pepr-uds-core.pepr.dev"].rules\n' +
      '- .webhooks[name="pepr-uds-core.pepr.dev"].clientConfig.caBundle\n' +
      '- .webhooks[name="pepr-uds-core.pepr.dev"].clientConfig.service.path',
    reason: 'Conflict',
    details: { causes: [Array] },
    code: 409
  },
  ok: false,
  status: 409,
  statusText: 'Conflict'
}

We could maybe move this convo into a separate issue though as I'm not sure it fits here 😅

[mjnagel]

Noting that this may also require changes to the new operator diagrams to ensure that the finalizer flow is properly represented.