[CHORE] Fix Dev

[ben851]

What

Dev is messed up because of the cloudwatch stuff. Need to add this to the delete scripts.

Why

Dev is important! BCP!

Acceptance Criteria

• Dev is running!

Resources

Triggers / Schedule

Describe the schedule or trigger for this chore. When should it be done?

• List any resources associated with this chore (ex. K8s release notes, AWS AMI changes)

[ben851]

Created a PR to include additional deletes and cleanups for the dev environment when cloudwatch is enabled. Have to wait till after this morning's release to merge it.

cds-snc/notification-terraform#1695

[ben851]

Updates merged. Will see how this runs at the end of the week.

[ben851]

@P0NDER0SA to "QA" nothing

[ben851]

Didn't deploy again this morning due to existing resources... I just reran the delete script and will try launching again. I'm not sure what's going on :|

[sastels]

update: Ben knows what's going on! New Relic. Always New Relic.

[ben851]

Bunch of problems with new relic, ses identities not being recreated, and cbs sensor bucket name being stolen. All fixed, moving to QA

[sastels]

infra dev is fine 🎉
will verify delete / create this weekend.

[P0NDER0SA]

Dev workflows (Delete and Recreate) did run, but there were two failures in the recreate for system status and sms to sqs callbacks.

Ben has work being done to migrate this workflow to use helmfile for dev like it does on Staging -- but it's still in progress since the code freeze and hasn't been applied yet.

[P0NDER0SA]

let's checjk the health on this again next Monday

[jimleroyer]

We let the automation runs to see if that works but there was a new change that prevented DEV from working properly.

[ben851]

Dev is up and running

[ben851]

Had a problem with AWS timeouts. I've increased it to 4hrs on dv

[ben851]

Now we have a problem where AWS can't log in :| Debugging and trying a different role
cds-snc/notification-terraform#1735

[ben851]

Dev fixed - moved to the correct role.