Setup AWS VPN for database access

[patheard]

Description

As a developer,
I want a secure way to access the Notify database instance locally,
so that I can troubleshoot issues Notify data without worrying about unauthorized access to the data.

Acceptance Criteria** (Definition of done)

• Setup AWS VPN for Staging and Production accounts.
• Configure VPN access for each team member that requires access to the database.
• Remove the jumpbox
• ADR for the VPN choice and options.

QA Steps

• Connect to the Staging and Production VPN using your credentials.
• Test that you can access the database while connected to the VPN.
• Test that you can access Redis while connected to the VPN.
• Check that the jumpbox pods are no longer deployed to Kubernetes.

Additional context

We currently have hasura to audit the manually executed SQL statements in production by developers but it's limited in some ways:

• Admin shared account: there is no way to exactly identify who executed what, i.e. we only have the what.
• There are times when we need to bypass hasura as the tool is limited. For example, to download the notifications table in its entirety (or dump the whole database to debug locally) is difficult with Hasura.

Hence this card would increase the overall security on how we access the database.

[jimleroyer]

Hey team! Please add your planning poker estimate with ZenHub @patheard @sastels

[sastels]

we might still want the jumpbox for non-db related things? @jimleroyer what else have we used it for?

[patheard]

I've also used it for Redis access, but that could be setup with the VPN as well.

[jimleroyer]

VPN is installed from within the network, but restrict the access? We need to allow extra usage, unlike the jumpbox? Just wondering. (if we can still access Redis, that would certainly be helpful!)

[patheard]

Yup, the way you set up the VPN is to grant it access to specific security groups, so in the case of Redis, we'd just need to allow the VPN traffic access to the EKS cluster security group:
https://github.com/cds-snc/notification-terraform/blob/main/aws/elasticache/elasticache.tf#L24-L26

[yaelberger-commits]

Please add your planning poker estimate with ZenHub @jzbahrai

[ben851]

I was testing this yesterday when I discovered costing issues in Dev. I will resume testing today to ensure nothing is broken when this moves to staging.

[ben851]

PR was approved, but I didn't merge at EOD. I will merge today.

[ben851]

Note that the VPN adds $200/month in costing to each AWS account.

[ben851]

Merged to staging. Need to test.

[ben851]

Running into some DNS issues in dev. Debugging

[ben851]

Will take a look at the DNS issues today.

I've been working on the internal DNS (unrelated) and have put a PR here
cds-snc/notification-terraform#1133

[ben851]

Did some more internal DNS refining yesterday. Switched nginx to an existing load balancer to better align with our releases.

[jimleroyer]

Ben is planning to roll out the Helm changes into staging this week. The migration only includes the Hasura utility and its dependencies, hence this is a relatively safe and incremental migration.

[jimleroyer]

Ben needs a review on this PR:
cds-snc/notification-manifests#2340

[ben851]

Created two PRs for blazer. Will get reviewed today

[ben851]

Refactored DNS to comply with Google OAuth requirements
Blazer working in dev! Waiting on PR for staging

[ben851]

Blazer working in staging. Will put in a PR to move this to prod.

[ben851]

Workflows added in prod, will test with next release.

[jimleroyer]

There is a PR for jump box but it wasn't approved yet. We will bring documentation, a script in the attic, to bring up an jump box instance on demand if we ever need to.

[ben851]

Added documentation and debug pod script to notification-attic
https://github.com/cds-snc/notification-attic/pull/62

[jimleroyer]

We just need to remove the jump box to get this card in qa.

[ben851]

cds-snc/notification-manifests#2508

[ben851]

Merged, jump box manually removed in staging and prod

[ben851]

@P0NDER0SA to QA

[P0NDER0SA]

QA nearly complete. All steps worked except for the fact that there is still a hasura pod.