Add postgresql #3

hellais · 2024-02-15T08:10:03Z

This adds a new postgresql instance managed via RDS

* Drop generic instance_type variable and move into main.tf

github-actions · 2024-02-15T08:10:45Z

Terraform Run Output 🤖

Format and Style 🖌`success`

Initialization ⚙️`success`

Validation 🤖`success`

Validation Output


$ terraform validate
Success! The configuration is valid.

Plan 📖`success`

Plan: 0 to add, 1 to change, 0 to destroy.

Show Plan


$ terraform plan
Acquiring state lock. This may take a few moments...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
data.aws_ssm_parameter.ecs_optimized_ami: Reading...
aws_acm_certificate.dataapi: Refreshing state... [id=arn:aws:acm:eu-central-1:082866812839:certificate/27b7c5e3-ef70-41e8-a6cf-ac0d58a99613]
aws_cloudwatch_log_group.app: Refreshing state... [id=tf-ecs-group/app-dataapi]
data.aws_ami.debian_ami: Reading...
aws_cloudwatch_log_group.ecs: Refreshing state... [id=tf-ecs-group/ecs-agent]
module.terraform_state_backend.data.aws_region.current: Reading...
aws_iam_role.app_instance: Refreshing state... [id=tf-ecs-ooni-instance-role]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=ooni-production-terraform-state-lock]
aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:cluster/ooni-ecs-cluster]
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
aws_vpc.main: Refreshing state... [id=vpc-08b101405472a0b46]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2050088263]
aws_iam_role.ecs_service: Refreshing state... [id=ooni_ecs_role]
data.aws_availability_zones.available: Reading...
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=ooni-production-terraform-state]
data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 0s [id=/aws/service/ecs/optimized-ami/amazon-linux-2/recommended]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2050088263]
aws_iam_instance_profile.app: Refreshing state... [id=tf-ecs-instprofile]
aws_iam_role_policy.ecs_service: Refreshing state... [id=ooni_ecs_role:ooni_ecs_policy]
aws_route53_record.dataapi_cert_validation["dataapi.prod.ooni.io"]: Refreshing state... [id=Z02418652BOD91LFA5S9X__74a0d0e91f58697b6838bed5a2fb4939.dataapi.prod.ooni.io._CNAME]
data.aws_ami.debian_ami: Read complete after 0s [id=ami-0e626c31414223120]
aws_iam_role_policy.instance: Refreshing state... [id=tf-ecs-ooni-instance-role:TfEcsOONIInstanceRole]
aws_ecs_task_definition.dataapi: Refreshing state... [id=ooni-dataapi-production-td]
data.aws_availability_zones.available: Read complete after 1s [id=eu-central-1]
aws_acm_certificate_validation.dataapi: Refreshing state... [id=2024-02-05 11:11:01.598 +0000 UTC]
aws_security_group.pg_sg: Refreshing state... [id=sg-06d7cf832e6855a37]
aws_security_group.lb_sg: Refreshing state... [id=sg-090ff83a71e7ad235]
aws_security_group.clickhouse_sg: Refreshing state... [id=sg-0c0ffdfe535bededa]
aws_alb_target_group.dataapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:targetgroup/ooni-ecs-dataapi/c16001d24f858f20]
aws_subnet.main[1]: Refreshing state... [id=subnet-02241f60bd951358d]
aws_internet_gateway.gw: Refreshing state... [id=igw-05319878d828e0a53]
aws_subnet.main[0]: Refreshing state... [id=subnet-06d47d18b015109d4]
aws_route_table.r: Refreshing state... [id=rtb-06814accd93aca9ce]
aws_security_group.instance_sg: Refreshing state... [id=sg-0f39450a61732d69b]
aws_db_subnet_group.main: Refreshing state... [id=ooni-main]
aws_alb.main: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:loadbalancer/app/ooni-alb-ecs/4cf8bd00b53f8961]
aws_instance.clickhouse_server_prod_tier1: Refreshing state... [id=i-0aaa354af74a5e988]
aws_route_table_association.a[1]: Refreshing state... [id=rtbassoc-034c4831c1754e284]
aws_route_table_association.a[0]: Refreshing state... [id=rtbassoc-049f8c5e84cb32e55]
aws_launch_template.app: Refreshing state... [id=lt-01b923563af0432b5]
aws_autoscaling_group.app: Refreshing state... [id=ooni-tier1-production-backend-asg20240119184843559900000002]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_route53_record.alb_dns: Refreshing state... [id=Z02418652BOD91LFA5S9X_dataapi.prod.ooni.io_A]
aws_alb_listener.front_end_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/b7ec77ceadac94c8]
aws_alb_listener.front_end: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/1cd9240a34a13129]
aws_db_instance.ooni_pg: Refreshing state... [id=db-LT4NDHTMMWPYB4TLALUJMYTZFU]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-02-02T21:32:23Z]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_route53_record.postgres_dns: Refreshing state... [id=Z035992527R8VEIX2UVO0_postgres.tier0.prod.ooni.nu_CNAME]
aws_ecs_service.dataapi: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production]
aws_eip.clickhouse_ip: Refreshing state... [id=eipalloc-0f0a20bba4a1d858c]
data.aws_ebs_volume.clickhouse_data_volume: Reading...
data.aws_ebs_volume.clickhouse_data_volume: Read complete after 1s [id=vol-0753909d310ed6d8d]
aws_route53_record.clickhouse_dns: Refreshing state... [id=Z035992527R8VEIX2UVO0_clickhouse.tier1.prod.ooni.nu_A]
aws_volume_attachment.clickhouse_data_volume_attachment: Refreshing state... [id=vai-2291175638]
local_file.ansible_inventory: Refreshing state... [id=e7b3f71fb68cc29f026b251ba92a9bfe9a6e4fd6]
null_resource.ansible_update_known_hosts: Refreshing state... [id=4418919173081010018]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_ecs_service.dataapi will be updated in-place
  ~ resource "aws_ecs_service" "dataapi" {
        id                                 = "arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production"
        name                               = "ooni-ecs-dataapi-production"
        tags                               = {
            "Name"       = "ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
      ~ triggers                           = {
          ~ "redeployment" = "2024-02-15T09:26:42Z" -> "2024-02-15T09:43:27Z"
        }
        # (15 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.


Pusher	@hellais
Action	pull_request
Working Directory	./tf/environments/production
Workflow	.github/workflows/check_deploy.yml
Last updated	Thu, 15 Feb 2024 09:43:34 GMT

github-actions · 2024-02-15T08:12:12Z

Ansible Run Output 🤖

Ansible Playbook Recap 🔍


clickhouse.tier1.prod.ooni.nu : ok=8    changed=1    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0

Ansible playbook output 📖`success`

Show Execution


$ ansible-playbook playbook.yml --check --diff -i inventory.ini

PLAY [ClickHouse servers] ******************************************************

TASK [Gathering Facts] *********************************************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : install clickhouse requirements] ****************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Check if ClickHouse GPG keyring exists] *********************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Create a temporary directory for GPG] ***********************
skipping: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Import ClickHouse GPG key] **********************************
skipping: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Remove temporary directory] *********************************
skipping: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Ensure the keyring is readable] *****************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Add ClickHouse repository] **********************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Install ClickHouse server and client] ***********************
Suggested packages:
  clickhouse-common-static-dbg
The following packages will be upgraded:
  clickhouse-client clickhouse-common-static clickhouse-server
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
changed: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Ensure ClickHouse service is started and enabled] ***********
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Configure ClickHouse users from template] *******************
ok: [clickhouse.tier1.prod.ooni.nu]

PLAY RECAP *********************************************************************
clickhouse.tier1.prod.ooni.nu : ok=8    changed=1    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0


Pusher	@hellais
Action	pull_request
Working Directory	./tf/environments/production
Workflow	.github/workflows/check_deploy.yml
Last updated	Thu, 15 Feb 2024 09:46:21 GMT

bassosimone

🐳

I suggested adding one-line comments before blocks as a policy, such that the code self-documents what we're defining and it's easier to grok.

tf/environments/production/main.tf

bassosimone

🐳

hellais added 2 commits February 14, 2024 17:45

Start adding support for postgresql

cf900ad

* Drop generic instance_type variable and move into main.tf

Add DNS record for postgresql

7c193af

hellais added 2 commits February 15, 2024 09:11

Rename security group

9480470

Fix port rule

d8279db

hellais added 10 commits February 15, 2024 09:17

DB Subnet is different than generic VPC

90b3004

Fix subnet reference

5ff9ec7

to meet az requirements we need 2 subnets

c7886ba

Pass password as a variable

3dc4d7b

Change DB Name

804bc23

Fix dns record

c6588b3

Change DB identifier name

20e2835

Drop db name and use default

313024e

Add default db name

418c539

Set apply immediately

eb0b593

hellais requested a review from bassosimone February 15, 2024 09:15

hellais added 6 commits February 15, 2024 10:17

Comment out apply immediately

93aafcf

Make the instance publicly accessible

a96f1a6

Add DNS support to VPC

1d4197f

Update known_hosts

90df6c2

Fix typo in known host updater

70eb614

Remove -x option since it makes shell script output unreadable

83e07ef

bassosimone previously approved these changes Feb 15, 2024

View reviewed changes

tf/environments/production/main.tf Show resolved Hide resolved

Add comment

cd2f711

hellais dismissed bassosimone’s stale review via cd2f711 February 15, 2024 09:42

bassosimone approved these changes Feb 15, 2024

View reviewed changes

hellais added this pull request to the merge queue Feb 20, 2024

Merged via the queue into main with commit c0e236b Feb 20, 2024
2 checks passed

hellais deleted the add-pg branch February 20, 2024 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add postgresql #3

Add postgresql #3

hellais commented Feb 15, 2024

github-actions bot commented Feb 15, 2024 •

edited

Loading

github-actions bot commented Feb 15, 2024 •

edited

Loading

bassosimone left a comment

bassosimone left a comment

Add postgresql #3

Add postgresql #3

Conversation

hellais commented Feb 15, 2024

github-actions bot commented Feb 15, 2024 • edited Loading

Terraform Run Output 🤖

Format and Style 🖌success

Initialization ⚙️success

Validation 🤖success

Plan 📖success

github-actions bot commented Feb 15, 2024 • edited Loading

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

Ansible playbook output 📖success

bassosimone left a comment

Choose a reason for hiding this comment

bassosimone left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 15, 2024 •

edited

Loading

Format and Style 🖌`success`

Initialization ⚙️`success`

Validation 🤖`success`

Plan 📖`success`

github-actions bot commented Feb 15, 2024 •

edited

Loading

Ansible playbook output 📖`success`