feature(search): corpus search infrastructure, backfill, and ingest pipeline

[kschelonka]

Infrastructure for corpus search, with ingest and backfill

TODOs:

• e2e manual testing in dev of corpus ingest and backfill process
• Pre-deploy corpus search and sagemaker infrastructure (for below step)
• Update lambdas with environment variables - search endpoint and sagemaker endpoint
• Add indices to prod corpus search cluster

Out of scope:

• Keyword search on corpus search cluster (cut over after backfill is done)
• Semantic search on user-list-search api (next up)

[github-actions]

Plan Result (user-list-search-production)

CI link

⚠️ Resource Deletion will happen ⚠️

This plan contains resource delete operation. Please check the plan result very carefully!

Plan: 3 to add, 8 to change, 2 to destroy.

• Create
  • module.corpus_embeddings.aws_iam_role_policy.snapshot_access_policy_attachment
  • module.corpus_embeddings.aws_opensearch_domain.corpus_search[0]
• Update
  • aws_iam_policy.ecs_task_role_policy
  • aws_iam_role_policy.corpus_events_hydration_lambda_execution_policy
  • aws_iam_role_policy.corpus_events_lambda_execution_policy
  • aws_iam_role_policy.snapshot_access_policy_attachment
  • aws_sqs_queue.corpus_events
  • aws_sqs_queue.corpus_events_hydration
  • aws_sqs_queue_policy.corpus_events_hydration_sqs_policy
  • aws_sqs_queue_policy.corpus_events_sqs_policy
• Delete
  • null_resource.apollo_update-task-definition
• Replace
  • aws_ecs_task_definition.apollo

Change Result (Click me)

  # data.aws_iam_policy_document.corpus_events_hydration_lambda_execution_policy will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "corpus_events_hydration_lambda_execution_policy" {
      + id      = (known after apply)
      + json    = (known after apply)
      + version = "2012-10-17"

      + statement {
          + actions   = [
              + "sqs:DeleteMessage*",
              + "sqs:GetQueueAttributes",
              + "sqs:ReceiveMessage*",
              + "sqs:SendMessage",
              + "sqs:SendMessageBatch",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusParserHydrator",
            ]
        }
      + statement {
          + actions   = [
              + "es:ESHttp*",
            ]
          + effect    = "Allow"
          + resources = [
              + (known after apply),
              + (known after apply),
              + (known after apply),
              + (known after apply),
              + (known after apply),
              + (known after apply),
            ]
        }
      + statement {
          + actions   = [
              + "sagemaker:InvokeEndpoint",
              + "sagemaker:InvokeEndpointAsync",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:sagemaker:us-east-1:996905175585:endpoint/CorpusEmbeddings-Prod-ep-mltvhddp",
            ]
        }
      + statement {
          + actions   = [
              + "ssm:GetParameter*",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod",
              + "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod/*",
            ]
        }
      + statement {
          + actions   = [
              + "logs:CreateLogGroup",
              + "logs:CreateLogStream",
              + "logs:DescribeLogStreams",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:*:*:*",
            ]
        }
      + statement {
          + actions   = [
              + "ec2:AttachNetworkInterface",
              + "ec2:CreateNetworkInterface",
              + "ec2:DeleteNetworkInterface",
              + "ec2:DescribeInstances",
              + "ec2:DescribeNetworkInterfaces",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
    }

  # data.aws_iam_policy_document.corpus_events_hydration_sqs_policy_document will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "corpus_events_hydration_sqs_policy_document" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "sqs:SendMessage",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusParserHydrator",
            ]

          + condition {
              + test     = "ArnEquals"
              + values   = [
                  + "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CollectionEventTopic",
                ]
              + variable = "aws:SourceArn"
            }
          + condition {
              + test     = "ArnEquals"
              + values   = [
                  + "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CorpusEventsTopic",
                ]
              + variable = "aws:SourceArn"
            }

          + principals {
              + identifiers = [
                  + "sns.amazonaws.com",
                ]
              + type        = "Service"
            }
        }
    }

  # data.aws_iam_policy_document.corpus_events_lambda_execution_policy will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "corpus_events_lambda_execution_policy" {
      + id      = (known after apply)
      + json    = (known after apply)
      + version = "2012-10-17"

      + statement {
          + actions   = [
              + "sqs:DeleteMessage*",
              + "sqs:GetQueueAttributes",
              + "sqs:ReceiveMessage*",
              + "sqs:SendMessage",
              + "sqs:SendMessageBatch",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusEvents",
            ]
        }
      + statement {
          + actions   = [
              + "es:ESHttp*",
            ]
          + effect    = "Allow"
          + resources = [
              + (known after apply),
              + (known after apply),
              + (known after apply),
              + (known after apply),
              + (known after apply),
              + (known after apply),
            ]
        }
      + statement {
          + actions   = [
              + "ssm:GetParameter*",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod",
              + "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod/*",
            ]
        }
      + statement {
          + actions   = [
              + "logs:CreateLogGroup",
              + "logs:CreateLogStream",
              + "logs:DescribeLogStreams",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:*:*:*",
            ]
        }
      + statement {
          + actions   = [
              + "ec2:AttachNetworkInterface",
              + "ec2:CreateNetworkInterface",
              + "ec2:DeleteNetworkInterface",
              + "ec2:DescribeInstances",
              + "ec2:DescribeNetworkInterfaces",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
    }

  # data.aws_iam_policy_document.corpus_events_sqs_policy_document will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "corpus_events_sqs_policy_document" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "sqs:SendMessage",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusEvents",
            ]

          + condition {
              + test     = "ArnEquals"
              + values   = [
                  + "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CollectionEventTopic",
                ]
              + variable = "aws:SourceArn"
            }
          + condition {
              + test     = "ArnEquals"
              + values   = [
                  + "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CorpusEventsTopic",
                ]
              + variable = "aws:SourceArn"
            }

          + principals {
              + identifiers = [
                  + "sns.amazonaws.com",
                ]
              + type        = "Service"
            }
        }
    }

  # data.aws_iam_policy_document.snapshot_access_policy will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "snapshot_access_policy" {
      + id      = (known after apply)
      + json    = (known after apply)
      + version = "2012-10-17"

      + statement {
          + actions   = [
              + "s3:DeleteObject",
              + "s3:GetObject",
              + "s3:PutObject",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::pocket-userlistsearch-prod-search-snapshots/*",
            ]
        }
      + statement {
          + actions   = [
              + "s3:ListBucket",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::pocket-userlistsearch-prod-search-snapshots",
            ]
        }
      + statement {
          + actions   = [
              + "iam:PassRole",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:iam::996905175585:role/UserListSearch-Prod-OSManualSnapshotRole",
            ]
        }
      + statement {
          + actions   = [
              + "es:ESHttpPut",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/*",
              + (known after apply),
            ]
        }
    }

  # aws_ecs_task_definition.apollo must be replaced
+/- resource "aws_ecs_task_definition" "apollo" {
      ~ arn                      = "arn:aws:ecs:us-east-1:996905175585:task-definition/UserListSearch-Prod-Apollo:579" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-east-1:996905175585:task-definition/UserListSearch-Prod-Apollo" -> (known after apply)
      ~ container_definitions    = jsonencode(
            [
              - {
                  - command                = [
                      - "--config=/etc/ecs/ecs-xray.yaml",
                    ]
                  - cpu                    = 0
                  - environment            = []
                  - essential              = true
                  - image                  = "amazon/aws-otel-collector"
                  - logConfiguration       = {
                      - logDriver     = "awslogs"
                      - options       = {
                          - awslogs-group         = "/ecs/UserListSearch/Prod/xray"
                          - awslogs-region        = "us-east-1"
                          - awslogs-stream-prefix = "ecs"
                        }
                      - secretOptions = []
                    }
                  - mountPoints            = []
                  - name                   = "aws-otel-collector"
                  - portMappings           = [
                      - {
                          - containerPort = 4138
                          - hostPort      = 4138
                          - protocol      = "tcp"
                        },
                      - {
                          - containerPort = 4137
                          - hostPort      = 4137
                          - protocol      = "tcp"
                        },
                    ]
                  - readonlyRootFilesystem = false
                  - repositoryCredentials  = {
                      - credentialsParameter = "arn:aws:secretsmanager:us-east-1:996905175585:secret:Shared/DockerHub"
                    }
                  - systemControls         = []
                  - volumesFrom            = []
                },
              - {
                  - cpu                    = 0
                  - environment            = [
                      - {
                          - name  = "AWS_APP_PREFIX"
                          - value = "UserListSearch-Prod"
                        },
                      - {
                          - name  = "AWS_SQS_ENDPOINT"
                          - value = "https://sqs.us-east-1.amazonaws.com"
                        },
                      - {
                          - name  = "CORPUS_INDEX_DE"
                          - value = "corpus_de"
                        },
                      - {
                          - name  = "CORPUS_INDEX_EN"
                          - value = "corpus_en"
                        },
                      - {
                          - name  = "CORPUS_INDEX_ES"
                          - value = "corpus_es"
                        },
                      - {
                          - name  = "CORPUS_INDEX_FR"
                          - value = "corpus_fr"
                        },
                      - {
                          - name  = "CORPUS_INDEX_IT"
                          - value = "corpus_it"
                        },
                      - {
                          - name  = "ELASTICSEARCH_DOMAIN"
                          - value = "userlistsearch-prod-v2"
                        },
                      - {
                          - name  = "ELASTICSEARCH_HOST"
                          - value = "vpc-userlistsearch-prod-v2-ee5gxwjmletue32zx64clfmdxu.us-east-1.es.amazonaws.com"
                        },
                      - {
                          - name  = "ELASTICSEARCH_INDEX"
                          - value = "list"
                        },
                      - {
                          - name  = "EVENT_BUS_NAME"
                          - value = "PocketEventBridge-Prod-Shared-Event-Bus"
                        },
                      - {
                          - name  = "NODE_ENV"
                          - value = "production"
                        },
                      - {
                          - name  = "SQS_USER_ITEMS_UPDATE_BACKFILL_URL"
                          - value = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-UserItemsUpdateBackfill"
                        },
                      - {
                          - name  = "SQS_USER_ITEMS_UPDATE_URL"
                          - value = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-UserItemsUpdate"
                        },
                      - {
                          - name  = "SQS_USER_LIST_IMPORT_BACKFILL_URL"
                          - value = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-UserListImportBackfill"
                        },
                    ]
                  - essential              = true
                  - image                  = "996905175585.dkr.ecr.us-east-1.amazonaws.com/userlistsearch-prod:latest"
                  - logConfiguration       = {
                      - logDriver     = "awslogs"
                      - options       = {
                          - awslogs-group         = "/ecs/UserListSearch/Prod/Apollo/node"
                          - awslogs-region        = "us-east-1"
                          - awslogs-stream-prefix = "ecs"
                        }
                      - secretOptions = []
                    }
                  - mountPoints            = []
                  - name                   = "node"
                  - portMappings           = [
                      - {
                          - containerPort = 4000
                          - hostPort      = 4000
                          - protocol      = "tcp"
                        },
                    ]
                  - readonlyRootFilesystem = false
                  - secrets                = [
                      - {
                          - name      = "CONTENT_AURORA_DB"
                          - valueFrom = "arn:aws:secretsmanager:us-east-1:996905175585:secret:UserListSearch/Prod/ParserAuroraDbCredentials"
                        },
                      - {
                          - name      = "PARSER_PRIVILEGED_SERVICE_ID"
                          - valueFrom = "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod/PARSER_PRIVILEGED_SERVICE_ID"
                        },
                      - {
                          - name      = "READITLA_DB"
                          - valueFrom = "arn:aws:secretsmanager:us-east-1:996905175585:secret:UserListSearch/Prod/DatabaseCredentials"
                        },
                      - {
                          - name      = "READITLA_DB_W"
                          - valueFrom = "arn:aws:secretsmanager:us-east-1:996905175585:secret:UserListSearch/Prod/DatabaseCredentials_w"
                        },
                      - {
                          - name      = "SENTRY_DSN"
                          - valueFrom = "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod/SENTRY_DSN"
                        },
                      - {
                          - name      = "UNLEASH_ENDPOINT"
                          - valueFrom = "arn:aws:ssm:us-east-1:996905175585:parameter/Shared/Prod/UNLEASH_ENDPOINT"
                        },
                      - {
                          - name      = "UNLEASH_KEY"
                          - valueFrom = "arn:aws:secretsmanager:us-east-1:996905175585:secret:UserListSearch/Prod/UNLEASH_KEY"
                        },
                    ]
                  - systemControls         = []
                  - volumesFrom            = []
                },
            ] # forces replacement
        ) -> (known after apply) # forces replacement
      ~ id                       = "UserListSearch-Prod-Apollo" -> (known after apply)
      ~ revision                 = 579 -> (known after apply)
        tags                     = {
            "app_code"       = "pocket"
            "component_code" = "pocket-userlistsearch"
            "costCenter"     = "Pocket"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "owner"          = "Pocket"
            "service"        = "UserListSearch"
        }
        # (12 unchanged attributes hidden)
    }

  # aws_iam_policy.ecs_task_role_policy will be updated in-place
  ~ resource "aws_iam_policy" "ecs_task_role_policy" {
        id          = "arn:aws:iam::996905175585:policy/UserListSearch-Prod-TaskRolePolicy"
        name        = "UserListSearch-Prod-TaskRolePolicy"
      ~ policy      = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      - Sid      = ""
                        # (3 unchanged attributes hidden)
                    },
                  + {
                      + Action   = [
                          + "sagemaker:InvokeEndpointAsync",
                          + "sagemaker:InvokeEndpoint",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:sagemaker:us-east-1:996905175585:endpoint/CorpusEmbeddings-Prod-ep-mltvhddp"
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags        = {}
        # (6 unchanged attributes hidden)
    }

  # aws_iam_role_policy.corpus_events_hydration_lambda_execution_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "corpus_events_hydration_lambda_execution_policy" {
        id          = "UserListSearch-Prod-CorpusParserHydratorLambdaExecutionRole:UserListSearch-Prod-CorpusParserHydratorAccessPolicy"
        name        = "UserListSearch-Prod-CorpusParserHydratorAccessPolicy"
      ~ policy      = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "sqs:SendMessageBatch",
                          - "sqs:SendMessage",
                          - "sqs:ReceiveMessage*",
                          - "sqs:GetQueueAttributes",
                          - "sqs:DeleteMessage*",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusParserHydrator"
                    },
                  - {
                      - Action   = "es:ESHttp*"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_it/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_fr/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_es/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_en/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_de/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/_bulk",
                        ]
                    },
                  - {
                      - Action   = "ssm:GetParameter*"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod/*",
                          - "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod",
                        ]
                    },
                  - {
                      - Action   = [
                          - "logs:PutLogEvents",
                          - "logs:DescribeLogStreams",
                          - "logs:CreateLogStream",
                          - "logs:CreateLogGroup",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:*:*:*"
                    },
                  - {
                      - Action   = [
                          - "ec2:DescribeNetworkInterfaces",
                          - "ec2:DescribeInstances",
                          - "ec2:DeleteNetworkInterface",
                          - "ec2:CreateNetworkInterface",
                          - "ec2:AttachNetworkInterface",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # aws_iam_role_policy.corpus_events_lambda_execution_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "corpus_events_lambda_execution_policy" {
        id          = "UserListSearch-Prod-CorpusEventsLambdaExecutionRole:UserListSearch-Prod-CorpusEventsAccessPolicy"
        name        = "UserListSearch-Prod-CorpusEventsAccessPolicy"
      ~ policy      = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "sqs:SendMessageBatch",
                          - "sqs:SendMessage",
                          - "sqs:ReceiveMessage*",
                          - "sqs:GetQueueAttributes",
                          - "sqs:DeleteMessage*",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusEvents"
                    },
                  - {
                      - Action   = "es:ESHttp*"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_it/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_fr/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_es/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_en/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/corpus_de/_bulk",
                          - "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/_bulk",
                        ]
                    },
                  - {
                      - Action   = "ssm:GetParameter*"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod/*",
                          - "arn:aws:ssm:us-east-1:996905175585:parameter/UserListSearch/Prod",
                        ]
                    },
                  - {
                      - Action   = [
                          - "logs:PutLogEvents",
                          - "logs:DescribeLogStreams",
                          - "logs:CreateLogStream",
                          - "logs:CreateLogGroup",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:*:*:*"
                    },
                  - {
                      - Action   = [
                          - "ec2:DescribeNetworkInterfaces",
                          - "ec2:DescribeInstances",
                          - "ec2:DeleteNetworkInterface",
                          - "ec2:CreateNetworkInterface",
                          - "ec2:AttachNetworkInterface",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # aws_iam_role_policy.snapshot_access_policy_attachment will be updated in-place
  ~ resource "aws_iam_role_policy" "snapshot_access_policy_attachment" {
        id          = "UserListSearch-Prod-OSManualSnapshotRole:UserListSearch-Prod-OSManualSnapshotS3AccessPolicy"
        name        = "UserListSearch-Prod-OSManualSnapshotS3AccessPolicy"
      ~ policy      = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "s3:PutObject",
                          - "s3:GetObject",
                          - "s3:DeleteObject",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:s3:::pocket-userlistsearch-prod-search-snapshots/*"
                    },
                  - {
                      - Action   = "s3:ListBucket"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:s3:::pocket-userlistsearch-prod-search-snapshots"
                    },
                  - {
                      - Action   = "iam:PassRole"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:iam::996905175585:role/UserListSearch-Prod-OSManualSnapshotRole"
                    },
                  - {
                      - Action   = "es:ESHttpPut"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:es:us-east-1:996905175585:domain/userlistsearch-prod-v2/*"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # aws_sqs_queue.corpus_events will be updated in-place
  ~ resource "aws_sqs_queue" "corpus_events" {
        id                                = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-CorpusEvents"
        name                              = "UserListSearch-Prod-CorpusEvents"
        tags                              = {
            "app_code"       = "pocket"
            "component_code" = "pocket-userlistsearch"
            "costCenter"     = "Pocket"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "owner"          = "Pocket"
            "service"        = "UserListSearch"
        }
      ~ visibility_timeout_seconds        = 500 -> 90
        # (18 unchanged attributes hidden)
    }

  # aws_sqs_queue.corpus_events_hydration will be updated in-place
  ~ resource "aws_sqs_queue" "corpus_events_hydration" {
      ~ delay_seconds                     = 500 -> 300
        id                                = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-CorpusParserHydrator"
        name                              = "UserListSearch-Prod-CorpusParserHydrator"
        tags                              = {
            "app_code"       = "pocket"
            "component_code" = "pocket-userlistsearch"
            "costCenter"     = "Pocket"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "owner"          = "Pocket"
            "service"        = "UserListSearch"
        }
        # (18 unchanged attributes hidden)
    }

  # aws_sqs_queue_policy.corpus_events_hydration_sqs_policy will be updated in-place
  ~ resource "aws_sqs_queue_policy" "corpus_events_hydration_sqs_policy" {
        id        = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-CorpusParserHydrator"
      ~ policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = "sqs:SendMessage"
                      - Condition = {
                          - ArnEquals = {
                              - "aws:SourceArn" = [
                                  - "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CorpusEventsTopic",
                                  - "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CollectionEventTopic",
                                ]
                            }
                        }
                      - Effect    = "Allow"
                      - Principal = {
                          - Service = "sns.amazonaws.com"
                        }
                      - Resource  = "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusParserHydrator"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # aws_sqs_queue_policy.corpus_events_sqs_policy will be updated in-place
  ~ resource "aws_sqs_queue_policy" "corpus_events_sqs_policy" {
        id        = "https://sqs.us-east-1.amazonaws.com/996905175585/UserListSearch-Prod-CorpusEvents"
      ~ policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = "sqs:SendMessage"
                      - Condition = {
                          - ArnEquals = {
                              - "aws:SourceArn" = [
                                  - "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CorpusEventsTopic",
                                  - "arn:aws:sns:us-east-1:996905175585:PocketEventBridge-Prod-CollectionEventTopic",
                                ]
                            }
                        }
                      - Effect    = "Allow"
                      - Principal = {
                          - Service = "sns.amazonaws.com"
                        }
                      - Resource  = "arn:aws:sqs:us-east-1:996905175585:UserListSearch-Prod-CorpusEvents"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # null_resource.apollo_update-task-definition will be destroyed
  # (because null_resource.apollo_update-task-definition is not in configuration)
  - resource "null_resource" "apollo_update-task-definition" {
      - id       = "3915947528482066678" -> null
      - triggers = {
          - "task_arn" = "arn:aws:ecs:us-east-1:996905175585:task-definition/UserListSearch-Prod-Apollo:579"
        } -> null
    }

  # module.corpus_embeddings.data.aws_iam_policy_document.snapshot_access_policy will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "snapshot_access_policy" {
      + id      = (known after apply)
      + json    = (known after apply)
      + version = "2012-10-17"

      + statement {
          + actions   = [
              + "s3:DeleteObject",
              + "s3:GetObject",
              + "s3:PutObject",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::pocket-corpusembeddings-prod-search-snapshots/*",
            ]
        }
      + statement {
          + actions   = [
              + "s3:ListBucket",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::pocket-corpusembeddings-prod-search-snapshots",
            ]
        }
      + statement {
          + actions   = [
              + "iam:PassRole",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:iam::996905175585:role/CorpusEmbeddings-Prod-OSManualSnapshotRole",
            ]
        }
      + statement {
          + actions   = [
              + "es:ESHttpPut",
            ]
          + effect    = "Allow"
          + resources = [
              + (known after apply),
            ]
        }
    }

  # module.corpus_embeddings.aws_iam_role_policy.snapshot_access_policy_attachment will be created
  + resource "aws_iam_role_policy" "snapshot_access_policy_attachment" {
      + id          = (known after apply)
      + name        = "CorpusEmbeddings-Prod-OSManualSnapshotS3AccessPolicy"
      + name_prefix = (known after apply)
      + policy      = (known after apply)
      + role        = "CorpusEmbeddings-Prod-OSManualSnapshotRole"
    }

  # module.corpus_embeddings.aws_opensearch_domain.corpus_search[0] will be created
  + resource "aws_opensearch_domain" "corpus_search" {
      + access_policies    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "es:*"
                      + Effect    = "Allow"
                      + Principal = "*"
                      + Resource  = "arn:aws:es:us-east-1:996905175585:domain/corpusembeddings-prod/*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + advanced_options   = (known after apply)
      + arn                = (known after apply)
      + dashboard_endpoint = (known after apply)
      + domain_id          = (known after apply)
      + domain_name        = "corpusembeddings-prod"
      + endpoint           = (known after apply)
      + engine_version     = "OpenSearch_2.13"
      + id                 = (known after apply)
      + kibana_endpoint    = (known after apply)
      + tags               = {
          + "app_code"       = "pocket"
          + "component_code" = "pocket-corpusembeddings"
          + "costCenter"     = "Pocket"
          + "env_code"       = "prod"
          + "environment"    = "Prod"
          + "owner"          = "Pocket"
          + "service"        = "CorpusEmbeddings"
        }
      + tags_all           = {
          + "app_code"       = "pocket"
          + "component_code" = "pocket-corpusembeddings"
          + "costCenter"     = "Pocket"
          + "env_code"       = "prod"
          + "environment"    = "Prod"
          + "owner"          = "Pocket"
          + "service"        = "CorpusEmbeddings"
        }

      + auto_tune_options {
          + desired_state       = "ENABLED"
          + rollback_on_disable = "DEFAULT_ROLLBACK"
          + use_off_peak_window = false
        }

      + cluster_config {
          + dedicated_master_count   = 3
          + dedicated_master_enabled = true
          + dedicated_master_type    = "c5.large.search"
          + instance_count           = 4
          + instance_type            = "m5.large.search"
          + zone_awareness_enabled   = true

          + zone_awareness_config {
              + availability_zone_count = 3
            }
        }

      + ebs_options {
          + ebs_enabled = true
          + iops        = (known after apply)
          + throughput  = (known after apply)
          + volume_size = 10
          + volume_type = "gp3"
        }

      + log_publishing_options {
          + cloudwatch_log_group_arn = "arn:aws:logs:us-east-1:996905175585:log-group:/aws/aes/domains/corpusembeddings-prod/error"
          + enabled                  = true
          + log_type                 = "ES_APPLICATION_LOGS"
        }
      + log_publishing_options {
          + cloudwatch_log_group_arn = "arn:aws:logs:us-east-1:996905175585:log-group:/aws/aes/domains/corpusembeddings-prod/slowindex"
          + enabled                  = true
          + log_type                 = "INDEX_SLOW_LOGS"
        }
      + log_publishing_options {
          + cloudwatch_log_group_arn = "arn:aws:logs:us-east-1:996905175585:log-group:/aws/aes/domains/corpusembeddings-prod/slowquery"
          + enabled                  = true
          + log_type                 = "SEARCH_SLOW_LOGS"
        }

      + vpc_options {
          + availability_zones = (known after apply)
          + security_group_ids = [
              + "sg-03bd413b638c57d43",
            ]
          + subnet_ids         = (sensitive value)
          + vpc_id             = (known after apply)
        }
    }

Plan: 3 to add, 8 to change, 2 to destroy.

Changes to Outputs:
  ~ ecs-task-arn           = "arn:aws:ecs:us-east-1:996905175585:task-definition/UserListSearch-Prod-Apollo:579" -> (known after apply)

ℹ️ Objects have changed outside of Terraform

This feature was introduced from Terraform v0.15.4.

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.corpus_embeddings.aws_opensearch_domain.corpus_search[0] has been deleted
  - resource "aws_opensearch_domain" "corpus_search" {
      - domain_name     = "corpusembeddings-prod" -> null
        id              = "arn:aws:es:us-east-1:996905175585:domain/corpusembeddings-prod"
        tags            = {
            "app_code"       = "pocket"
            "component_code" = "pocket-corpusembeddings"
            "costCenter"     = "Pocket"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "owner"          = "Pocket"
            "service"        = "CorpusEmbeddings"
        }
        # (3 unchanged attributes hidden)

        # (7 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the

[kschelonka]

Added the Sentry DSN value to the prod account. I'm looking into the corpus search cluster to see if there is any way I can get around having to destroy it.