DATA-1506 State Diffs POC

[jasadams-imtbl]

Description

Add new tables

• storage_diffs
• balance_diffs
• nonce_diffs

Jira

DATA-1506

Type of change

• Bug fix
• New feature
• Documentation
• Configuration

How Has This Been Tested?

Happy path testing only

[github-actions]

Terraform Initialization ⚙️success

Terraform Validation 🤖success

Terraform Plan 📖success

Show Plan

google_storage_bucket_object.blockchain_etl_dataflow: Refreshing state... [id=prod-im-data-imx-resource-dataflow-templates/blockchain-etl-dataflow/metadata.json]
google_dataflow_flex_template_job.zkevm_imtbl_mainnet_job: Refreshing state... [id=2024-03-24_21_07_21-14742475751914167127]
google_dataflow_flex_template_job.zkevm_imtbl_testnet_job: Refreshing state... [id=2024-03-24_21_07_21-15021805784158976880]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # google_dataflow_flex_template_job.zkevm_imtbl_mainnet_job must be replaced
-/+ resource "google_dataflow_flex_template_job" "zkevm_imtbl_mainnet_job" {
      ~ additional_experiments       = [] -> (known after apply)
      ~ id                           = "2024-03-24_21_07_21-14742475751914167127" -> (known after apply)
      ~ job_id                       = "2024-03-24_21_07_21-14742475751914167127" -> (known after apply)
      - labels                       = {} -> null
      - machine_type                 = "n1-standard-1" -> null
      - max_workers                  = 0 -> null
      ~ name                         = "zkevm-imtbl-mainnet-etl-dataflow-20240325-040720" # forces replacement -> (known after apply) # forces replacement
      - num_workers                  = 0 -> null
      ~ parameters                   = {
          ~ "allowedTimestampSkewSeconds" = "5184000" -> "31536000"
            # (11 unchanged elements hidden)
        }
      ~ service_account_email        = "[email protected]" -> (known after apply)
      ~ staging_location             = "gs://dataflow-staging-us-central1-692943280739/staging" -> (known after apply)
      ~ state                        = "JOB_STATE_RUNNING" -> (known after apply)
      ~ temp_location                = "gs://prod-im-data-imx-resource/ethereum-etl/zkevm-imtbl-mainnet-streaming/temp" -> (known after apply)
      ~ type                         = "JOB_TYPE_STREAMING" -> (known after apply)
        # (9 unchanged attributes hidden)
    }

  # google_dataflow_flex_template_job.zkevm_imtbl_testnet_job must be replaced
-/+ resource "google_dataflow_flex_template_job" "zkevm_imtbl_testnet_job" {
      ~ additional_experiments       = [] -> (known after apply)
      ~ id                           = "2024-03-24_21_07_21-15021805784158976880" -> (known after apply)
      ~ job_id                       = "2024-03-24_21_07_21-15021805784158976880" -> (known after apply)
      - labels                       = {} -> null
      - machine_type                 = "n1-standard-1" -> null
      - max_workers                  = 0 -> null
      ~ name                         = "zkevm-imtbl-testnet-etl-dataflow-20240325-040720" # forces replacement -> (known after apply) # forces replacement
      - num_workers                  = 0 -> null
      ~ service_account_email        = "[email protected]" -> (known after apply)
      ~ staging_location             = "gs://dataflow-staging-us-central1-692943280739/staging" -> (known after apply)
      ~ state                        = "JOB_STATE_RUNNING" -> (known after apply)
      ~ temp_location                = "gs://prod-im-data-imx-resource/ethereum-etl/zkevm-imtbl-testnet-streaming/temp" -> (known after apply)
      ~ type                         = "JOB_TYPE_STREAMING" -> (known after apply)
        # (10 unchanged attributes hidden)
    }

Plan: 2 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Pusher: @jasadams-imtbl, Action: pull_request, Workflow: Deploy blockchain etl datafow into GCP

[github-actions]

Terraform Initialization ⚙️success

Terraform Validation 🤖success

Terraform Plan 📖success

Show Plan

google_storage_bucket_object.blockchain_etl_dataflow: Refreshing state... [id=prod-im-data-imx-resource-dataflow-templates/blockchain-etl-dataflow/metadata.json]
google_dataflow_flex_template_job.zkevm_imtbl_testnet_job: Refreshing state... [id=2024-03-24_21_07_21-15021805784158976880]
google_dataflow_flex_template_job.zkevm_imtbl_mainnet_job: Refreshing state... [id=2024-03-24_21_07_21-14742475751914167127]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # google_dataflow_flex_template_job.zkevm_imtbl_mainnet_job must be replaced
-/+ resource "google_dataflow_flex_template_job" "zkevm_imtbl_mainnet_job" {
      ~ additional_experiments       = [] -> (known after apply)
      ~ id                           = "2024-03-24_21_07_21-14742475751914167127" -> (known after apply)
      ~ job_id                       = "2024-03-24_21_07_21-14742475751914167127" -> (known after apply)
      - labels                       = {} -> null
      - machine_type                 = "n1-standard-1" -> null
      - max_workers                  = 0 -> null
      ~ name                         = "zkevm-imtbl-mainnet-etl-dataflow-20240325-040720" # forces replacement -> (known after apply) # forces replacement
      - num_workers                  = 0 -> null
      ~ service_account_email        = "[email protected]" -> (known after apply)
      ~ staging_location             = "gs://dataflow-staging-us-central1-692943280739/staging" -> (known after apply)
      ~ state                        = "JOB_STATE_RUNNING" -> (known after apply)
      ~ temp_location                = "gs://prod-im-data-imx-resource/ethereum-etl/zkevm-imtbl-mainnet-streaming/temp" -> (known after apply)
      ~ type                         = "JOB_TYPE_STREAMING" -> (known after apply)
        # (10 unchanged attributes hidden)
    }

  # google_dataflow_flex_template_job.zkevm_imtbl_testnet_job must be replaced
-/+ resource "google_dataflow_flex_template_job" "zkevm_imtbl_testnet_job" {
      ~ additional_experiments       = [] -> (known after apply)
      ~ id                           = "2024-03-24_21_07_21-15021805784158976880" -> (known after apply)
      ~ job_id                       = "2024-03-24_21_07_21-15021805784158976880" -> (known after apply)
      - labels                       = {} -> null
      - machine_type                 = "n1-standard-1" -> null
      - max_workers                  = 0 -> null
      ~ name                         = "zkevm-imtbl-testnet-etl-dataflow-20240325-040720" # forces replacement -> (known after apply) # forces replacement
      - num_workers                  = 0 -> null
      ~ service_account_email        = "[email protected]" -> (known after apply)
      ~ staging_location             = "gs://dataflow-staging-us-central1-692943280739/staging" -> (known after apply)
      ~ state                        = "JOB_STATE_RUNNING" -> (known after apply)
      ~ temp_location                = "gs://prod-im-data-imx-resource/ethereum-etl/zkevm-imtbl-testnet-streaming/temp" -> (known after apply)
      ~ type                         = "JOB_TYPE_STREAMING" -> (known after apply)
        # (10 unchanged attributes hidden)
    }

Plan: 2 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Pusher: @jasadams-imtbl, Action: pull_request, Workflow: Deploy blockchain etl datafow into GCP