DATA-1787 Add proxy fields to contracts table

[jasadams-imtbl]

Description

• Add two new proxy attributes to the contracts table.
• Increase the maximum number of workers for the autoscaling pipeline

Jira

DATA-1787

Type of change

• Bug fix
• New feature
• Documentation
• Configuration

How Has This Been Tested?

Currently running in dev-im-data

[github-actions]

Terraform Initialization ⚙️success

Terraform Validation 🤖success

Terraform Plan 📖success

Show Plan

google_storage_bucket_object.blockchain_etl_dataflow: Refreshing state... [id=prod-im-data-imx-resource-dataflow-templates/blockchain-etl-dataflow/metadata.json]
google_dataflow_flex_template_job.zkevm_imtbl_mainnet_job: Refreshing state... [id=2024-05-27_18_48_44-7874113948184943905]
google_dataflow_flex_template_job.zkevm_imtbl_testnet_job: Refreshing state... [id=2024-05-27_18_48_43-584048505193288470]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # google_dataflow_flex_template_job.zkevm_imtbl_mainnet_job must be replaced
-/+ resource "google_dataflow_flex_template_job" "zkevm_imtbl_mainnet_job" {
      ~ additional_experiments       = [] -> (known after apply)
      ~ id                           = "2024-05-27_18_48_44-7874113948184943905" -> (known after apply)
      ~ job_id                       = "2024-05-27_18_48_44-7874113948184943905" -> (known after apply)
      - labels                       = {} -> null
      - machine_type                 = "n1-standard-1" -> null
      - max_workers                  = 0 -> null
      ~ name                         = "zkevm-imtbl-mainnet-etl-dataflow-20240528-014841" # forces replacement -> (known after apply) # forces replacement
      - num_workers                  = 0 -> null
      ~ parameters                   = {
          ~ "maxNumWorkers"               = "1" -> "4"
            # (11 unchanged elements hidden)
        }
      ~ service_account_email        = "[email protected]" -> (known after apply)
      ~ staging_location             = "gs://dataflow-staging-us-central1-692943280739/staging" -> (known after apply)
      ~ state                        = "JOB_STATE_RUNNING" -> (known after apply)
      ~ temp_location                = "gs://prod-im-data-imx-resource/ethereum-etl/zkevm-imtbl-mainnet-streaming/temp" -> (known after apply)
      ~ type                         = "JOB_TYPE_STREAMING" -> (known after apply)
        # (9 unchanged attributes hidden)
    }

  # google_dataflow_flex_template_job.zkevm_imtbl_testnet_job must be replaced
-/+ resource "google_dataflow_flex_template_job" "zkevm_imtbl_testnet_job" {
      ~ additional_experiments       = [] -> (known after apply)
      ~ id                           = "2024-05-27_18_48_43-584048505193288470" -> (known after apply)
      ~ job_id                       = "2024-05-27_18_48_43-584048505193288470" -> (known after apply)
      - labels                       = {} -> null
      - machine_type                 = "n1-standard-1" -> null
      - max_workers                  = 0 -> null
      ~ name                         = "zkevm-imtbl-testnet-etl-dataflow-20240528-014841" # forces replacement -> (known after apply) # forces replacement
      - num_workers                  = 0 -> null
      ~ parameters                   = {
          ~ "allowedTimestampSkewSeconds" = "15552000" -> "5184000"
            # (11 unchanged elements hidden)
        }
      ~ service_account_email        = "[email protected]" -> (known after apply)
      ~ staging_location             = "gs://dataflow-staging-us-central1-692943280739/staging" -> (known after apply)
      ~ state                        = "JOB_STATE_RUNNING" -> (known after apply)
      ~ temp_location                = "gs://prod-im-data-imx-resource/ethereum-etl/zkevm-imtbl-testnet-streaming/temp" -> (known after apply)
      ~ type                         = "JOB_TYPE_STREAMING" -> (known after apply)
        # (9 unchanged attributes hidden)
    }

Plan: 2 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Pusher: @jasadams-imtbl, Action: pull_request, Workflow: Deploy blockchain etl datafow into GCP

[roopak-immutable]

Hey @jasadams-imtbl - Any specific insights or issues you came across that prompted this increase?

[jasadams-imtbl]

not really @roopak-immutable, I just realised that with the maximum workers set to 1 we effectively have autoscaling disabled. With the transaction growth we were experiencing, it won't be much longer before we will need at least 2 workers. Just getting ahead of the game and creating some headroom. FYI, its still been sitting at 1 worker since I changed it in dev