terraform-google-pubsub

This module makes it easy to create Google Cloud Pub/Sub topic and subscriptions associated with the topic.

Compatibility

This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue. If you haven't upgraded and need a Terraform 0.12.x-compatible version of this module, the last released version intended for Terraform 0.12.x is v1.9.0.

Usage

This is a simple usage of the module. Please see also a simple setup provided in the example directory.

module "pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 7.0"

  topic      = "tf-topic"
  project_id = "my-pubsub-project"
  push_subscriptions = [
    {
      name                       = "push"                                               // required
      ack_deadline_seconds       = 20                                                   // optional
      push_endpoint              = "https://example.com"                                // required
      x-goog-version             = "v1beta1"                                            // optional
      oidc_service_account_email = "[email protected]"                                     // optional
      audience                   = "example"                                            // optional
      expiration_policy          = "1209600s"                                           // optional
      dead_letter_topic          = "projects/my-pubsub-project/topics/example-dl-topic" // optional
      max_delivery_attempts      = 5                                                    // optional
      maximum_backoff            = "600s"                                               // optional
      minimum_backoff            = "300s"                                               // optional
      filter                     = "attributes.domain = \"com\""                        // optional
      enable_message_ordering    = true                                                 // optional
    }
  ]
  pull_subscriptions = [
    {
      name                         = "pull"                                               // required
      ack_deadline_seconds         = 20                                                   // optional
      dead_letter_topic            = "projects/my-pubsub-project/topics/example-dl-topic" // optional
      max_delivery_attempts        = 5                                                    // optional
      maximum_backoff              = "600s"                                               // optional
      minimum_backoff              = "300s"                                               // optional
      filter                       = "attributes.domain = \"com\""                        // optional
      enable_message_ordering      = true                                                 // optional
      service_account              = "[email protected]"          // optional
      enable_exactly_once_delivery = true                                                 // optional
    }
  ]
  bigquery_subscriptions = [
    {
      name                = "bigquery"              // required
      table               = "project.dataset.table" // required
      use_topic_schema    = true                    // optional
      use_table_schema    = false                   // optional
      write_metadata      = false                   // optional
      drop_unknown_fields = false                   // optional
    }
  ]
  cloud_storage_subscriptions = [
    {
      name                     = "cloud-storage"        // required
      bucket                   = "example-bucket"       // required
      filename_prefix          = "log_events_"          // optional
      filename_suffix          = ".avro"                // optional
      filename_datetime_format = "YYYY-MM-DD/hh_mm_ssZ" // optional
      max_duration             = "60s"                  // optional
      max_bytes                = "10000000"             // optional
      max_messages             = "10000"                // optional
      output_format            = "avro"                 // optional
      write_metadata           = false                  // optional
      use_topic_schema         = false                  // optional
    }
  ]
}

Inputs

Name	Description	Type	Default	Required
bigquery_subscriptions	The list of the Bigquery push subscriptions.	list(map(string))	[]	no
cloud_storage_subscriptions	The list of the Cloud Storage push subscriptions.	list(map(string))	[]	no
create_subscriptions	Specify true if you want to create subscriptions.	bool	true	no
create_topic	Specify true if you want to create a topic.	bool	true	no
grant_bigquery_project_roles	Specify true if you want to add bigquery.metadataViewer and bigquery.dataEditor roles to the default Pub/Sub SA.	bool	true	no
grant_token_creator	Specify true if you want to add token creator role to the default Pub/Sub SA.	bool	true	no
message_storage_policy	A map of storage policies. Default - inherit from organization's Resource Location Restriction policy.	map(any)	{}	no
project_id	The project ID to manage the Pub/Sub resources.	string	n/a	yes
pull_subscriptions	The list of the pull subscriptions.	list(map(string))	[]	no
push_subscriptions	The list of the push subscriptions.	list(map(string))	[]	no
schema	Schema for the topic.	object({ name = string type = string definition = string encoding = string })	null	no
subscription_labels	A map of labels to assign to every Pub/Sub subscription.	map(string)	{}	no
topic	The Pub/Sub topic name.	string	n/a	yes
topic_kms_key_name	The resource name of the Cloud KMS CryptoKey to be used to protect access to messages published on this topic.	string	null	no
topic_labels	A map of labels to assign to the Pub/Sub topic.	map(string)	{}	no
topic_message_retention_duration	The minimum duration in seconds to retain a message after it is published to the topic.	string	null	no

Outputs

Name	Description
id	The ID of the Pub/Sub topic
subscription_names	The name list of Pub/Sub subscriptions
subscription_paths	The path list of Pub/Sub subscriptions
topic	The name of the Pub/Sub topic
topic_labels	Labels assigned to the Pub/Sub topic
uri	The URI of the Pub/Sub topic

Requirements

Installation Dependencies

• Terraform >= 0.13.0
• terraform-provider-google plugin >= v2.13

Configure a Service Account

In order to execute this module you must have a Service Account with the following:

Roles

• roles/pubsub.admin

Enable APIs

In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:

• Cloud Pub/Sub API

Service Account Credentials

You can pass the service account credentials into this module by setting the following environment variables:

• GOOGLE_CREDENTIALS
• GOOGLE_CLOUD_KEYFILE_JSON
• GCLOUD_KEYFILE_JSON

See more details.