Loop over provider aliases

[thebigbone]

I am trying to create multiple VMs across different proxmox hosts and for that, I have defined three providers with alias:

provider "proxmox" {                                                                                                                                                                          
  alias     = "host1"                                                                                                                                                                         
  endpoint  = var.proxmox_hosts[0].endpoint                                                                                                                                                   
  username  = var.proxmox_hosts[0].username                                                                                                                                                   
  password  = var.proxmox_hosts[0].password                                                                                                                                                   
  insecure  = true                                                                                                                                                                            
}                                                                                                                                                                                             
                                                                                                                                                                                              
provider "proxmox" {                                                                                                                                                                          
  alias     = "host2"                                                                                                                                                                         
  endpoint  = var.proxmox_hosts[1].endpoint                                                                                                                                                   
  username  = var.proxmox_hosts[1].username                                                                                                                                                   
  password  = var.proxmox_hosts[1].password                                                                                                                                                   
  insecure  = true                                                                                                                                                                            
}                                                                                                                                                                                             
                                                                                                                                                                                              
provider "proxmox" {                                                                                                                                                                          
  alias     = "host3"                                                                                                                                                                        
  endpoint  = var.proxmox_hosts[2].endpoint                                                                                                                                                   
  username  = var.proxmox_hosts[2].username                                                                                                                                                   
  password  = var.proxmox_hosts[2].password                                                                                                                                                   
  insecure  = true                                                                                                                                                                            
} 

Now, I when I try to access the alias for provider in resource block:

resource "proxmox_virtual_environment_vm" "k3s_worker" {                                                                                                                                      
  count = length(var.proxmox_hosts) * var.vm_hostcount  
  provider = proxmox.${var.proxmox_hosts[floor(count.index / var.vm_hostcount)].name}
  # rest of code

With that, I get this error:
╷
│ Error: Invalid character
│
│ on main.tf line 36, in resource "proxmox_virtual_environment_vm" "k3s_worker":
│ 36: provider = proxmox.${var.proxmox_hosts[floor(count.index / var.vm_hostcount)].name}
│
│ This character is not used within the language.
╵
╷
│ Error: Invalid attribute name
│
│ on main.tf line 36, in resource "proxmox_virtual_environment_vm" "k3s_worker":
│ 36: provider = proxmox.${var.proxmox_hosts[floor(count.index / var.vm_hostcount)].name}
│
│ An attribute name is required after a dot.

The $ is not being identified.

[abstractionfactory]

Hi @thebigbone by the looks of it you are trying to dynamically construct a provider alias, which is not supported:

proxmox.${var.proxmox_hosts[floor(count.index / var.vm_hostcount)].name}

If it were supported, this would resolve to:

provider.provider_a

However, what you need to do in order to use provider iteration is using bracketed indexes in conjunction with for_each, not count, as described here.

[thebigbone]

However, what you need to do in order to use provider iteration is using bracketed indexes in conjunction with for_each, not count, as described here.

Thanks for the response. I saw that, how would I then identify different same providers then? because looping over alias isn't supported as you said.

[cam72cam]

You create one provider {} block for proxmox and then use a for_each inside of it. Please read the documentation carefully :)

[threadcrux]

I would think about the problem from a different abstraction. You'll likely know what provider you want ahead of time for the given resource but your data likely doesn't have this. You can modularization the resource code, provide a list of resources into the module that has the assignments but does the calculation beforehand.

The secret sauce behind the below code is arbitrarily picking nodes ahead of time, take a look at the very bottom with the list of cloudinit machines and "node_name".

When the module gets constructed there is a list creator that only takes relevant nodes out of the list and gives them to that instance of the provider that is dynamically created. cloudinit_machines = [for x in var.cloudinit_machines : x if x.node_name == each.key]

This is not a fully working example, I had to scrape a bunch of irrelevant code and slap this example together

Module

/modules/proxmox_vm/virtual_machine.tf

resource "proxmox_virtual_environment_vm" "virtual_machine" {
  for_each = {
    for key, value in var.cloudinit_machines :
    key => value
  }
 ... rest of code
}

/modules/proxmox_vm/encapsulating_vars.tf

variable "cloudinit_machines" {
  type = list(
    object(
      {
        #hypervisor variables
        node_name                 = string

        #cloudinit
        cloud_img_name            = string
        cloudinit_file_name       = string
        cloudinit_user            = optional(string, "user")

        #VM variables
        vm_name                   = optional(string, "ping")
        fqdn                      = optional(string, "ping")
        core_count                = optional(number, 1)
        cpu_type                  = optional(string, "host")
        memory                    = optional(number, 512)
        datastore                 = optional(string, "local-lvm")
        disk_interface            = optional(string, "virtio0")
        disk_size                 = optional(number, 4)
        disk_discard_setting      = optional(string, "on")
        dns_servers               = optional(list(string), ["10.10.10.2"])
        gateway                   = optional(string, "10.10.10.1")
        network_bridge            = optional(string, "vmbr0")
        network_device_model      = optional(string, "virtio")
        ip_address                = optional(string, "dhcp")
        cidr                      = optional(string, "22")

        #machine variables
        bios                      = optional(string, "ovmf")
        machine                   = optional(string, "q35")
        os_type                   = optional(string, "l26")

        #provisioning variables
        ansible_roles             = optional(list(string), [""])
      }
    )
  )
}

Use Module

/infra/main.tf

variable "pves" {
  description = "A list of regions that should have a deployment."
  type        = map(object({
      api_token = string
      ip = string
  }))
  default = { 
    pve1 = {
      api_token = "",
      ip = ""
    },
    pve2 = {
      api_token = "",
      ip = ""
    },
    pve3 = {
      api_token = "",
      ip = ""
    },
  }
}
module "pve_cloudimg" {
  source = "../modules/proxmox_vm"
  node_name = each.key
  providers = {
      proxmox = proxmox.name[each.key]
  }
  cloudinit_files = var.cloudinit_files
  cloud_images = var.cloud_images
  cloudinit_machines = [for x in var.cloudinit_machines : x if x.node_name == each.key]
  for_each = var.pves
}

/infra/vars.tf

variable "cloudinit_machines" {
  default = [
    {
      vm_name   = "eli"
      node_name = "pve1"

      cloud_img_name      = "ubuntu-2404-noble-cloudimg-amd64.img"
      cloudinit_file_name = "ubuntu-noble-2404-init.yml"
      cloudinit_user      = "user"
      disk_size           = 32
      core_count          = 64
      memory              = 1048576
      fqdn                = "eli.dev.ila.ssa.iso"
      gateway             = "10.10.10.1"
      ip_address          = "10.10.10.236"
      network_bridge      = "vmbr1"
      cidr                = "22"

      #provisioning variables
      ansible_roles = ["ansible/server-setup/ubuntu.yml"]
    },
    {
      vm_name   = "caton"
      node_name = "pve2"

      cloud_img_name      = "ubuntu-2404-noble-cloudimg-amd64.img"
      cloudinit_file_name = "ubuntu-noble-2404-init.yml"
      cloudinit_user      = "user"
      disk_size           = 32
      core_count          = 128
      memory              = 1048576
      fqdn                = "caton.dev.ila.ssa.iso"
      gateway             = "10.10.10.1"
      ip_address          = "10.10.10.192"
      cidr                = "22"

      #provisioning variables
      ansible_roles = ["ansible/server-setup/ubuntu.yml"]
    }
  ]
}