Terraform Init Fails when Authority Information is sent in the DNS Response

[b-diggity]

When running a terraform init on a system that is providing the authority information in the DNS response, Terraform isn't querying the correct IP address for registry.terraform.io, causing the init to fail. If the authority information isn't provided in the DNS response, Terraform works without issue.

Terraform Version

$ terraform version
Terraform v1.0.3
on linux_amd64

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.52.0"
    }
  }
}

Expected Behavior

$ tf init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 3.52.0"...
- Installing hashicorp/aws v3.52.0...
- Installed hashicorp/aws v3.52.0 (signed by HashiCorp)


Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Actual Behavior

$ tf init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 3.52.0"...
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/aws: could not connect to registry.terraform.io: Failed to request discovery document: Get
│ "https://registry.terraform.io/.well-known/terraform.json": net/http: TLS handshake timeout
╵

Steps to Reproduce

On WSL (windows subsystem for Linux) Ubuntu 20.04 with the authority information being sent in the DNS repsonse:
terraform init

Additional Context

When running a terraform init, Terraform reaches out to registry.terraform.io. However, a packet capture shows that Terraform is trying to reach out to a NS server returned in the DNS request. For example, based on the output below, terraform is trying to talk to 23.235.32.32. However, this is wrong. It should be connecting to 199.232.6.49. The TLS handshake message is misleading because the server hello packet is never sent from the NS server (nor should it). Here is a nslookup:

$ nslookup registry.terraform.io
Server:         x.x.x.x
Address:        x.x.x.x#53

Non-authoritative answer:
registry.terraform.io   canonical name = dualstack.q2.shared.global.fastly.net.
Name:   dualstack.q2.shared.global.fastly.net
Address: 199.232.6.49
Name:   ns1.fastly.net
Address: 23.235.32.32
Name:   ns2.fastly.net
Address: 104.156.80.32
Name:   ns3.fastly.net
Address: 23.235.36.32
Name:   ns4.fastly.net
Address: 104.156.84.32
Name:   dualstack.q2.shared.global.fastly.net
Address: 2a04:4e42:3e::561
Name:   ns1.fastly.net
Address: 23.235.32.32
Name:   ns2.fastly.net
Address: 104.156.80.32
Name:   ns3.fastly.net
Address: 23.235.36.32
Name:   ns4.fastly.net
Address: 104.156.84.32

And here is a dig:

$ dig registry.terraform.io

; <<>> DiG 9.16.1-Ubuntu <<>> registry.terraform.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11595
;; flags: qr rd ad; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;registry.terraform.io.         IN      A

;; ANSWER SECTION:
registry.terraform.io.  0       IN      CNAME   dualstack.q2.shared.global.fastly.net.
dualstack.q2.shared.global.fastly.net. 0 IN A   199.232.6.49
ns1.fastly.net.         0       IN      A       23.235.32.32
ns2.fastly.net.         0       IN      A       104.156.80.32
ns3.fastly.net.         0       IN      A       23.235.36.32
ns4.fastly.net.         0       IN      A       104.156.84.32

;; Query time: 10 msec
;; SERVER: 172.26.64.1#53(172.26.64.1)
;; WHEN: Wed Aug 04 10:49:17 CDT 2021
;; MSG SIZE  rcvd: 284

This is what a dig looks like when terraform works, notice the answer section doens't include the authority info in the response:

$ dig registry.terraform.io

; <<>> DiG 9.16.1-Ubuntu <<>> registry.terraform.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12411
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;registry.terraform.io.         IN      A

;; ANSWER SECTION:
registry.terraform.io.  0       IN      CNAME   dualstack.q2.shared.global.fastly.net.
dualstack.q2.shared.global.fastly.net. 0 IN A   199.232.6.49

;; Query time: 10 msec
;; SERVER: 172.26.64.1#53(172.26.64.1)
;; WHEN: Wed Aug 04 10:57:50 CDT 2021
;; MSG SIZE  rcvd: 164

[jbardin]

Hi @b-diggity,

Could you provide some more information on the DNS configuration you are using? You mention the added authority section, but neither response contains an ;; AUTHORITY SECTION: in the output. The ns server A records with 0 ttl do look strange to me, but it's been a while since I've had to troubleshoot these things.

We've seen various similar issues in combination with Windows or WSL, but it's usually turned out to be network configuration problems that Terraform cannot avoid on its own. Terraform also is not specifically interpreting the DNS response on it's own, but I do not see any mention of similar issues upstream either.

[b-diggity]

After more searching, I may have jumped the gun. I found microsoft/WSL#5806, which indicates the issue is on Microsoft's side in regards to how they are returning the DNS response.

[jbardin]

Thanks for the link @b-diggity. I'll close this one out for now.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.