pdns 4.9.1 fails TSIG update with client subnet header

[bjacquin]

• This is not a support question, I have read about opensource and will send support questions to the IRC channel, Github Discussions or the mailing list.
• I have read and understood the 'out in the open' support policy

• Program: pdns
• Issue type: Bug report

Short description

It appears pdns 4.8.4 and 4.9.1 are not able to process TSIG update when request includes client subnet header, providing no error message, but closing connection abruptly. In my current setup, dnsdist 1.9.5 is fronting pdns 4.9.1.

Please note non TSIG request with client subnet header are properly processed.

• dnsdist configuration:

setVerbose(true);

setACL({'0.0.0.0/0', '::/0'})

addLocal('0.0.0.0:53', {
  reusePort=true,
  tcpFastOpenQueueSize=512,
})

addLocal('[::]:53', {
  reusePort=true,
  tcpFastOpenQueueSize=512,
})

newServer({
  pool="authoritative",
  address='[::1]:29053',
  useClientSubnet=true,
  healthCheckMode='up'
})

addAction(
  AllRule(),
  PoolAction("authoritative")
)

• pdns configuration:

loglevel=7
log-dns-details=yes
log-dns-queries=yes

local-address=[::1]:29053

reuseport=yes
tcp-fast-open=1

launch=gmysql
gmysql-socket=/run/mysqld/mysqld.sock
gmysql-user=xxxx
gmysql-password=xxxx
gmysql-dbname=powerdns
gmysql-dnssec=yes

dnsupdate=yes

allow-dnsupdate-from=xxxx:xxxx:xxxx:xxxx::/64
edns-subnet-processing=yes

UDP use case

When dnsdist uses UDP to connect to pdns, dnsdist is reporting the following error:

Jul 20 12:26:18 pdns.internal dnsdist[96450]: Got query for pants-off.xyz|SOA from [xxxx:xxxx:xxxx:xxxx:d847:7e51:510e:61fa]:33405, relayed to [::1]:29053
Jul 20 12:26:19 pdns.internal dnsdist[96450]: Had a downstream timeout from [::1]:29053 () for query for pants-off.xyz|SOA from [xxxx:xxxx:xxxx:xxxx:d847:7e51:510e:61fa]:33405

Although pdns does not report any log or error, even with loglevel=7.

TCP use case

When dnsdist uses UDP to connect to pdns (or forced with tcpOnly=true), dnsdist is reporting the following error:

Jul 20 12:28:21 pdns.internal dnsdist[96491]: Got query for pants-off.xyz|SOA from [xxxx:xxxx:xxxx:xxxx:d847:7e51:510e:61fa]:9306 (DoUDP, 155 bytes), relayed to [::1]:29053
Jul 20 12:28:21 pdns.internal dnsdist[96491]: Got an exception while handling (reading from backend) TCP query from [xxxx:xxxx:xxxx:xxxx:d847:7e51:510e:61fa]:9306: EOF while reading message

Again here, pdns does not report any log or error, even with loglevel=7.

Environment

• Operating system: Gentoo
• Software version: dnsdist 1.9.5, pdns 4.9.1.
• Software source: Operating system repository

Steps to reproduce

See above for configuration detail

Expected behaviour

I would expect:

1. pdns to report why the request are being rejected
2. pdns to process the request

Actual behaviour

nsupdate -y ... times out on the client side.

Other information

• Setting edns-subnet-processing=true in pdns configuration does not affect the behaviour.
• Setting useClientSubnet=false in dnsdist allow requests to be processed successfully.

I have also made an analyziz comparing network level bytes from the client side and what is being received by pdns after dnsdist, there are not notable changes: DNS transaction ID and added client subnet header:

• DNS payload in hex format sent by the client:

0000   03 b0 28 00 00 01 00 00 00 01 00 01 09 70 61 6e
0010   74 73 2d 6f 66 66 03 78 79 7a 00 00 06 00 01 02
0020   78 78 c0 0c 00 01 00 01 00 00 00 0f 00 04 7f 00
0030   00 01 04 61 63 6d 65 00 00 fa 00 ff 00 00 00 00
0040   00 3d 0b 68 6d 61 63 2d 73 68 61 32 35 36 00 00
0050   00 66 9b a3 6a 01 2c 00 20 bb d2 36 32 cf 8d ca
0060   3c 6a c8 26 ea b8 58 5f ab 6f 19 9f 87 9f f9 4d
0070   da 28 92 78 a3 8a 46 c7 04 03 b0 00 00 00 00

• DNS payload in hex format received by the pdns:

0000   08 00 28 00 00 01 00 00 00 01 00 02 09 70 61 6e
0010   74 73 2d 6f 66 66 03 78 79 7a 00 00 06 00 01 02
0020   78 78 c0 0c 00 01 00 01 00 00 00 0f 00 04 7f 00
0030   00 01 04 61 63 6d 65 00 00 fa 00 ff 00 00 00 00
0040   00 3d 0b 68 6d 61 63 2d 73 68 61 32 35 36 00 00
0050   00 66 9b a3 6a 01 2c 00 20 bb d2 36 32 cf 8d ca
0060   3c 6a c8 26 ea b8 58 5f ab 6f 19 9f 87 9f f9 4d
0070   da 28 92 78 a3 8a 46 c7 04 03 b0 00 00 00 00 00
0080   00 29 02 00 00 00 00 00 00 0f 00 08 00 0b 00 02
0090   38 00 xx xx xx xx xx xx xx

• Diff between the two:

@@ -1,9 +1,11 @@
-0000   03 b0 28 00 00 01 00 00 00 01 00 01 09 70 61 6e
+0000   08 00 28 00 00 01 00 00 00 01 00 02 09 70 61 6e
 0010   74 73 2d 6f 66 66 03 78 79 7a 00 00 06 00 01 02
 0020   78 78 c0 0c 00 01 00 01 00 00 00 0f 00 04 7f 00
 0030   00 01 04 61 63 6d 65 00 00 fa 00 ff 00 00 00 00
 0040   00 3d 0b 68 6d 61 63 2d 73 68 61 32 35 36 00 00
 0050   00 66 9b a3 6a 01 2c 00 20 bb d2 36 32 cf 8d ca
 0060   3c 6a c8 26 ea b8 58 5f ab 6f 19 9f 87 9f f9 4d
-0070   da 28 92 78 a3 8a 46 c7 04 03 b0 00 00 00 00
+0070   da 28 92 78 a3 8a 46 c7 04 03 b0 00 00 00 00 00
+0080   00 29 02 00 00 00 00 00 00 0f 00 08 00 0b 00 02
+0090   38 00 xx xx xx xx xx xx xx

[Habbie]

Hey, thanks for the extensive ticket! However, we don't consider this a bug. ECS really modifies the packet, and thus breaks TSIG signatures. I'm converting this to a Discussion and we'll take it from there.

[bjacquin]

I'm actually also using PROXY protocol in fact, however Proxy protocol really indicates me where the TCP connection came from when Client Subnet can be present in request received by dnsdist in case multiple proxy/relay where used on the way.

[bjacquin]

Hey, thanks for the extensive ticket! However, we don't consider this a bug. ECS really modifies the packet, and thus breaks TSIG signatures. I'm converting this to a Discussion and we'll take it from there.

This is fair enough I believe, however not having logs/error is quite distracting :)

Although, I have attempted to specifically route request with TSIG using the following configuration to disable client subnet for request with TSIG, however dnsdist does not route the request to be proper backend and keep sending requests to authoritative pool

newServer({
  pool="authoritative",
  address='[::1]:29053',
  useClientSubnet=true,
  healthCheckMode='up'
})

newServer({
  pool="tsig",
  address='[::1]:29053',
  useClientSubnet=false,
  healthCheckMode='up'
})

addAction(
  QTypeRule(DNSQType.TSIG),
  PoolAction("tsig")
)

addAction(
  AllRule(),
  PoolAction("authoritative")
)

[kpfleming]

The queries you are trying to match probably are DNSOpcode.Update with an RR of DNSQType.A (or DNSQType.AAAA). They are contained in a packet which also contains a DNSQType.TSIG as a second RR. You would need to send both RRs to the auth server, assuming dnsdist can keep them together.

[Habbie]

dnsdist in general considers packets, not RRs, so that should be fine

[bjacquin]

Thanks, this works indeed!

[Habbie]

Although pdns does not report any log or error, even with loglevel=7.

This is weird though, and worth a separate issue ticket!

[bjacquin]

This is actually one of the Expected behaviour from the ticket, do you want me to duplicate current ticket focusing on that matter ?

[Habbie]

yes, I focused on the title more than all of the content, but I also did not want to retitle the ticket and not have this discussion about your real problem here :)

[bjacquin]

I have created #14513 to follow up on logging matter