dns-over-https DoH

[graingert]

Normally a DNS resolver would be something you plug in at a different level, however for DoH you need an http client already and for some optimizations (eg using cloudflare DoH to only access the Cloudflare web should only require one connection) you need to be using the same machinery for your DoH and regular web requests

As such httpx should implement DoH directly

[sethmlarson]

DoH can be implemented as a middleware via capturing the request URL domain name and rewrite it to an IP address that's been resolved via DoH. I agree we should implement it once and distribute it via a package.

[tomchristie]

DoH can be implemented as a middleware via capturing the request URL domain name and rewrite it to an IP address that's been resolved via DoH.

That's a great call!

[graingert]

It would also be good to support both the Chromium style lookup current DNS-over-UDP provider IP in table approach, and the Firefox everything via Cloudflare approach, and allow a custom DoH provider

[sethmlarson]

Do you have a link to Chromium's approach? I haven't heard of that.

[graingert]

@sethmlarson https://blog.chromium.org/2019/09/experimenting-with-same-provider-dns.html

[graingert]

The table is enumerated here: https://www.chromium.org/developers/dns-over-https

[graingert]

Whatever resolver system is used to resolve the initial DoH server should also be pluggable, eg aiodns etc

[tomchristie]

Also: I’d really like to see this sort of thing tackled as third party middleware, rather than including it in the core package.

If we have third party packages that later become sufficiently proven and well used, then we can take a call on including directly in the package itself.

[toppk]

DoH can be implemented as a middleware via capturing the request URL domain name and rewrite it to an IP address that's been resolved via DoH. I agree we should implement it once and distribute it via a package.

It's a little bit tricky if your are doing TLS, as the http hostname often must stay as the fqdn for certificate validation reasons.

I do think that plugable name resolution could be useful in several use cases:

1. DoH, DoT
2. DNS SRV records
3. Service discovery (eureka, consul, etcd, zookeeper based implementations)
4. custom DNS RR loadbalancing/high availability handling
5. minimize dns traffic generated by agents.

I currently do (4) and (5) in requests and (3) in httpx. For requests I had to monkey patch inside urllib3 (urllib3.util.connection.create_connection, if you're curious), so I'd be happy to see a sufficient middleware api in httpx for these use cases. Note that for cases 2 and 3, it's not just name resolution but also port selection.

Is there a middleware/plugin system inside of httpx currently that shows how such a system might work? I wasn't planning on porting my requests hacks just yet, but this might be fun.

[tomchristie]

Closing this as currently-out-of-scope.
Feasible that we might want to cover this in the core package at some point in the future, but for the foreseeable future let's instead push for making it possible in a third party package via #345