Caching DoH proxy for ENS/UD TLD resolution

[lidel]

Until it is possible to do it trustlessly, we need to be able to reliably resolve ENS DNSLinks in web browser via DoH HTTP endpoint.

Problem

Right now, we don't really have a reliable endpoint that has liberal CORS and works in web browser, which makes Helia example/demo work blocked.

Example: both resolver.cloudflare-eth.com/dns-query and eth.link/dns-query are missing CORS headers:

$ curl -s -H "accept: application/dns-json" "https://resolver.cloudflare-eth.com/dns-query?name=_dnslink.vitalik.eth&type=TXT" -i
HTTP/2 200
date: Thu, 01 Feb 2024 16:08:52 GMT
content-type: application/dns-json
content-length: 539
cf-ray: 84eb749d5bd63bbb-WAW
cf-cache-status: HIT
accept-ranges: bytes
age: 282
cache-control: s-maxage=3600
last-modified: Thu, 01 Feb 2024 16:04:10 GMT
server: cloudflare

{"AD":true,"CD":false,"RA":true,"RD":true,"TC":false,"Status":0,"Question":[{"name":"_dnslink.vitalik.eth.","type":16}],"Answer":[{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"dnslink=/ipfs/bafybeictzkdhq2pgttegnhofoerz2txevkfz3tudrgdxuebplxn6sw6n2y\""},{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"contenthash=0xe3010170122053ca867869e69cc8669dc571239d4ee4aa8b9dce8389877a102f5ddbe95bcdd6\""},{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"a=0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045\""}]}

$ curl -s -H "accept: application/dns-json" "https://eth.link/dns-query?name=_dnslink.vitalik.eth&type=TXT" -i
HTTP/2 200
date: Thu, 01 Feb 2024 16:09:42 GMT
content-type: application/dns-json
content-length: 539
cf-ray: 84eb75d7d9f434d4-WAW
cf-cache-status: HIT
accept-ranges: bytes
age: 3
cache-control: s-maxage=3600
last-modified: Thu, 01 Feb 2024 16:09:39 GMT
server: cloudflare

{"AD":true,"CD":false,"RA":true,"RD":true,"TC":false,"Status":0,"Question":[{"name":"_dnslink.vitalik.eth.","type":16}],"Answer":[{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"dnslink=/ipfs/bafybeictzkdhq2pgttegnhofoerz2txevkfz3tudrgdxuebplxn6sw6n2y\""},{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"contenthash=0xe3010170122053ca867869e69cc8669dc571239d4ee4aa8b9dce8389877a102f5ddbe95bcdd6\""},{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"a=0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045\""}]}%

While we work with Cloudflare / ENS community to fix this, we should have a temporary resolved under our control that has correct CORS and caching set up.

Proposed solution

Add Nginx config that sets up Caching Proxy at https://delegated-ipfs.dev/dns-query that proxies requests to resolver.cloudflare-eth.com/dns-query and adds liberal CORS.

It can have the same caching policy as /routing/v1.

If/Once Cloudflare fixes CORS, we can replace proxying with simple HTTP 301.

[lidel]

This is short-term band-aid that will unblock Helia demos and examples @SgtPooki / @achingbrain / @aschmahmann want to do for ETHDenver, and will allow delegated unverified .eth resolution work in browser while ENS community works on truly trustless ENS verification, to witch we will switch if/once it is ready.

@ns4plabs are you able to help with this? Or point me where the config for delegated-ipfs.dev lives so I can PR? Should be pretty simple to add /dns-query route to existing server and add CORS headers manually there (access-control-allow-origin: *)

[lidel]

This is now a lower priority due to eth.limo providing DoH endpoint with correct CORS:

$  curl -X OPTIONS   -H "accept: application/dns-json" "https://dns.eth.limo/dns-query?name=vitalik.eth&type=TXT" -H "Origin: https://example.com" -i 
HTTP/2 204
access-control-allow-credentials: false
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-allow-origin: *
alt-svc: h3=":8443"; ma=2592000
cache-control: max-age=300, must-revalidate
clear-site-data: "cookies"
content-security-policy: frame-ancestors 'self';
cross-origin-resource-policy: cross-origin
date: Mon, 05 Feb 2024 18:49:43 GMT
permissions-policy: interest-cohort=(), battery=()
referrer-policy: strict-origin-when-cross-origin
server: eth.limo
set-cookie: _gat=DELETE_ALL_GA_COOKIES_SET_BY_OTHER_SUBDOMAINS; Path=/; Expires=Thu, 01 Jan 1970 11:59:00 GMT; Domain=.eth.limo
set-cookie: _gid=DELETE_ALL_GA_COOKIES_SET_BY_OTHER_SUBDOMAINS; Path=/; Expires=Thu, 01 Jan 1970 11:59:00 GMT; Domain=.eth.limo
set-cookie: _ga=DELETE_ALL_GA_COOKIES_SET_BY_OTHER_SUBDOMAINS; Path=/; Expires=Thu, 01 Jan 1970 11:59:00 GMT; Domain=.eth.limo
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Access-Control-Request-Headers
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-powered-by: Express
x-true-host: dns.eth.limo
x-xss-protection: 1; mode=block

$ curl -H "accept: application/dns-json" "https://dns.eth.limo/dns-query?name=vitalik.eth&type=TXT" -H "Origin: https://example.com" -i 
HTTP/2 200
access-control-allow-credentials: false
access-control-allow-origin: *
alt-svc: h3=":8443"; ma=2592000
cache-control: max-age=300, must-revalidate
clear-site-data: "cookies"
content-security-policy: frame-ancestors 'self';
content-type: application/x-javascript
cross-origin-resource-policy: cross-origin
date: Mon, 05 Feb 2024 18:50:43 GMT
permissions-policy: interest-cohort=(), battery=()
referrer-policy: strict-origin-when-cross-origin
server: eth.limo
set-cookie: _gat=DELETE_ALL_GA_COOKIES_SET_BY_OTHER_SUBDOMAINS; Path=/; Expires=Thu, 01 Jan 1970 11:59:00 GMT; Domain=.eth.limo
set-cookie: _gid=DELETE_ALL_GA_COOKIES_SET_BY_OTHER_SUBDOMAINS; Path=/; Expires=Thu, 01 Jan 1970 11:59:00 GMT; Domain=.eth.limo
set-cookie: _ga=DELETE_ALL_GA_COOKIES_SET_BY_OTHER_SUBDOMAINS; Path=/; Expires=Thu, 01 Jan 1970 11:59:00 GMT; Domain=.eth.limo
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-powered-by: Express
x-true-host: dns.eth.limo
x-xss-protection: 1; mode=block

{"Status":"0","RD":false,"RA":false,"AD":false,"CD":false,"TC":false,"Question":[{"type":16,"name":"vitalik.eth"}],"Answer":[{"type":16,"name":"vitalik.eth","data":"dnslink=/ipfs/bafybeihbtkwlg5j2vjswkaexoe2agxbppgdacw4oq5lzhqqn6iayqqrbpy/","ttl":300}]}

[SgtPooki]

This is now a lower priority due to eth.limo providing DoH endpoint with correct CORS:

Does this mean we can add eth.limo to verified-fetch DoH endpoints? ipfs/helia-verified-fetch#20

[lidel]

Closing, as Cloudflare one at https://resolver.cloudflare-eth.com/dns-query now has correct CORS headers (access-control-allow-origin: *), using it is preferable as it supports more than just ENS (it supports both .eth and .crypto).

$ url -s -H "accept: application/dns-json" "https://resolver.cloudflare-eth.com/dns-query?name=_dnslink.vitalik.eth&type=TXT" -i -H "Origin: https://example.com" -i -X GET
HTTP/2 200
date: Mon, 26 Feb 2024 14:25:46 GMT
content-type: application/dns-json
content-length: 539
cache-control: s-maxage=3600
access-control-allow-methods: POST, GET
access-control-allow-origin: *
server: cloudflare
cf-ray: 85b8dbf4cd6a35c0-WAW

{"AD":true,"CD":false,"RA":true,"RD":true,"TC":false,"Status":0,"Question":[{"name":"_dnslink.vitalik.eth.","type":16}],"Answer":[{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"dnslink=/ipfs/bafybeiawq7pbt4krnopfmcvymvp2uz4ohibd5p7ugskkybvdmwa2v7evpy\""},{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"contenthash=0xe301017012201687de19f1516b9e560ab8655faa678e3a023ebff43494ac06a36581aafc957e\""},{"name":"_dnslink.vitalik.eth","type":16,"TTL":3600,"data":"\"a=0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045\""}]}

Remaining work needs to happen in ipfs/service-worker-gateway#23 (use the same DoH endpoints as Kubo and Rainbow)

[SgtPooki]

@lidel that resolver does not have the allowed-origin set properly, as i'm still getting this in the browser:

Access to fetch at 'https://resolver.cloudflare-eth.com/dns-query?name=specs.ipfs.tech&type=TXT' from origin 'http://helia-sw-gateway.localhost' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header contains multiple values '*, *', but only one is allowed. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

you can test this in any browser dev console with fetch('https://resolver.cloudflare-eth.com/dns-query?name=specs.ipfs.tech&type=TXT')

[lidel]

Alright this is unfortunate, access-control-allow-origin: * is sent twice.

I'll open a PR to set up the proxy I suggested last month.

[2color]

@lidel Were you in contact with Cloudflare and asked them to add the CORS headers (which are now broken)?

---

(a duplicate but an easier test)

➜  ~ curl -I "https://resolver.cloudflare-eth.com/dns-query?name=specs.ipfs.tech&type=TXT"
HTTP/2 403
date: Tue, 27 Feb 2024 14:24:44 GMT
content-type: text/html; charset=UTF-8
cf-ray: 85c117d40fbe8866-WAW
cf-cache-status: DYNAMIC
access-control-allow-origin: *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *, *
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
vary: Accept-Encoding
x-frame-options: SAMEORIGIN
referrer-policy: same-origin
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-methods: POST, GET
access-control-allow-origin: *
server: cloudflare

[lidel]

@2color I did, looks like there is a bug at CF and header is added to cached value instead of overriding it. Reported upstream.

For us, https://github.com/ipshipyard/waterworks-infra/pull/30 should solve this because it forces liberal CORS after reading response from CF:

proxy_hide_header Access-Control-Allow-Origin;
proxy_hide_header Access-Control-Allow-Methods;
add_header Access-Control-Allow-Origin "*";
add_header Access-Control-Allow-Methods "POST, GET";