@helia/verified-fetch
+
Helia Verified Fetch
@helia/verified-fetch
helia-verified-fetch
--A fetch-like API for obtaining verified & trustless IPFS content on the web
-
About
- -@helia/verified-fetch provides a fetch-like API for retrieving content from the IPFS network.
All content is retrieved in a trustless manner, and the integrity of all bytes are verified by comparing hashes of the data.
-By default, providers for CIDs are found using delegated routing endpoints.
-Data is retrieved using the following strategies:
--
-
- Directly from providers, using Bitswap over WebSockets and WebRTC if available. -
- Directly from providers exposing a trustless gateway over HTTPS. -
- As a fallback, if no providers reachable from a browser are found, data is retrieved using recursive gateways, e.g.
trustless-gateway.linkwhich can be configured.
-
This is a marked improvement over fetch which offers no such protections and is vulnerable to all sorts of attacks like Content Spoofing, DNS Hijacking, etc.
A verifiedFetch function is exported to get up and running quickly, and a createVerifiedFetch function is also available that allows customizing the underlying Helia node for complete control over how content is retrieved.
Browser-cache-friendly Response objects are returned which should be instantly familiar to web developers.
+About
This monorepo contains the @helia/verified-fetch package and its corresponding interop tests.
Getting started
See the @helia/verified-fetch package for how to get started with the package including usage examples.
Learn more in the announcement blog post and check out the ready-to-run example.
-You may use any supported resource argument to fetch content:
--
-
- CID instance -
- IPFS URL -
- IPNS URL -
Example - Getting started
import { verifiedFetch } from '@helia/verified-fetch'
const resp = await verifiedFetch('ipfs://bafy...')
const json = await resp.json()
-Example - Using a CID instance to fetch JSON
import { verifiedFetch } from '@helia/verified-fetch'
import { CID } from 'multiformats/cid'
const cid = CID.parse('bafyFoo') // some json file
const response = await verifiedFetch(cid)
const json = await response.json()
-Example - Using IPFS protocol to fetch an image
import { verifiedFetch } from '@helia/verified-fetch'
const response = await verifiedFetch('ipfs://bafyFoo') // CID for some image file
const blob = await response.blob()
const image = document.createElement('img')
image.src = URL.createObjectURL(blob)
document.body.appendChild(image)
-Example - Using IPNS protocol to stream a big file
import { verifiedFetch } from '@helia/verified-fetch'
const response = await verifiedFetch('ipns://mydomain.com/path/to/very-long-file.log')
const bigFileStreamReader = await response.body?.getReader()
-Configuration
Custom HTTP gateways and routers
Out of the box @helia/verified-fetch uses a default set of trustless gateways for fetching blocks and HTTP delegated routers for performing routing tasks - looking up peers, resolving/publishing IPNS names, etc.
It's possible to override these by passing gateways and routers keys to the createVerifiedFetch function:
Example - Configuring gateways and routers
import { createVerifiedFetch } from '@helia/verified-fetch'
const fetch = await createVerifiedFetch({
gateways: ['https://trustless-gateway.link'],
routers: ['http://delegated-ipfs.dev']
})
const resp = await fetch('ipfs://bafy...')
const json = await resp.json()
-Usage with customized Helia
For full control of how @helia/verified-fetch fetches content from the distributed web you can pass a preconfigured Helia node to createVerifiedFetch.
The helia module is configured with a libp2p node that is suited for decentralized applications, alternatively @helia/http is available which uses HTTP gateways for all network operations.
-You can see variations of Helia and js-libp2p configuration options at https://ipfs.github.io/helia/interfaces/helia.HeliaInit.html.
-import { trustlessGateway } from '@helia/block-brokers'
import { createHeliaHTTP } from '@helia/http'
import { delegatedHTTPRouting, httpGatewayRouting } from '@helia/routers'
import { createVerifiedFetch } from '@helia/verified-fetch'
const fetch = await createVerifiedFetch(
await createHeliaHTTP({
blockBrokers: [
trustlessGateway()
],
routers: [
delegatedHTTPRouting('http://delegated-ipfs.dev'),
httpGatewayRouting({
gateways: ['https://mygateway.example.net', 'https://trustless-gateway.link']
})
]
})
)
const resp = await fetch('ipfs://bafy...')
const json = await resp.json()
-Custom content-type parsing
By default, if the response can be parsed as JSON, @helia/verified-fetch sets the Content-Type header as application/json, otherwise it sets it as application/octet-stream - this is because the .json(), .text(), .blob(), and .arrayBuffer() methods will usually work as expected without a detailed content type.
If you require an accurate content-type you can provide a contentTypeParser function as an option to createVerifiedFetch to handle parsing the content type.
The function you provide will be called with the first chunk of bytes from the file and should return a string or a promise of a string.
-Example - Customizing content-type parsing
import { createVerifiedFetch } from '@helia/verified-fetch'
import { fileTypeFromBuffer } from '@sgtpooki/file-type'
const fetch = await createVerifiedFetch({
gateways: ['https://trustless-gateway.link'],
routers: ['http://delegated-ipfs.dev']
}, {
contentTypeParser: async (bytes) => {
// call to some magic-byte recognition library like magic-bytes, file-type, or your own custom byte recognition
const result = await fileTypeFromBuffer(bytes)
return result?.mime
}
})
-Custom DNS resolvers
If you don't want to leak DNS queries to the default resolvers, you can provide your own list of DNS resolvers to createVerifiedFetch.
Note that you do not need to provide both a DNS-over-HTTPS and a DNS-over-JSON resolver, and you should prefer dnsJsonOverHttps resolvers for usage in the browser for a smaller bundle size. See https://github.com/ipfs/helia/tree/main/packages/ipns#example---using-dns-json-over-https for more information.
Example - Customizing DNS resolvers
import { createVerifiedFetch } from '@helia/verified-fetch'
import { dnsJsonOverHttps, dnsOverHttps } from '@multiformats/dns/resolvers'
const fetch = await createVerifiedFetch({
gateways: ['https://trustless-gateway.link'],
routers: ['http://delegated-ipfs.dev'],
dnsResolvers: [
dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'),
dnsOverHttps('https://my-dns-resolver.example.com/dns-query')
]
})
-Example - Customizing DNS per-TLD resolvers
DNS resolvers can be configured to only service DNS queries for specific -TLDs:
-import { createVerifiedFetch } from '@helia/verified-fetch'
import { dnsJsonOverHttps, dnsOverHttps } from '@multiformats/dns/resolvers'
const fetch = await createVerifiedFetch({
gateways: ['https://trustless-gateway.link'],
routers: ['http://delegated-ipfs.dev'],
dnsResolvers: {
// this resolver will only be used for `.com` domains (note - this could
// also be an array of resolvers)
'com.': dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'),
// this resolver will be used for everything else (note - this could
// also be an array of resolvers)
'.': dnsOverHttps('https://my-dns-resolver.example.com/dns-query')
}
})
-Custom Hashers
By default, @helia/verified-fetch supports sha256, sha512, and identity hashers.
If you need to use a different hasher, you can provide a custom hasher function as an option to createVerifiedFetch.
Example - Passing a custom hashing function
import { createVerifiedFetch } from '@helia/verified-fetch'
import { blake2b256 } from '@multiformats/blake2/blake2b'
const verifiedFetch = await createVerifiedFetch({
gateways: ['https://ipfs.io'],
hashers: [blake2b256]
})
const resp = await verifiedFetch('ipfs://cid-using-blake2b256')
-IPLD codec handling
IPFS supports several data formats (typically referred to as codecs) which are included in the CID. @helia/verified-fetch attempts to abstract away some of the details for easier consumption.
DAG-PB
DAG-PB is the codec we are most likely to encounter, it is what UnixFS uses under the hood.
-Using the DAG-PB codec as a Blob
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://Qmfoo')
const blob = await res.blob()
console.info(blob) // Blob { size: x, type: 'application/octet-stream' }
-Using the DAG-PB codec as an ArrayBuffer
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://Qmfoo')
const buf = await res.arrayBuffer()
console.info(buf) // ArrayBuffer { [Uint8Contents]: < ... >, byteLength: x }
-Using the DAG-PB codec as a stream
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://Qmfoo')
const reader = res.body?.getReader()
if (reader == null) {
throw new Error('Could not create reader from response body')
}
while (true) {
const next = await reader.read()
if (next?.done === true) {
break
}
if (next?.value != null) {
console.info(next.value) // Uint8Array(x) [ ... ]
}
}
-Content-Type
When fetching DAG-PB data, the content type will be set to application/octet-stream unless a custom content-type parser is configured.
JSON
The JSON codec is a very simple codec, a block parseable with this codec is a JSON string encoded into a Uint8Array.
Using the JSON codec
import * as json from 'multiformats/codecs/json'
const block = new TextEncoder().encode('{ "hello": "world" }')
const obj = json.decode(block)
console.info(obj) // { hello: 'world' }
-Content-Type
When the JSON codec is encountered, the Content-Type header of the response will be set to application/json.
DAG-JSON
DAG-JSON expands on the JSON codec, adding the ability to contain CIDs which act as links to other blocks, and byte arrays.
CIDs and byte arrays are represented using special object structures with a single "/" property.
Using DAG-JSON has two important caveats:
-
-
- Your
JSONstructure cannot contain an object with only a"/"property, as it will be interpreted as a special type.
- - Since
JSONhas no technical limit on number sizes,DAG-JSONalso allows numbers larger thanNumber.MAX_SAFE_INTEGER. JavaScript requires use ofBigInts to represent numbers larger than this, andJSON.parsedoes not support them, so precision will be lost.
-
Otherwise this codec follows the same rules as the JSON codec.
Using the DAG-JSON codec
import * as dagJson from '@ipld/dag-json'
const block = new TextEncoder().encode(`{
"hello": "world",
"cid": {
"/": "baeaaac3imvwgy3zao5xxe3de"
},
"buf": {
"/": {
"bytes": "AAECAwQ"
}
}
}`)
const obj = dagJson.decode(block)
console.info(obj)
// {
// hello: 'world',
// cid: CID(baeaaac3imvwgy3zao5xxe3de),
// buf: Uint8Array(5) [ 0, 1, 2, 3, 4 ]
// }
-Content-Type
When the DAG-JSON codec is encountered in the requested CID, the Content-Type header of the response will be set to application/json.
DAG-JSON data can be parsed from the response by using the .json() function, which will return CIDs/byte arrays as plain { "/": ... } objects:
import { verifiedFetch } from '@helia/verified-fetch'
import * as dagJson from '@ipld/dag-json'
const res = await verifiedFetch('ipfs://bafyDAGJSON')
// either:
const obj = await res.json()
console.info(obj.cid) // { "/": "baeaaac3imvwgy3zao5xxe3de" }
console.info(obj.buf) // { "/": { "bytes": "AAECAwQ" } }
-Alternatively, it can be decoded using the @ipld/dag-json module and the .arrayBuffer() method, in which case you will get CID objects and Uint8Arrays:
import { verifiedFetch } from '@helia/verified-fetch'
import * as dagJson from '@ipld/dag-json'
const res = await verifiedFetch('ipfs://bafyDAGJSON')
// or:
const obj = dagJson.decode<any>(await res.arrayBuffer())
console.info(obj.cid) // CID(baeaaac3imvwgy3zao5xxe3de)
console.info(obj.buf) // Uint8Array(5) [ 0, 1, 2, 3, 4 ]
-DAG-CBOR
DAG-CBOR uses the Concise Binary Object Representation format for serialization instead of JSON.
-This supports more datatypes in a safer way than JSON and is smaller on the wire to boot so is usually preferable to JSON or DAG-JSON.
-Content-Type
Not all data types supported by DAG-CBOR can be successfully turned into JSON and back into the same binary form.
When a decoded block can be round-tripped to JSON, the Content-Type will be set to application/json. In this case the .json() method on the Response object can be used to obtain an object representation of the response.
When it cannot, the Content-Type will be application/octet-stream - in this case the @ipld/dag-json module must be used to deserialize the return value from .arrayBuffer().
Detecting JSON-safe DAG-CBOR
If the Content-Type header of the response is application/json, the .json() method may be used to access the response body in object form, otherwise the .arrayBuffer() method must be used to decode the raw bytes using the @ipld/dag-cbor module.
import { verifiedFetch } from '@helia/verified-fetch'
import * as dagCbor from '@ipld/dag-cbor'
const res = await verifiedFetch('ipfs://bafyDagCborCID')
let obj
if (res.headers.get('Content-Type') === 'application/json') {
// DAG-CBOR data can be safely decoded as JSON
obj = await res.json()
} else {
// response contains non-JSON friendly data types
obj = dagCbor.decode(await res.arrayBuffer())
}
console.info(obj) // ...
-The Accept header
The Accept header can be passed to override certain response processing, or to ensure that the final Content-Type of the response is the one that is expected.
If the final Content-Type does not match the Accept header, or if the content cannot be represented in the format dictated by the Accept header, or you have configured a custom content type parser, and that parser returns a value that isn't in the accept header, a 406: Not Acceptable response will be returned:
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://bafyJPEGImageCID', {
headers: {
accept: 'image/png'
}
})
console.info(res.status) // 406 - the image was a JPEG but we specified PNG as the accept header
-It can also be used to skip processing the data from some formats such as DAG-CBOR if you wish to handle decoding it yourself:
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://bafyDAGCBORCID', {
headers: {
accept: 'application/octet-stream'
}
})
console.info(res.headers.get('accept')) // application/octet-stream
const buf = await res.arrayBuffer() // raw bytes, not processed as JSON
-Redirects
If a requested URL contains a path component, that path component resolves to -a UnixFS directory, but the URL does not have a trailing slash, one will be -added to form a canonical URL for that resource, otherwise the request will -be resolved as normal.
-import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://bafyfoo/path/to/dir')
console.info(res.url) // ipfs://bafyfoo/path/to/dir/
-It's possible to prevent this behaviour and/or handle a redirect manually -through use of the redirect -option.
-Example - Redirect: follow
This is the default value and is what happens if no value is specified.
-import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://bafyfoo/path/to/dir', {
redirect: 'follow'
})
console.info(res.status) // 200
console.info(res.url) // ipfs://bafyfoo/path/to/dir/
console.info(res.redirected) // true
-Example - Redirect: error
This causes a TypeError to be thrown if a URL would cause a redirect.
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://bafyfoo/path/to/dir', {
redirect: 'error'
})
// throw TypeError('Failed to fetch')
-Example - Redirect: manual
Manual redirects allow the user to process the redirect. A 301 -is returned, and the location to redirect to is available as the "location" -response header.
-This differs slightly from HTTP fetch which returns an opaque response as the -browser itself is expected to process the redirect and hide all details from -the user.
-
import { verifiedFetch } from '@helia/verified-fetch'
const res = await verifiedFetch('ipfs://bafyfoo/path/to/dir', {
redirect: 'manual'
})
console.info(res.status) // 301
console.info(res.url) // ipfs://bafyfoo/path/to/dir
console.info(res.redirected) // false
console.info(res.headers.get('location')) // ipfs://bafyfoo/path/to/dir/
-Comparison to fetch
This module attempts to act as similarly to the fetch() API as possible.
The fetch() API takes two parameters:
-
-
- A resource -
- An options object -
Resource argument
This library supports the following methods of fetching web3 content from IPFS:
--
-
- IPFS protocol:
ipfs://<cidv0>&ipfs://<cidv1>
- - IPNS protocol:
ipns://<peerId>&ipns://<publicKey>&ipns://<hostUri_Supporting_DnsLink_TxtRecords>
- - CID instances: An actual CID instance
CID.parse('bafy...')
-
As well as support for pathing & params for items 1 & 2 above according to IPFS - Path Gateway Specification & IPFS - Trustless Gateway Specification. Further refinement of those specifications specifically for web-based scenarios can be found in the Web Pathing Specification IPIP.
-If you pass a CID instance, it assumes you want the content for that specific CID only, and does not support pathing or params for that CID.
-Options argument
This library does not plan to support the exact Fetch API options object, as some of the arguments don't make sense. Instead, it will only support options necessary to meet IPFS specs related to specifying the resultant shape of desired content.
-Some of those header specifications are:
--
-
- https://specs.ipfs.tech/http-gateways/path-gateway/#request-headers -
- https://specs.ipfs.tech/http-gateways/trustless-gateway/#request-headers -
- https://specs.ipfs.tech/http-gateways/subdomain-gateway/#request-headers -
Where possible, options and Helia internals will be automatically configured to the appropriate codec & content type based on the verified-fetch configuration and options argument passed.
Known Fetch API options that will be supported:
--
-
signal- An AbortSignal that a user can use to abort the request.
-redirect- A string that specifies the redirect type. One offollow,error, ormanual. Defaults tofollow. Best effort to adhere to the Fetch API redirect parameter.
-headers- An object of headers to be sent with the request. Best effort to adhere to the Fetch API headers parameter.-
-
accept- A string that specifies the accept header. Relevant values:-
-
vnd.ipld.raw. (default)
-vnd.ipld.car
-vnd.ipfs.ipns-record
-
-
-method- A string that specifies the HTTP method to use for the request. Defaults toGET. Best effort to adhere to the Fetch API method parameter.
-body- An object that specifies the body of the request. Best effort to adhere to the Fetch API body parameter.
-cache- Will basically act asforce-cachefor the request. Best effort to adhere to the Fetch API cache parameter.
-
Non-Fetch API options that will be supported:
--
-
onProgress- Similar to HeliaonProgressoptions, this will be a function that will be called with a progress event. Supported progress events are:-
-
helia:verified-fetch:error- An error occurred during the request.
-helia:verified-fetch:request:start- The request has been sent
-helia:verified-fetch:request:complete- The request has been sent
-helia:verified-fetch:request:error- An error occurred during the request.
-helia:verified-fetch:request:abort- The request was aborted prior to completion.
-helia:verified-fetch:response:start- The initial HTTP Response headers have been set, and response stream is started.
-helia:verified-fetch:response:complete- The response stream has completed.
-helia:verified-fetch:response:error- An error occurred while building the response.
-
-
Some in-flight specs (IPIPs) that will affect the options object this library supports in the future can be seen at https://specs.ipfs.tech/ipips, a subset are:
--
-
- IPIP-0412: Signaling Block Order in CARs on HTTP Gateways -
- IPIP-0402: Partial CAR Support on Trustless Gateways -
- IPIP-0386: Subdomain Gateway Interop with _redirects -
- IPIP-0328: JSON and CBOR Response Formats on HTTP Gateways -
- IPIP-0288: TAR Response Format on HTTP Gateways -
Response types
This library's purpose is to return reasonably representable content from IPFS. In other words, fetching content is intended for leaf-node content -- such as images/videos/audio & other assets, or other IPLD content (with link) -- that can be represented by https://developer.mozilla.org/en-US/docs/Web/API/Response#instance_methods. The content type you receive back will depend upon the CID you request as well as the Accept header value you provide.
All content we retrieve from the IPFS network is obtained via an AsyncIterable, and will be set as the body of the HTTP Response via a ReadableStream or other efficient method that avoids loading the entire response into memory or getting the entire response from the network before returning a response to the user.
If your content doesn't have a mime-type or an IPFS spec, this library will not support it, but you can use the helia library directly for those use cases. See Unsupported response types for more information.
Handling response types
For handling responses we want to follow conventions/abstractions from Fetch API where possible:
--
-
- For JSON, assuming you abstract any differences between dag-json/dag-cbor/json/and json-file-on-unixfs, you would call
.json()to get a JSON object.
- - For images (or other web-relevant asset) you want to add to the DOM, use
.blob()or.arrayBuffer()to get the raw bytes.
- - For plain text in utf-8, you would call
.text()
- - For streaming response data, use something like
response.body.getReader()to get aReadableStream.
-
Unsupported response types
-
-
- Returning IPLD nodes or DAGs as JS objects is not supported, as there is no currently well-defined structure for representing this data in an HTTP Response. Instead, users should request
aplication/vnd.ipld.caror use thehelialibrary directly for this use case.
- - Others? Open an issue or PR! -
Response headers
This library will set the HTTP Response headers to the appropriate values for the content type according to the appropriate IPFS Specifications.
-Some known header specifications:
--
-
- https://specs.ipfs.tech/http-gateways/path-gateway/#response-headers -
- https://specs.ipfs.tech/http-gateways/trustless-gateway/#response-headers -
- https://specs.ipfs.tech/http-gateways/subdomain-gateway/#response-headers +
@helia/verified-fetchA fetch-like API for obtaining verified & trustless IPFS content on the web
+@helia/verified-fetch-interopInterop tests for @helia/verified-fetchTypeError- If the resource argument is not a string, CID, or CID string.
-TypeError- If the options argument is passed and not an object.
-TypeError- If the options argument is passed and is malformed.
-AbortError- If the content request is aborted due to user aborting provided AbortSignal. Note that this is aAbortErrorfrom@libp2p/interfaceand not the standardAbortErrorfrom the Fetch API.
-
Packages
-
+
Possible Scenarios that could cause confusion
Attempting to fetch the CID for content that does not make sense
If you request bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze, which points to the root of the en.wikipedia.org mirror, a response object does not make sense.
Errors
Known Errors that can be thrown:
--
-
Install
$ npm i @helia/verified-fetch
-Browser <script> tag
Loading this module through a script tag will make it's exports available as HeliaVerifiedFetch in the global namespace.
<script src="https://unpkg.com/@helia/verified-fetch/dist/index.min.js"></script>
-API Docs
-
-
- https://ipfs.github.io/helia-verified-fetch/modules.html +
- https://ipfs.github.io/helia-verified-fetch/modules/_helia_verified_fetch.html
License
Licensed under either of
-
@@ -255,4 +28,4 @@
Please be aware that all interactions related to this repo are subject to the IPFS Code of Conduct.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Create and return a Helia node
+