Recursive retrieval of a dag

[2color]

Given a CID of a UnixFS director, e.g. bafybeiakerpabz5sbcl7e3sokxygunvtcj7lhyjl4me62uf2dxqzza5nha, how would I recursively retrieve the whole DAG with Helia?

[achingbrain]

Could you specify a bit more detail about what you're trying to do?

Is there some sort of output format you want or do you just want to have the blocks locally?

[2color]

This came up in the context of the Service Worker Gateway.

If you want to make a site available offline, it's useful the have all the blocks in the blockstore.

[achingbrain]

You could pin it, then the whole DAG will be recursively fetched and the blocks will be in the blockstore.

[2color]

Thanks.

Here's the code for anyone interested in what this looks like programatically

helia/packages/utils/src/pins.ts

Lines 71 to 139 in aecac3d

	async * add (cid: CID<unknown, number, number, Version>, options: AddOptions = {}): AsyncGenerator<CID, void, undefined> {
	const pinKey = toDSKey(cid)
	if (await this.datastore.has(pinKey)) {
	throw new Error('Already pinned')
	}
	const depth = Math.round(options.depth ?? Infinity)
	if (depth < 0) {
	throw new Error('Depth must be greater than or equal to 0')
	}
	// use a queue to walk the DAG instead of recursion so we can traverse very large DAGs
	const queue = new Queue<AsyncGenerator<CID>>({
	concurrency: DAG_WALK_QUEUE_CONCURRENCY
	})
	for await (const childCid of this.#walkDag(cid, queue, {
	...options,
	depth
	})) {
	await this.#updatePinnedBlock(childCid, (pinnedBlock: DatastorePinnedBlock) => {
	// do not update pinned block if this block is already pinned by this CID
	if (pinnedBlock.pinnedBy.find(c => uint8ArrayEquals(c, cid.bytes)) != null) {
	return false
	}
	pinnedBlock.pinCount++
	pinnedBlock.pinnedBy.push(cid.bytes)
	return true
	}, options)
	yield childCid
	}
	const pin: DatastorePin = {
	depth,
	metadata: options.metadata ?? {}
	}
	await this.datastore.put(pinKey, cborg.encode(pin), options)
	}
	/**
	* Walk a DAG in an iterable fashion
	*/
	async * #walkDag (cid: CID, queue: Queue<AsyncGenerator<CID>>, options: WalkDagOptions): AsyncGenerator<CID> {
	if (options.depth === -1) {
	return
	}
	const codec = await this.getCodec(cid.code)
	const bytes = await this.blockstore.get(cid, options)
	const block = createUnsafe({ bytes, cid, codec })
	yield cid
	// walk dag, ensure all blocks are present
	for await (const [,cid] of block.links()) {
	yield * await queue.add(async () => {
	return this.#walkDag(cid, queue, {
	...options,
	depth: options.depth - 1
	})
	})
	}
	}

[achingbrain]

An nice feature might be using the pin name or pin metadata to let the user select previous versions of the site? Kind of like the Wayback Machine.

[2color]

Yeah, that would be nice. It's a bit tricky adding a UI to the SW Gateway without interfering with the site backed by the CID you are viewing.