Dat support

[avaer]

Supports loading URLs from the dat:// protocol.

You can test it out with

exokit -x webvr dat://55861001e843ad2a0d0c1e1274c07cbc4fffa0a7cd83648741bc93c55c027acf/index.html

• Ensures that all protocols get a :// after the protocol
• Adds dat-fetch to the protocols list
• Adds dat to the list f protocols to normalize

TODO:

• Figure out how to get the Dat libraries to work without process.nextTick
• Have the regex which detects protocols pull from the list of protocols in fetch.js?
• Configure persistent storage for Dat archives

Closes #1213

Continuation of #1333.

[avaer]

What does this trigger?

-- Would it be compatible with both the Android and the Magic Leap build toolchains?

[RangerMauve]

This just builds up the browserify-ed bundle from the JS source.

Since browserify is just JS it should work on any platform without any hassle.

[avaer]

I see, that makes sense.

Maybe we could put the output somewhere other than src in that case? That directory feels like it should only have checked-in source and we already have a build directory. 🤔

[RangerMauve]

SGTM. Would just build/DatArchive.bundle.js work for that?

[avaer]

Maybe build/js/DatArchive.bundle.js, so it doesn't get too messy with the other binary stuff there.

[avaer]

@RangerMauve How would storage be configured on the different platforms here?

For example, on Android and Magic Leap there are only certain locations that are writable, which is why we have the whole dataPath detection flow.

This PR looks good, but I think we need to ensure compat and across the platforms 👍.

[RangerMauve]

That's a great point regarding storage! I'm just finishing up bringing the DatArchive API to the SDK and I was thinking of bringing that in to replace node-dat-archive (also for perf reasons).

It's got an easy to use storageLocation parameter that we can use. Where should I store the data by default?

[avaer]

Where should I store the data by default?

We generally try to resolve the writable location as dataPath, which is passed through to window contexts.

If this is a thread-local parameter coming into DatArchive, it would need to be set once per thread.

[RangerMauve]

Cool, I'll check that out. I think I'll have time either tomorrow or on Sunday. 😁

[RangerMauve]

K, I updated it to use the SDK and got the bundle compiling into build/js/. It's currently using the default dat storage path so that archive created with the SDK will share storage when they're loaded in Exokit and vise versa.

[RangerMauve]

This might also bring exokit-web support since the SDK should Just Work™ on the web. 🤯

[RangerMauve]

Not sure if this is actually working at the moment though. Gonna test it next time I'm on my laptop. :P

[avaer]

This might also bring exokit-web support since the SDK should Just Work™ on the web.

Cool, does it just use a gateway automatically, or something else?

[RangerMauve]

Yeah! It's connecting to https://discoveryswarm.mauve.moe/

If you want I can help set up an exokit-specific one and configure it to connect there instead. (or whatever other configs we might need)

[avaer]

Yeah, would prefer not to have any third-party network dependencies. We have Exokit-specific domains we can use.

That would only matter for exokit-web though, right?

[RangerMauve]

Yeah, it's all configurable. Also the SDK will try to auto-detect a local discovery-swarm server if it exists (which gets bundled in dat-store for instance)

[avaer]

Where would we configure that in this code?

[RangerMauve]

Basically, we'll need to require dat-sdk/promise instead of dat-sdk/auto and then pass in the options into the factory. https://github.com/datproject/sdk/blob/master/index.js#L30

There's a swarmOpts object that gets passed into discovery-swarm-web and there you can specify a discovery field for the server you want to use.