Joystream Javascript SDK vision

[kdembler]

Intro

I think it's quite clear some sort of Joystream SDK is badly needed. As it is today, to write even simple application on top of Joystream, you need to have solid understanding of a lot of the system characteristic, ways to interact with it, etc. Even integrating with our storage system is troublesome and would be a challenge for a newcomer. If we want people to experiment with Joystream and build new applications, some sort of SDK is a must. Having a set of solid, tested and reusable libraries should make not only development of new applications much easier and welcoming, it can also help by improving our current applications. For example, the storage integration layer in Atlas could be improved in a lot of ways and storage SDK could be a way to do that.

Proposal

I propose we develop a set of libraries to interact with different pieces of the system. Ones that immediately come to mind, in order of execution, as I imagine:

1. Utilities - set of basic utilities that will be used as building blocks for other libraries. This could include things like, block height<>date conversion, $JOY utilities, utilities for metaprotocol and more.
2. Storage - full integration with the Joystream storage system, allowing to programatically upload an asset, download, etc. Possibly also containing utilities for leads to manage the system.
3. Extrinsics - actual methods for interacting with the Joystream blockchain. It's useful to have TypeScript wrappers over raw RPC methods that can create necessary types/prepare the data. This will probably be the hardest to get right and it's to be seen whether that's a single package.

I propose we use the Polkadot.js naming scheme and use npm package names like @joystream/utils, @joystream/storage etc. I think all of those should also be kept in a single repository (ideally under Joystream GitHub org) as a monorepo, with each package having its own proper versioning. @bedeho will it be possible to give AWG access to Github repo and npm organization? If that's an issue, as an alternative, those could be published under @joystreamjs org.

For storage and extrinsics libraries, proper RFC with proposed API will be created in advance so the community can discuss how the libraries should shape up.

I think our ultimate goal should be to integrate those libraries into Pioneer and Atlas, once mature enough. When discussing the libraries, we should keep all the possible Pioneer/Atlas usecases in mind, to make sure our SDKs are robust enough to support our apps.

[bedeho]

1. Thank you for proposing this, it was about time we started moving on this.
2. I buy the premise very strongly.
3. All the specifics you suggest is fine, just let me know what name you want for this repo. I'm unsure about how we manage write permissions on this, my preference is that only you can do this for now, in particular if we are going to start to use this codebase in original monrepo applications that lots of people will continously be running, trusting us as maintainers. Any ideas here?
4. What do you think about including the now increasing portfolio of metadata formats we are relying on? This is starting to now get quite complex,
   with apps-as-first-class (with Orion signing), argus access policies (Standardise Argus access policy signals in metadata #4332) and authentication (Argus: content delivery authentication #3152), and it will only get more complex as people start inventing new versions of the same messages.
5. Where do you see QN fitting in here? I think a lot of things people want to do in this API requires QN or Orionv2 to really work.
6. Do you see this as an SDK for making any Joystream product, so even Colossus and Argus themselves, or are you specifically thinking narrowly on Consumer apps like Atlas? Will the SDK perhaps have distinct parts with a focus on different use cases?
7. I suspect that a great source of inspiration for what is currently causing complexity for app-developers is just to ask all current builders where they currently see complexity in their apps, and then you as the leader of the SDK can try to synthesise all of that into a vision for what should be focused on, and what people just should live with. So that means
   • Pioneer: Theo, Radek
   • Atlas: Klaudiusz, Bartosz
   • CLI: Zeeshan, Leszek
   • Argus/Colossus: Leszek, Mokhtar, Zeeshan
   • YT-synch: Zeeshan
   • Orionv2/QN: Leszek
   • Integration tests: Mokhtar, Ignazio
   • L1.media/JSS: l1.dev

What do you think?
9. I'd encourage you to look at the Lens protocol backend. They have really focused on making the dev. experience for Lens protocol very smooth, with gas relays, no need for signing etc. We have similar plans based on Orionv2.
10. How can we get off to a great start in terms of documentation? It is always really painful to try to go back end do that later.

[kdembler]

All the specifics you suggest is fine, just let me know what name you want for this repo. I'm unsure about how we manage write permissions on this, my preference is that only you can do this for now, in particular if we are going to start to use this codebase in original monrepo applications that lots of people will continously be running, trusting us as maintainers. Any ideas here?

I actually still have permissions to create a repo, so I can take care of that. I think for now only I will hold write access, but I would say that GitHub access is not that important, it's mostly the NPM access. Even if somebody would push malicious code, it only gets to consumer apps if:

1. Someone with NPM access does a release
2. App maintainer uses the new NPM version.

Obviously we need to be careful here. Will it be possible to grant me write access to specific NPM packages in our org? I wouldn't want to rely on JSG for every release.

What do you think about including the now increasing portfolio of metadata formats we are relying on? This is starting to now get quite complex,
with apps-as-first-class (with Orion signing), argus access policies (#4332) and authentication (#3152), and it will only get more complex as people start inventing new versions of the same messages.

I'm not sure what you mean exactly by "including" it. We already have the @joystream/metadata-protobuf package that holds all the metadata formats. Is there something else you think we're lacking?

Where do you see QN fitting in here? I think a lot of things people want to do in this API requires QN or Orionv2 to really work.

Some parts of the SDK will probably need data from QN. I'm not yet sure what the approach should be, but my initial thinking was to make relevant parts work in one of 2 modes:

1. SDK fetches all the data by itself, given an GraphQL endpoint
2. Apps fetch the data themselves and only pass it to the SDK.

I think both approaches should be available to make sure you can get started easily, but remain flexible enough so you can tailor it to your needs.

Do you see this as an SDK for making any Joystream product, so even Colossus and Argus themselves, or are you specifically thinking narrowly on Consumer apps like Atlas?

I think the general goal should be to support any product, but we should start with needs of consumer apps and expand later.

Will the SDK perhaps have distinct parts with a focus on different use cases?

Yep, separate packages would have different focus, but we will see how exactly that shapes up.

I suspect that a great source of inspiration for what is currently causing complexity for app-developers is just to ask all current builders where they currently see complexity in their apps

Will be definitely doing that.

How can we get off to a great start in terms of documentation? It is always really painful to try to go back end do that later.

I will try to document anything that SDK does in-code, including the API, but I think some examples of putting it all together will come once we have at least some part of the SDK actually done

[bedeho]

Obviously we need to be careful here. Will it be possible to grant me write access to specific NPM packages in our org? I wouldn't want to rely on JSG for every release.

Will check.

I'm not sure what you mean exactly by "including" it. We already have the @joystream/metadata-protobuf package that holds all the metadata formats. Is there something else you think we're lacking?

I guess putting all of these together in a way that makes it easy to do what people actually want. Example, in stead of having to compose 2 or 3 such messages together in the right way, perhaps do signing on some part with an app-backend, you can just do one call which does all of it end to end, including publish appropriaet extrinsic on chain. Just a thought, and obviously low level control is something that also should be possible.