feat: Adjust JSON structure to allow for other mod sources

[GeckoEidechse]

META PR, no verification needed (although it does touch verified-mods.json, so give it a double check nevertheless)

Will require an update to launcher as well.

Closes #22 as it adjusts the project to allow other sources in the future.
(Does not add another source in itself though)

Requires:

• Launcher PR with updated logic: Adjust for restructured verified-mods.json NorthstarLauncher#642

Do NOT merge until corresponding launcher PR has been merged and released and Parkour event has concluded.

[GeckoEidechse]

Marking as ready for review for discussion etc but should not be merged until the conditions in PR descriptions have been met ^^

[GeckoEidechse]

cc @EM4Volts

[Alystrasz]

Instead of having a thunderstore member and one member per platform that will invariably lead to code duplication, shouldn't we agree on a manifest format that is platform agnostic?

[ASpoonPlaysGames]

Instead of having a thunderstore member and one member per platform that will invariably lead to code duplication, shouldn't we agree on a manifest format that is platform agnostic?

That's not really up to us though, different platforms may enforce different identifiers for mods, code duplication to some degree is pretty inevitable

[Alystrasz]

That's not really up to us though, different platforms may enforce different identifiers for mods, code duplication to some degree is pretty inevitable

Do you have any kind of identifier in mind?
Lemme try to propose some kind of format.

[Alystrasz]

Ok so the issue with the current implementation is that it only considers thunderstore (launcher too, since it builds thunderstore mod download URL using DependencyPrefix and Version members from manifest); explicitely mentioning them in the manifest would theorically allow us to support any platform:

{
    "Odd.s2space": {
        // Repository of the source code, must be public
        "Repository": "https://github.com/uniboi/s2space",
        "Versions": [
            {
                "Version": "0.0.5",

                // Ensure corresponding release exists on source repo
                "CommitHash": "f27b8f1f05d5278aa8f47ead2d9e70f39f274173",

                // Direct download URI, used by launcher to download mod
                "DownloadLink": "https://gcdn.thunderstore.io/live/repository/packages/odds-s2space-0.0.5.zip",

                // Archive checksum, used by launcher to verify mod integrity
                "Checksum": "670987e07806e8dffcb591bad8724f29abc18d9baa304d9ab4fae7804bd86bc2"
            },
            {
                "Version": "1.0.0",
                "CommitHash": "0eca593ac1666aeeaa061a5c545d917d47d6d113",
                "DownloadLink": "https://whatever.my.platform.is/packages/odds-s2space-1.0.0.zip",
                "Checksum": "eef8ae5a88df68b249529f31713d0219b18391202dfeedc2d109936c729c571c"
            },
            ...
        ]
    }
}

[GeckoEidechse]

One thing I'm concerned with a fully generic approach is that if we move over to a completely generic version, how do we handle namespacing etc in the remote folder in Northstar?

Different platforms may have different formats for the mods which means we'd have to try different methods for loading a mod per remote platform on each folder.
Additionally, we'd then have to also check for namespacing collisions between two mods (while unlikely to happen it's something you'd have to keep in mind for each verification)

If we separate on platform, we can handle the logic specific to that platform. For example, apparently GitHub provides zip hashes of release via API, so we can check in advance if the hashes match before even downloading (and then still recheck again afterwards to ensure no corruption during transfer).

Additionally, we can also stick the mods into different subfolders in remote dir, based on the platform and handle name spacing based on that platform.
E.g.

.
└── remote
    └── mods
        ├── github
        │   └── ghuser.ghrepo.version
        └── thunderstore
            └── Alystrasz-Parkour-1.2.3

We can of course still share logic between platforms for e.g. downloading the zip files to avoid code duplication.

[Alystrasz]

I thought the point of the R2Northstar/Northstar#618 discussion was to have a unified format for all mods, no matter where they come from?

Regarding the mod format changing from platform to platform, to me, it is the responsiblity of mod managers to format said mods to match the unified, agreed-upon format.

[GeckoEidechse]

I thought the point of the R2Northstar/Northstar#618 discussion was to have a unified format for all mods, no matter where they come from?

We are still quite a bit away from that though. Like we haven't even started out hashing out the standard yet and I don't we want MAD¹ to be held back by that ^^"

 

Regarding the mod format changing from platform to platform, to me, it is the responsiblity of mod managers to format said mods to match the unified, agreed-upon format.

Given that we're talking about verified mods here, wouldn't the "mod manager" in this case be Northstar itself as verified mods are for MAD which is handled by Northstar itself? Or am I missing something here?

Footnotes

1. mod-auto-download ↩

[GeckoEidechse]

Something that came to mind right now, having top level keys per platform like thunderstore or github, also means that we could always just add another key like generic for downloading zip from any HTTP server, exactly like proposed in your suggestion.

While I'm still strongly in favour to limiting verified mods to only certain platforms, should we ever decide to move to the generic solution, we could simply keep the existing format and just only extend the generic key allowing for further backwards compatibility for older verified mods ^^

[ASpoonPlaysGames]

I thought the point of the R2Northstar/Northstar#618 discussion was to have a unified format for all mods, no matter where they come from?

Yes but even still, platforms have their own requirements. As an example, TS requires an icon.png file in the root of the package. Even if we have a unified mod format, different platforms will package the same mod in different ways, and we therefore need different ways to handle this

[Alystrasz]

Yeah kinda agree with both of you on that, but it's the launcher who needs to handle different platform mod formats, not this repository

[GeckoEidechse]

Yeah, and so splitting the sources into different categories in the JSON allows us keep the code in launcher simple and easily extendible, as we don't need to add any logic that first tries to figure out what specific platform a download source is coming from ^^

[Alystrasz]

The launcher would still need to install mod from different format, and since figuring out the platform from download URL is straightforward, I don't get your point

[GeckoEidechse]

Idk, I find parsing strings/URLs a bit annoying...

In the end, it boils down to how we wanna structure the data.

If we don't make the platform identifier the root key, I'd still suggest having it in the object that defines the mod so that we don't have to deal with parsing the download URL but rather can just generate it ourselves.

This is especially useful when the platform URL might change like for example for us the Thunderstore URL is https://northstar.thunderstore.io/ but for Lethal Company it's https://thunderstore.io/c/lethal-company/ ¹. Not sure if we ever get moved over but if we would be, old links might break so keep stuff like that in mind I guess.

Also the mods on the platform might not all follow the same structure. Like with the proposed mod rewrite, old mods will have a different layout then newer ones until re-verified, meaning we either drop all old mods in the process or offer an identifier like thunderstore-legacy during the transition period.

With using platform identifier as root key we could also nicely match it up with how we could place mods in the remote mods dir like mentioned above so we can keep them properly namespaced.

That being said, I don't have the resources atm to refactor this PR if we feel like we should have a different layout than the one proposed in this PR rn.
Feel free to give it a shot though. You might also wanna consider just asking around in #development channel which format would be preferred ^^"

Footnotes

1. https://thunderstore.io/c/northstar/ works for Northstar as well btw ^^ ↩

[GeckoEidechse]

Superseded by #41