Introduce "Multi Wiki Server" Plugin

[Jermolene]

Introduction

Please see https://mws.tiddlywiki.com/ for instructions on how to use MWS

This PR introduces the "Multi Wiki Server" (MWS) plugin. It adds support for hosting multiple wikis at the same time. For performance and scalability, it uses SQLite to store tiddlers.

Hosted wikis are independent of the main wiki, which is still used for administration functions:

• storing configuration for managing the hosted wikis
• generating templates for static HTML routes

Related Work

SQLite Browser Wasm Experiments

Last year, I worked on #7329 which involved using the Wasm version of SQLite in the browser to make an alternate implementation of the wiki store object. The current status is that everything works but the overall performance is worse than the current JavaScript store implementation. I believe that it is possible to significantly improve on the JS performance by compiling filters directly into SQL. I hope to return to that work, but in the meantime I think we can get a lot of value from SQLite on the server without requiring the ability to execute filters against tiddlers in the database.

Bob.Exe

MWS has similar goals to @inmysocks's Bob.Exe at https://github.com/OokTech/TW5-Bob with the key differences being:

• MWS uses SQLite for storing the content of the wikis
• MWS uses the recipe/bag model for sharing content

Progress

• New tiddler schema that uses a record for each field instead of trying to pack them into a single record
• SqlTiddlerStore class for storing and retrieving tiddlers with bags and recipes
• Tests for the SqlTiddlerStore class
• New /wiki/:recipe_name HTTP route that serves a TW5 wiki template with the tiddlers from a given recipe spliced in
• Support for hosted wikis to synchronise via the API
• Use a persistent disk-based database
• Improved syncing that has persistent tiddler revision numbers, and thus can cope with server restarts
• Allow configuration of the hosted wikis via the main wiki (without needing a restart of the server) [much more to do, but the basics are there]
• Support for use of _canonical_uri for tiddlers >10MB
• Support for fields containing large blobs that are stored in a separate content addressable file store indexed by the MD5 hash of the content of the blob. This will allow seamless handling of very large tiddlers (eg video files)
• Support for authentication and authorisation
• Consider caching the wiki template
• Instant syncing from server to browser
• Fix uploading the same attachment twice – in particular, might be problematic if done via two browsers that happen to send a different content type
• Fix editing tiddlers that contain attachments
• Avoid making text tiddlers become attachments
• Support for deleting attachments when the associated tiddler is modified or deleted
• Update --mws-save-archive to handle attachments
• Apply length limits while processing multipart form data to prevent DDOS attacks
• Support HTTP range requests to allow streaming videos to support seeking within video

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Jul 16, 2024 9:38am

[pmario]

@Jermolene -- Is this PR intended to collect early feedback, or just to track your own progress?

[Jermolene]

@Jermolene -- Is this PR intended to collect early feedback, or just to track your own progress?

It's no different from any other draft PR, which is to say that it is the product of working in the open, and is published so that others can review and provide feedback.

[linonetwo]

Cool, can different bags belong to different folders?

(So they can be synced to different Github repos, some of them are private bags and in private Github repo, and some public will be used to build public gh pages blog).

[Jermolene]

Hi @linonetwo

Cool, can different bags belong to different folders?

This is not a hierarchical structure; bags are flat and do not contain other bags.

Bag can appear in multiple recipes at the same time.

(So they can be synced to different Github repos, some of them are private bags and in private Github repo, and some public will be used to build public gh pages blog).

That is correct, there are several motivations that might apply for placing a particular set of tiddlers in a bag of their own:

• To be able to independently include those tiddlers in any other recipe (ie to reuse them in other wikis)
• To make it easier to sync those tiddlers independently
• To be able to apply access controls to those tiddlers (bags represent access control boundaries)

[linonetwo]

Sorry for the confusion,

can different bags belong to different folders?

I want to ask about the folder structure on the file system, will it still be one tiddlers/ folder?

I know bag is a field on the .tid file or .meta file, so maybe all of these "bags" are still on the same tiddlers/ folder on the file system?

[Jermolene]

I want to ask about the folder structure on the file system, will it still be one tiddlers/ folder?

The tiddlers for the hosted wikis are stored in a SQLite database, they are not stored as files in the file system.

[pmario]

@Jermolene -- This may be of interest: https://nodejs.org/docs/v22.12.0/api/sqlite.html#sqlite

[Jermolene]

@Jermolene -- This may be of interest: https://nodejs.org/docs/v22.12.0/api/sqlite.html#sqlite

Goodness it will make life easier to have sqlite integrated into Node.js, excellent. When is it scheduled to hit the release version of Node.js?

[pmario]

At the moment it is Stability Index: 1.1 -- Active Development

See: https://nodejs.org/api/documentation.html#stability-index

May be middle of 2025 -- Once it reaches 1.2 RC we could do more serious testing.

[pmario]

I did a test with Deno's SQLite module, but it crashes -- The API seems to be incompatible.