Early Discussion: New save format to reduce save size and number of files

[randombk]

Is your feature request related to a problem? Please describe.

Context and Motivation

CBN currently stores saves as loose uncompressed JSON files. Each map chunk is a separate file. Each player-chunk tuple is another file.

With very little exploration, CBN saves balloon to over a GB spread out over many hundred thousand individual files. This slows down saving/loading and makes filesystem operations on saves particularly slow (i.e. backups, syncing across machines, etc).

Describe the solution you'd like

Proposal

Migrate to a new save format where map tiles are stored as compressed JSON blobs in a SQLite3 database.

• This would consolidate the multiple loose files, reduce storage sizes, all while minimizing the incremental complexity and dependencies introduced.
• Very minimal changes to the codebase are needed - no JSON schemas are changed, only the filesystem read/write layer.
• Migration to SQLite (and ACID transactions) would also help reduce save corruption.
• This format would also be easy to manually manipulate if/when necessary.
• Maintain backwards compatibility and incremental migration for existing saves.
• Improved efficiency of save format would also potentially open the door to saving more detailed information about the world in the future.

Proposed Migration Path

1. Backend cleanup of the world save/load codebase to make it easier to support multiple save formats. Keep the format unchanged at this stage.
2. Implement the new format under an experimental feature flag/option. There should be a migration path for old saves.
3. Roll the format out as the default format. Keep backwards compatibilty for a long/indefinite period of time.
4. (Optional) deprecate the old format, and keep around a script to upgrade from the old format

Additional context

No response

[Coolthulhu]

The reason for multiple save files is that we're semi-often modifying the files separately.
One benefit to current implementation is that the results are human-readable. Would the SQLite variant allow easy access to the data itself?

[thedyze]

How about simple zip files? Would allow any user to have easy access.

[Zireael07]

@thedyze Performance penalty for packing/unpacking and very tiny benefit (slightly beter organization)