Splitstore: the road to production readiness

[vyzo]

The following three prs bring splitstore to beta readiness, where users could be encouraged to use it for non-critical infrastructure nodes:

• splitstore sortless compaction #8008 -- sortless compaction.
• splistore cold object reification redux #8029 -- cold object reification.
• splitstore: set badger as the default default markset type #8034 -- set badger as the default markset type.

With these prs the splitstore is now code complete, with the exception of coldstore garbage collection (see below). As such, it is a good point to mark beta readiness and get wider testing from users.

In order to get to production readiness, a few more things remain to be done:

• document the splitstore in user docs #7120 -- documentation!
• Issues with syncing from scratch and long resyncs in the splitstore #6769 -- resolve from scratch syncing, which will currently blow the hotstore.
• Splitstore: Online Garbage Collection for the Coldstore #6577 -- online garbage collection for the coldstore.
• integration tests, where the splitstore is used in the lotus integration test suite.
• release stress test, where an instance of splitstore is started form a 2-day old snapshot and run for 2-3 compactions to verify that it syncs properly. We should probably test a matrix of options, with universal/discard coldstore and badger/map markset.
• (optionally) Splitstore Harness Test #6725 where we construct a harness test that can be used to verify correct behaviour and provide a regression test suite for future changes.
• implement a system to ship coldstores instead of snapshots.