Automatically prune history of a document

[fflorent]

History actions are mostly useful when displayed through the UI, otherwise even for much advanced users, it can be very difficult to consult and have a clear vision of the actions that have been done.

Also these actions can dramatically increase the size of the document.

Because of that, we tend to think it would make sense to regularly prune the oldest ones. The documents snapshots would help find older history in most cases. What do you think?

[fflorent]

Functional considerations:

I have several questions in mind if that what we would like to do:

• should we let the admin configure that? should a value of 0 be valid and tell we don't want action history at all?
• how many actions do we want to keep by default?
• do we want to prune history actions per table or per document? the latest option seems much simpler, the first one maybe more user-friendly

---

Technical considerations:

• I would introduce a throttled function that would prune the document history each time an action has been inserted in DB;

[paulfitz]

Hi @fflorent, there is some existing auto-pruning logic:

grist-core/app/server/lib/ActionHistoryImpl.ts

Lines 16 to 23 in 7bae7a8

	// History will from time to time be pruned back to within these limits
	// on rows and the maximum total number of bytes in the "body" column.
	// Pruning is done when the history has grown above these limits, to
	// the specified factor.
	const ACTION_HISTORY_MAX_ROWS = 1000;
	const ACTION_HISTORY_MAX_BYTES = 1000 * 1000 * 1000; // 1 GB.
	const ACTION_HISTORY_GRACE_FACTOR = 1.25; // allow growth to 1250 rows / 1.25 GB.
	const ACTION_HISTORY_CHECK_PERIOD = 10; // number of actions between size checks.

Making it configurable would be easy and desirable.

Pruning to 0 would currently leave the document unusable, since in there is currently no way to work with a document with a null "current last action". I'm sure this could be overcome. An alternative could be to add an empty action and then prune everything before it.

Pruning history actions per table would be a significant project, since it would require rewriting history, not just removing it.

One reason not to prune much history: I don't know if you use the functionality for working-on-a-copy, making-edits, then replacing-original? Any feature like this relies on versions of a Grist document being comparable in the 3-way sense needed for version control (meaning that the common ancestor of the versions is known). Once you prune history, documents that diverged before the beginning of that history become incomparable in the version control sense - you can see changes, but there's no way to know if they were e.g. removals on one side or additions on the other.

An idea that rattles around from time to time is external storage of history.

[paulfitz]

External storage of history is attractive since documents could retain hashes of history items - these already exist - and only consult their actual content when needed, such as for version comparison. There could be a shared mechanism with external attachments. But this is a relatively large change.

[fflorent]

Thanks @paulfitz! I understand that it should be quite easy to let the admin configure the history, but we should advise (or force) them to have a minimal size for the feature you mention (and at least have at least one history action in the document if that's what they really want).

External storage of history is attractive since documents could retain hashes of history items - these already exist - and only consult their actual content when needed, such as for version comparison. There could be a shared mechanism with external attachments. But this is a relatively large change.

I created this issue to track the idea

[vviers]

Nice-to-have : automatically and retroactively prune document history in snapshots as well once ACTION_HISTORY_MAX_ROWS and ACTION_HISTORY_MAX_BYTES are configured

[fflorent]

We will only implement solution in #1262, which is enough for our own use case.