Concurrency

[arxanas]

Re the tech docs here, good exposition! 94fda79

git-branchless also has an operation log, but it's only linear. I've been wondering how to implement syncing across multiple clients, and the outlined approach seems pretty good.

Is each node in the operation log represented as a snapshot on disk? If so, doesn't that mean there's relatively little sharing between each node (i.e. the node is copied entirely if a single branch is changed)?

Did you consider a revlog approach at all? (Does jj use a revlog for commits?) Are the nodes stored in the regular jj object storage?

[martinvonz]

Is each node in the operation log represented as a snapshot on disk? If so, doesn't that mean there's relatively little sharing between each node (i.e. the node is copied entirely if a single branch is changed)?

Yep, that's how it currently works. I plan to add some delta encoding later, but it's not been much of a problem for me in my tiny repo.

My repo currently has 11,700 operations. It takes 700 ms to list them, despite the simple format. There are only 43 heads, 1 local branch, one remote branch, and one git ref at the current operation head. That makes recent view objects about 1.2 KiB in size. However, there are many much larger view objects (up to 184 KiB) from before I removed evolution (because then all the old heads were also tracked in each view object). The total size is 725 MiB.

By the way, regarding GC of the operation log, there's also no way of removing old operations yet (jj op log will crash if you manually remove old operations). I obviously need to add support for that some day.

Did you consider a revlog approach at all? (Does jj use a revlog for commits?) Are the nodes stored in the regular jj object storage?

I considered the revlog for a very short amount of time before I dismissed it :) The issue is that they can't be written to by concurrent processes (and are thus not rsync-/Dropbox-safe either).

And no, jj doesn't use revlogs for commits; it uses a very naive format similar to Git's loose object store. The nodes are not stored in the same format, but it's very similar (i.e. a naive content-addressed format with loose objects).

[arxanas]

rsync/Dropbox-safety might be an interesting point to call out explicitly in the README/documentation somewhere. It's not something I've been particularly comfortable with doing with Git.

For Dropbox-safety, does it actually turn out to be safe if only some of the files are synced and then the operation is interrupted? For example, if you sync a head file but not the pointed-to objects (or some but not all of the pointed-to objects in the parentage chain), will jj recover?

[martinvonz]

rsync/Dropbox-safety might be an interesting point to call out explicitly in the README/documentation somewhere. It's not something I've been particularly comfortable with doing with Git.

I had been thinking of adding that, so thanks for the reminder :) I've added it to the README.

For Dropbox-safety, does it actually turn out to be safe if only some of the files are synced and then the operation is interrupted? For example, if you sync a head file but not the pointed-to objects (or some but not all of the pointed-to objects in the parentage chain), will jj recover?

It's safe in the sense that data won't be lost, but it may simply crash if part of the repo has not yet been replicated. In the extreme case, you can imagine that everything has been replicated except for the initial version of the project's README. Then you'd be able to run jj log -p (once I've implemented the -p flag :)) and it would crash only once it reaches the first commit. However, it should start working once the replication has finished.

According to some simple testing I did yesterday, Git, Mercurial, and Pijul all lose changes instead. That's in the relatively simple case where the operations all start in a consistent state. If replication happens while the command runs, then at least Mercurial sometimes creates a broken repo.