worktree -> index -> tree

[jeffparsons]

Hi @Byron,

Do you already have a rough API (or set of APIs) shape in mind for the worktree -> index -> tree flow? I found #301 and #293 but unless I missed something, they don't really seem focused on this direction.

Context

I'm playing around in my meager spare time with using gitoxide to implement a better version of part of something I built for $DAY_JOB a couple of years back using libgit2. My employer is hoping to open-source the whole thing eventually, but of course with companies that tends to take time. My personal hope in the meantime is to carve off little bits of functionality along the way into individually-useful open source libraries.

The first of these libraries would be for filtering Git trees, including automatically making a new tree with your worktree changes. The two modes of operation would be:

• Compute an OID of the filtered tree without actually constructing/storing the trees along the way, for efficiently making a content-based identity of some subset (not a necessarily just a single subtree) of the tree. This would be to replace the part of the tool mentioned above that is currently implemented using libgit2. I've already implemented the part of this that operates on trees that already exist in the Git repo (super-easy with gitoxide — thanks! 😀) but I'd love to be able to include a version that implicitly includes all your working tree changes. The purpose is to create a stable identifier of this unique version of the relevant files in the tree, which should be the same after committing if you commit all your working tree changes. I.e. it should be based on the exact content that would be committed (taking into account .gitignore rules etc.), regardless of whether or not it is committed.
• Later, construct, store, and even transmit the actual trees (wrapped in a temporary commit, not on any branch) to support a fun use case involving only letting the build for some component in a monorepo touch files that it has declared it uses. This would support hypothetical new functionality in the tool mentioned above.

[Byron]

Thanks for starting this conversation! Your inability to find anything about the index -> tree is definitely a sign of me not having that on the road-map yet. Trying to get to a full repository clone doesn't require that yet (but it requires to generate an index from a tree, unrelated).

A few things I picked out that I can reference here

• creating objects in memory is possible with the Sink type in git-odb - based on this it should be easy to build an in-memory object store or build trees fast.
• diffing the work tree directory with the index isn't implemented at all and isn't on the road-map either.
  • Finding modifications is quite easy as it needs only a few stat comparisons, doing it right ultimately means to implement it exactly like git does. That could certainly be a contribution even though I don't know how the API would look like.
  • Finding untracked files efficiently is not an easy feat (git has quite some optimizations there) but if you were to implement a simple version you could get an exclude-file cache to skip excluded files during iteration, and see if the rest can be found in the index.

My recommendation here is to use what's in gitoxide to implement modification checks and finding untracked files yourself because you can. From there it might be to get bits and pieces merged upstream, or changes you need to implement this in the first place.

From an effort perspective, I wonder why you would want to change to gitoxide in the first place. Is it for performance, or for usability, or for reliability?

[jeffparsons]

I just somehow fat-fingered and deleted my whole reply just before sending it, so I'm going to reply with something less rambling instead. 🙀

diffing the work tree directory with the index isn't implemented at all and isn't on the road-map either.

This is probably the most helpful piece of information for me. I'll probably be better off sticking with libgit2 for now, and treating "teach gitoxide to diff worktrees against trees" etc. as a separate stream of work, so I don't end up making a frankenlibrary that takes forever to deliver any value!

From an effort perspective, I wonder why you would want to change to gitoxide in the first place. Is it for performance, or for usability, or for reliability?

I don't actually have a good business case for switching yet, which is why I'm looking into this strictly in my own time for now.

My reasons, roughly ordered:

1. My own personal happiness. Even though I appreciate all that libgit2 has done for me, I have also found a lot of sadness in getting to the bottom of its bugs and performance issues, particularly because a lot of the code is uncommented and employs "clever" patterns that obscure the actual logic and provide excellent habitat for bugs.
2. Performance and control. I ran into a frustrating limitation of libgit2 a few years back. IIRC when diffing a worktree against a tree, it does the work to format a textual patch even if all you want is a list of which files have changed. (This was a while back; I could be misremembering the detail.)
3. Hackability. It feels like there's a much better chance of me being able to successfully contribute changes upstream here.
4. Remove one more C dependency. I'd like to be able to support Wasm down the line, and in my experience having fewer non-Rust dependencies tends to make this a lot easier.

[Byron]

I am glad I could help!

And I am particularly thankful for seeing your motivations to make a switch! They are a validation of what I thought one of the value propositions of gitoxide is going to be.