Scaling memory consumption?

[chrisspen]

Is there any way to make the publish command scale better so it doesn't consume so much memory?

I'm trying to run it over a directory that has a few hundred thousand files, and the command completes, but it consumes 21 GB of memory (that's nearly all of the working memory on my 32GB system), which causes a lot of swapping.

If you give me some hints about where the main memory usage is, I could potentially optimize it and push up a PR for the fix.