Fetching supervoxel sizes after cleave takes a long time

[stuarteberg]

When a user commits a cleave, neu3 apparently fetches the /size of every supervoxel in the body. (I'm not sure why.) It fetches those sizes via independent calls to GET .../size, which is inefficient relative to fetching them all at once via a single call to /sizes.

For large bodies with lots of supervoxels (e.g. 10k-100k supervoxels), this process is the bottleneck when performing a cleave. It can take several minutes.

cc @knechtc @hubbardp @tingzhao

[tingzhao]

The sizes are used to check if a large portion of a body is cleaved off. I'll change it to /sizes to improve the effeciency.

[tingzhao]

I just realized that neu3 is actually using /sizes to fetch body sizes in batch. It will try to fetch one by one only when the batch fetching fails, but I couldn't find such an example. Did this problem happen often? @knechtc @stuarteberg

[stuarteberg]

It will try to fetch one by one only when the batch fetching fails

How can batch fetching fail? Some sort of timeout?

I couldn't find such an example. Did this problem happen often?

One example, from today's work in the motor neuron branch: 11706 in uuid 0898ddc exhibits the problem:

http://emdata5.janelia.org:8400/#/repo/0898ddca1d6e44b0be172586f615b1d7

[stuarteberg]

@tingzhao

It will try to fetch one by one only when the batch fetching fails,

If the batch fetching fails, then the body is probably quite large, and the 1-by-1 fetching will take a VERY long time.

but I couldn't find such an example.

As I write this, @knechtc is waiting for CNS halfbrain body 8900646457 to be cleaved. It has ~45k supervoxels, so we expect the above check to take approximately 3-6 hours before the cleave command is finally performed.

[stuarteberg]

BTW, the batch fetching really shouldn't fail once janelia-flyem/dvid#262 is resolved.