change ARC primarycache away from all for crucible datasets

[faithanalog]

Currently untested - creating the draft now to see if CI is ok with it.

oxidecomputer/crucible#692

Logic is caching block data downstairs doesn't make sense. 3x downstairs means 3x cache of the same data, taking memory away from other usecases. It's on the other side of a network link so what is that cache really getting, if anything. Plus, the guest VM will have its own block cache. Want more block cache? Give the VM more ram.

Right now I'm threading config in from the very stop of when the DatasetRequestss are created in sled-agent/src/rack_setup/plan/service.rs. We could instead make this attribute be dependent on the DatasetKind, so that the code which does the final bridging from the sled-agent layer down through the lower "running zfs commands" layer is the only part that changes.

Doing it at the level my current code does feels like it might be digging some low-level stuff too high in the stack. On the other hand, if any other service datasets want to make this change later, it's very clear how to do it. I don't have strong feelings here, happy to leave as-is or change things around.

[jclulow]

FWIW, I think it would be best to make this decision inside the crucible zone, probably in the crucible agent, assuming it is still responsible for creating and destroying the datasets for regions.

If other services end up making similar decisions, I think they should likely still be responsible for it on their own as well. If, for example, we decide Clickhouse is double caching: once in its internal page cache, and once in the ARC, we might choose to make a Clickhouse configuration change to increase the page cache size in the same version update as we disable the ARC level caching. And then we might decide that was wrong and change it all back the other way.

This is ultimately similar to other ways in which zones are self assembling: we give them a dataset, and what they do with it is up to them. The sled agent does not need to know.

[faithanalog]

ah ok, yeah ill go look at doing that instead then

[faithanalog]

closing this for now, but I'll leave the branch around until after I chase down doing it in crucible-agent in case it's worth peering at.