Periodically report memory usage

[andyleiserson]

The reporting is wired into seq_join. The reporting interval increases as the record count increases, which results in a tolerable amount of log messages for loops with many iterations, while still providing some reporting for shorter loops. It's still fairly chatty, so I put it at debug level. Here's a sample -- 1141 lines reporting memory usage vs. 583 other log lines -- but I reduced the frequency after generating this, so it's ~half of that now.

Note that the reporting depends on jemalloc. That is the default on linux. It can be activated on other platforms with a feature.

[akoshelev]

I originally disabled jemalloc for MacOS builds because using it had to value but building that crate takes a lot of time on M1 Pro. Now, with live profiling enabled, we could reconsider that as it is valuable to observe memory usage on dev machines

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.24%. Comparing base (6531c97) to head (007630e).
Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1503      +/-   ##
==========================================
- Coverage   93.24%   93.24%   -0.01%     
==========================================
  Files         238      241       +3     
  Lines       43688    43877     +189     
==========================================
+ Hits        40739    40912     +173     
- Misses       2949     2965      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andyleiserson]

There is more work needed before this could be merged. Right now jemalloc is only used in helper and oneshot_ipa, but telemetry::memory doesn't know whether it's active. The behavior when it's not is printing lots of 0 MiB allocated.

[akoshelev]

Seems that there is no perf impact, so should be good to go

[andyleiserson]

I originally disabled jemalloc for MacOS builds because using it had to value but building that crate takes a lot of time on M1 Pro. Now, with live profiling enabled, we could reconsider that as it is valuable to observe memory usage on dev machines

As we discussed, I left the jemalloc defaults unchanged since the build time concern remains outstanding. jemalloc is active by default on Linux, can be activated on MacOS with the jemalloc feature.

I also reduced the reporting frequency, so there are now fewer memory usage log messages than when I originally wrote the description.