Metrics Orchestrator

[jasonlyik]

Should break out the metrics orchestration (initialization, hooks, accumulation, etc.) into a separate MetricsOrchestrator managing class. Currently it is built into the Benchmark functionality, but this limits extension and customized metrics for the user.

User custom metrics should have abstract class, which describes how to fill in the metric. Current AccumulatedMetric is already a good example.

• We may want different abstract base classes for StaticMetric, non-accumulated workload metric, and the existing AccumulatedMetric.

All workload metrics have the same interface (model, preds, data), and all static metrics have the interface (model)

[jasonlyik]

Notes 9/12

Refactor metrics to all be objects, add legacy support where if all strings then will work but send warning that support will end

User interface could look like:
from neurobench.metrics import ...
# import all from .metrics but can have dir structure with static/workload split
from neurobench.metrics import StaticMetric, WorkloadMetric # custom objs