Multi eval dataset logging

[snarayan21]

Previously, only one eval dataloader was supported in llm-foundry (although multiple ICL eval tasks and the gauntlet were supported). Now users can add custom eval datasets by turning eval_dataloader into a list of dataloaders in their yaml. It would look like this:

Before:

eval_loader:
  name: text
  dataset: ...
  drop_last: false
  num_workers: 8

After:

eval_loader:
- label: first
  name: text
  dataset: ...
  drop_last: false
  num_workers: 8
- label: second
  name: text
  dataset: ...
  drop_last: false
  num_workers: 8

Users must specify a label for each eval dataloader so that these metrics are logged separately. This functionality was tested on wandb, mlflow, and tensorboard, shown below.

wandb:

mlflow:

tensorboard:

[dakinggg]

Implementation looks good, thanks! Please remove the accidentally committed files, and add a simple unit test.

[irenedea]

LGTM with some minor comments

[irenedea]

Can we add a unit test that tests two eval loaders with two different datasets?

[irenedea]

LGTM! Just a minor comment about using tmp_path instead