Basic evaluate CLI command / codepath

[djsaunde]

Description

Title.

Motivation and Context

I'd like to use this codepath to validate my differential transformers work. I.e., when converting a base model to one where we've swapped out the attentions for (zero-init) differential attentions, we expect the loss to be the same as before the conversion.

This will be generally useful for evaluating models post training, comparing models apples-to-apples, evaluating on different datasets, etc.

How has this been tested?

Only basic testing in Runpod on a A40 instance with the axolotlai/axolotl-cloud:main-latest image.
For example, with this simple config:

base_model: HuggingFaceTB/SmolLM2-135M
datasets:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca
test_datasets:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca
    split: train
sdp_attention: true
gradient_accumulation_steps: 1
learning_rate: 1e-4
max_steps: 1
micro_batch_size: 1
sequence_len: 2048
special_tokens:
  pad_token: <|endoftext|>

We get:

# axolotl evaluate ../smollm.yaml 
...
[2024-12-13 21:50:17,302] [INFO] [axolotl.eval.evaluate:88] [PID:72580] [RANK:0] Starting evaluation...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:56<00:00, 35.47it/s]
[2024-12-13 21:51:14,254] [INFO] [axolotl.eval.evaluate:101] [PID:72580] [RANK:0] Evaluation completed!
[2024-12-13 21:51:14,254] [INFO] [axolotl.eval.evaluate:102] [PID:72580] [RANK:0] Metrics:
[2024-12-13 21:51:14,254] [INFO] [axolotl.eval.evaluate:104] [PID:72580] [RANK:0] eval_loss: 1.8818285465240479
[2024-12-13 21:51:14,254] [INFO] [axolotl.eval.evaluate:104] [PID:72580] [RANK:0] eval_model_preparation_time: 0.0047
[2024-12-13 21:51:14,254] [INFO] [axolotl.eval.evaluate:104] [PID:72580] [RANK:0] eval_runtime: 56.8085
[2024-12-13 21:51:14,254] [INFO] [axolotl.eval.evaluate:104] [PID:72580] [RANK:0] eval_samples_per_second: 35.206
[2024-12-13 21:51:14,255] [INFO] [axolotl.eval.evaluate:104] [PID:72580] [RANK:0] eval_steps_per_second: 35.206
[2024-12-13 21:51:14,255] [INFO] [axolotl.eval.evaluate:116] [PID:72580] [RANK:0] Evaluation results saved to model-out/eval_results.txt

Types of changes

Adding a new top-level module axolotl.evaluate as well as a corresponding CLI module axolotl.cli.evaluate.

Test coverage is still outstanding. Done.

I'd also like to add evaluation on the training dataset (if passed). Done.

[winglian]

Minor nit on the print statement. Would be good to figure out a way to document that some of the evaluate/train code is duplicated so that we know to make changes in the other down the line. Otherwise lgtm