Add eval loader to eval script #742

aspfohl · 2023-11-16T02:01:21Z

No description provided.

llmfoundry/data/dataloader.py

dakinggg

Could you please do a test run that does both normal eval and ICL eval to make sure it works?

llmfoundry/data/dataloader.py

scripts/eval/eval.py

llmfoundry/utils/builders.py

scripts/eval/eval.py

llmfoundry/utils/builders.py

scripts/eval/eval.py

scripts/train/train.py

tests/test_eval.py

llmfoundry/data/dataloader.py

tests/test_dataloader.py

scripts/train/train.py

tests/data_utils.py

llmfoundry/utils/builders.py

dakinggg

LGTM! Please include a manual test of:
(1) a training run with an eval loader
(2) an eval run with an eval loader

scripts/eval/eval.py

aspfohl · 2023-11-30T20:44:37Z

@dakinggg

a training run with an eval loader

mpt-1b-ctx-8k-gpus-8-3IPwqf

yaml diff from foundry

an eval run with an eval loader

mpt-eval-YwAj8C (stopped early)

yaml diff from foundry

dakinggg

LGTM, thanks!

aspfohl added 2 commits November 15, 2023 18:00

Add eval loader to eval script

774c62e

small input tests

6c7af31

aspfohl commented Nov 16, 2023

View reviewed changes

llmfoundry/data/dataloader.py Show resolved Hide resolved

aspfohl requested review from bmosaicml and dakinggg November 16, 2023 17:52

aspfohl marked this pull request as ready for review November 16, 2023 17:52

Merge branch 'main' into anna/evalloader

0797a6c

dakinggg reviewed Nov 16, 2023

View reviewed changes

llmfoundry/data/dataloader.py Show resolved Hide resolved

scripts/eval/eval.py Outdated Show resolved Hide resolved

llmfoundry/utils/builders.py Outdated Show resolved Hide resolved

aspfohl and others added 6 commits November 27, 2023 12:52

updates

13cbc13

fix typing and formatting

204d2f7

fixes, add tests

5b85218

Merge branch 'main' into anna/evalloader

1a0a3da

remove circular dependency

972ed96

tests pass

9840589

aspfohl requested review from dakinggg and irenedea November 28, 2023 01:10

irenedea reviewed Nov 28, 2023

View reviewed changes

dakinggg reviewed Nov 28, 2023

View reviewed changes

scripts/train/train.py Outdated Show resolved Hide resolved

dakinggg reviewed Nov 28, 2023

View reviewed changes

tests/data_utils.py Outdated Show resolved Hide resolved

dakinggg reviewed Nov 29, 2023

View reviewed changes

tests/data_utils.py Outdated Show resolved Hide resolved

aspfohl added 2 commits November 29, 2023 23:42

nits + small fixes

bc1c7c7

add metrics at the end, refactor to put icl/gauntlet as helpers

1050def

aspfohl commented Nov 30, 2023

View reviewed changes

llmfoundry/utils/builders.py Show resolved Hide resolved

aspfohl and others added 2 commits November 29, 2023 16:46

Merge branch 'main' into anna/evalloader

beefdc3

NOT

549abb6

dakinggg reviewed Nov 30, 2023

View reviewed changes

scripts/eval/eval.py Outdated Show resolved Hide resolved

metrics instead of models, add unit tests

084d8b6

aspfohl requested review from irenedea and dakinggg November 30, 2023 20:44

dakinggg approved these changes Nov 30, 2023

View reviewed changes

aspfohl merged commit 3100859 into main Nov 30, 2023
10 checks passed

aspfohl deleted the anna/evalloader branch November 30, 2023 22:02

aspfohl mentioned this pull request Dec 9, 2023

Async eval callback #702

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add eval loader to eval script #742

Add eval loader to eval script #742

aspfohl commented Nov 16, 2023

dakinggg left a comment

dakinggg left a comment

aspfohl commented Nov 30, 2023 •

edited

Loading

dakinggg left a comment

Add eval loader to eval script #742

Add eval loader to eval script #742

Conversation

aspfohl commented Nov 16, 2023

dakinggg left a comment

Choose a reason for hiding this comment

dakinggg left a comment

Choose a reason for hiding this comment

aspfohl commented Nov 30, 2023 • edited Loading

dakinggg left a comment

Choose a reason for hiding this comment

aspfohl commented Nov 30, 2023 •

edited

Loading