Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor in_context_learning_evaluation.py #2713

Merged
merged 126 commits into from
Jan 26, 2024
+2,415 −1,246
Merged

Refactor in_context_learning_evaluation.py #2713

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
a49db7d
extremely wip commit w/ ICLdataset class
maxisawesome Oct 27, 2023
914f207
more extremely broken wip
maxisawesome Oct 30, 2023
330a97e
add split keys
maxisawesome Nov 3, 2023
453df28
first pass at moving QA to new format
maxisawesome Nov 6, 2023
89cf3f4
linting
maxisawesome Nov 8, 2023
e585e1f
linting
maxisawesome Nov 8, 2023
5078373
tests pass!
maxisawesome Nov 9, 2023
b4e00e4
fix repeated defaults, gold_idx --> gold
maxisawesome Nov 9, 2023
a588767
basic HF parsing but test not passing
maxisawesome Nov 9, 2023
cea8ff8
fix cot. wip
maxisawesome Nov 10, 2023
ae4c6bc
del device and world_size from tests
maxisawesome Nov 11, 2023
76a3f33
change to .map
maxisawesome Nov 11, 2023
0090b82
fix schema
maxisawesome Nov 11, 2023
ea03380
tests passing w/ collate refactor
maxisawesome Nov 13, 2023
bebbbda
finish HF tests
maxisawesome Nov 13, 2023
11419e1
Merge branch 'mosaicml:dev' into icl_refactor
maxisawesome Nov 14, 2023
1907cee
add hf batch parsing
maxisawesome Nov 14, 2023
503914a
linting
maxisawesome Nov 15, 2023
db0bcb8
add doc strings, rm hf_parsing_vars
maxisawesome Nov 16, 2023
c2dd31c
revert question_prelimiter back to prelimiter
maxisawesome Nov 16, 2023
3ad77e6
fix tests
maxisawesome Nov 16, 2023
14cf5e7
add more docstrings
maxisawesome Nov 16, 2023
f768563
add doc strings, fix hf w/ categories
maxisawesome Nov 17, 2023
b354d43
add doc strings and default check
maxisawesome Nov 17, 2023
23f8735
linting
maxisawesome Nov 17, 2023
887a31c
add temperature
maxisawesome Nov 17, 2023
b5afb91
remove need for hf:// on hf links
maxisawesome Nov 17, 2023
a033fb0
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Nov 23, 2023
c43f65b
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Nov 23, 2023
20c3542
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Nov 23, 2023
16a5d95
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Nov 23, 2023
5a15d68
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Nov 23, 2023
0aa6f6b
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Nov 24, 2023
ce63845
fix comments, add test for check hf uri, still wip
maxisawesome Nov 24, 2023
5705116
Merge branch 'icl_refactor' of github.com:maxisawesome/composer into …
maxisawesome Nov 24, 2023
8fa8da9
add gpu tests back
maxisawesome Nov 24, 2023
49c8b7d
update fix_eos_on_preamble
maxisawesome Nov 24, 2023
e62e12b
update comments
maxisawesome Nov 25, 2023
c98ca7d
add return types
maxisawesome Nov 25, 2023
b3635ab
typing, comments
maxisawesome Nov 26, 2023
77e8b1e
init RAG Generation task
Nov 26, 2023
75bf465
init _construct_context for RAG eval
Nov 26, 2023
288e8a8
fix context key, move hf test dataset, few docstrings
maxisawesome Nov 27, 2023
e8362a8
fix docstrings, add second path for schema
maxisawesome Nov 27, 2023
9a1a071
init collate_fn, _tokenize_example functions (bug exists)
Nov 27, 2023
c8d4538
fix typo in warning error
Nov 27, 2023
b9c34ae
Merge branch 'icl_refactor' into icl_refactor
eitanturok Nov 28, 2023
ea8c0eb
remove canonical_solution from batch
maxisawesome Nov 28, 2023
17198ff
missed one canonical_sllution
maxisawesome Nov 28, 2023
c07a4d9
remove encoded dataset to have just one dataset var
maxisawesome Nov 28, 2023
8c5df99
rename sample to example
maxisawesome Nov 28, 2023
bad4b30
improve comment
maxisawesome Nov 28, 2023
be9c6a5
Merge branch 'icl_refactor' of github.com:maxisawesome/composer into …
maxisawesome Nov 28, 2023
bbd00b5
edit RAGtask
maxisawesome Nov 28, 2023
a4b63b7
rm hf parsing func
maxisawesome Nov 30, 2023
ab87b70
fix docstring, rename fewshot fun
maxisawesome Nov 30, 2023
44694b0
docstring
maxisawesome Nov 30, 2023
a12da95
change default split_batch to check types
maxisawesome Nov 30, 2023
7ad3199
remove need to set split_keys
maxisawesome Nov 30, 2023
e5de590
doc string update
maxisawesome Nov 30, 2023
24c50df
improve comments
maxisawesome Nov 30, 2023
6e41225
rm stacked_keys for tokenize_labels bool
maxisawesome Nov 30, 2023
7bad80c
initial wip in comments
maxisawesome Nov 30, 2023
393bc4b
make _conv_tokens_to_tensors func
maxisawesome Nov 30, 2023
5a04489
wip - sketch out batch_mappings
maxisawesome Nov 30, 2023
1f393c0
linting and debugging statements to help me remember where I'm doing wip
maxisawesome Dec 4, 2023
d0e7957
all tests except one sus schema test passing
maxisawesome Dec 5, 2023
5caffbf
fix missing fewshot for schema
maxisawesome Dec 5, 2023
bf53ec1
rm temperature add generation_kwargs
maxisawesome Dec 5, 2023
b90ca7f
add defaults that are currently set in llm-foundry builders.py
maxisawesome Dec 5, 2023
820e7fc
fix defaults in tests, add some comments
maxisawesome Dec 6, 2023
51c0571
tests wip
maxisawesome Dec 6, 2023
be900ab
tests for new funcs
maxisawesome Dec 7, 2023
20b0ecf
rm RAG task
maxisawesome Dec 7, 2023
2285f3f
more docstring
maxisawesome Dec 7, 2023
817030e
tests passing
maxisawesome Dec 7, 2023
fd0e204
wip
maxisawesome Dec 7, 2023
d3abb91
wip
maxisawesome Dec 7, 2023
3be5fe8
add dict to data_spec
maxisawesome Dec 8, 2023
6f93061
Merge pull request #2 from maxisawesome/icl_refactor_debug
maxisawesome Dec 8, 2023
988422e
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Dec 12, 2023
3a28e19
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Dec 12, 2023
bfbb70a
Apply suggestions from code review
maxisawesome Dec 12, 2023
1ce6f7a
default_batch to base_batch and some docstrings
maxisawesome Dec 12, 2023
8ea72be
update comments and fix test. move spacing to default get_answer
maxisawesome Dec 14, 2023
85097ce
improved docstrings
maxisawesome Dec 17, 2023
4139fbc
finish schema/mc tests
maxisawesome Dec 18, 2023
eb382fa
address pr review comments
maxisawesome Dec 20, 2023
ea337d5
lintign
maxisawesome Dec 20, 2023
735e0ee
fixing import, add type
maxisawesome Dec 26, 2023
350ce5b
update comments
maxisawesome Jan 17, 2024
3487a0a
Merge branch 'dev' into icl_refactor
maxisawesome Jan 17, 2024
111e1d1
passing running btu not passing. merged dev into refactor
maxisawesome Jan 18, 2024
97b32f2
update keys
maxisawesome Jan 18, 2024
9b2fcde
add typechecks for token ids
maxisawesome Jan 18, 2024
ca4f7b8
rm outdated test
maxisawesome Jan 18, 2024
9628d19
fix tests
maxisawesome Jan 18, 2024
e0c0117
add microbatch test
maxisawesome Jan 19, 2024
cc01493
pyright fixes
maxisawesome Jan 22, 2024
174a1d6
linting attempts
maxisawesome Jan 23, 2024
a894be2
linting wip
maxisawesome Jan 23, 2024
a87a7d7
fix linting
maxisawesome Jan 24, 2024
eae8a1c
add early stopping and do_normalization documentation
maxisawesome Jan 24, 2024
fc970b1
Merge branch 'mosaicml_dev' into icl_refactor
maxisawesome Jan 24, 2024
5be0cc9
fix linting
maxisawesome Jan 24, 2024
1fd12fc
fix linting
maxisawesome Jan 25, 2024
5f12dc5
fix final dist test issue
maxisawesome Jan 25, 2024
f531dfc
fix isort
maxisawesome Jan 25, 2024
3e71cb3
fix linting
maxisawesome Jan 25, 2024
e487934
fix docstrings
maxisawesome Jan 25, 2024
c5ca3f8
fix docstrings
maxisawesome Jan 25, 2024
712a33d
add warning filters
maxisawesome Jan 25, 2024
b305a4b
fix warnings
maxisawesome Jan 25, 2024
fbef372
Merge branch 'mosaicml_dev' into icl_refactor
maxisawesome Jan 25, 2024
9ed99fd
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Jan 25, 2024
37d5f9b
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Jan 25, 2024
5ff9a30
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Jan 25, 2024
c44c763
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Jan 25, 2024
7e07084
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Jan 25, 2024
b8cae18
Update composer/datasets/in_context_learning_evaluation.py
maxisawesome Jan 25, 2024
a840abb
add capitalization
maxisawesome Jan 25, 2024
c65aab8
revert default changes
maxisawesome Jan 25, 2024
3d5c700
change update_generate_kwargs to public
maxisawesome Jan 25, 2024
5e11203
fix type
maxisawesome Jan 25, 2024
d48a3a9
move pad_tok_id error
maxisawesome Jan 25, 2024
e90a862
Merge branch 'dev' into icl_refactor
dakinggg Jan 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading