Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a config not to shuffle merged dataset #1394

Merged
merged 7 commits into from
Mar 19, 2024
Merged

Add a config not to shuffle merged dataset #1394

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
53b0bf7
Add a config not to shuffle merged dataset
seungduk-yanolja Mar 11, 2024
10d19d2
Update README.md
seungduk-yanolja Mar 11, 2024
4e05beb
Update src/axolotl/utils/config/models/input/v0_4_1/__init__.py
seungduk-yanolja Mar 13, 2024
59ccd21
invert the condition name
seungduk-yanolja Mar 13, 2024
75579fb
update README
seungduk-yanolja Mar 13, 2024
abfc7dd
info -> debug
seungduk-yanolja Mar 13, 2024
203a358
Merge branch 'OpenAccess-AI-Collective:main' into not_shuffle
seungduk-yanolja Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -678,6 +678,10 @@ datasets:
# For `completion` datsets only, uses the provided field instead of `text` column
field:

# If false, the datasets will not be shuffled and will keep their original order in `datasets`.
# The same applies to the `test_datasets` option and the `pretraining_dataset` option. Default is true.
shuffle_merged_datasets: true

# A list of one or more datasets to eval the model with.
# You can use either test_datasets, or val_set_size, but not both.
test_datasets:
Expand Down
1 change: 1 addition & 0 deletions src/axolotl/utils/config/models/input/v0_4_1/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,7 @@ class Config:

datasets: Optional[conlist(Union[SFTDataset, DPODataset], min_length=1)] = None # type: ignore
test_datasets: Optional[conlist(Union[SFTDataset, DPODataset], min_length=1)] = None # type: ignore
shuffle_merged_datasets: Optional[bool] = True
dataset_prepared_path: Optional[str] = None
dataset_shard_num: Optional[int] = None
dataset_shard_idx: Optional[int] = None
Expand Down
13 changes: 10 additions & 3 deletions src/axolotl/utils/data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,8 +415,11 @@ def for_d_in_datasets(dataset_configs):
dataset = concatenate_datasets(datasets)

if len(datasets) > 1:
LOG.info("shuffle merged datasets")
dataset = dataset.shuffle(seed=seed)
if cfg.shuffle_merged_datasets:
LOG.debug("shuffle merged datasets")
dataset = dataset.shuffle(seed=seed)
else:
LOG.debug("NOT shuffling merged datasets")

dataset, _ = process_datasets_for_packing(cfg, dataset, None)

Expand Down Expand Up @@ -819,7 +822,11 @@ def wrap_pretraining_dataset(
else:
encode = functools.partial(encode_pretraining, tokenizer, max_tokens)

dataset = dataset.shuffle(seed=seed, buffer_size=buffer_size)
if cfg.shuffle_merged_datasets:
dataset = dataset.shuffle(seed=seed, buffer_size=buffer_size)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unsure if this is intended to shuffle the pre-training dataset (it is a single dataset) within the buffer size?

else:
LOG.debug("NOT shuffling merged pretraining datasets")

dataset = dataset.map(
encode,
batched=True,
Expand Down
Loading