Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

log packing ratio progress #1070

Merged
merged 13 commits into from
Apr 4, 2024
Merged

log packing ratio progress #1070

merged 13 commits into from
Apr 4, 2024

Conversation

milocress
Copy link
Contributor

@milocress milocress commented Mar 27, 2024 •
edited
Loading

Logs packing ratio

log-test-ZUxVSC

@milocress milocress requested review from dakinggg and irenedea March 27, 2024 18:25
dakinggg
dakinggg reviewed Mar 27, 2024
View reviewed changes
Copy link
Collaborator

@dakinggg dakinggg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you share a run so I can see the logs produced by this?

@milocress
Copy link
Contributor Author

milocress commented Mar 27, 2024 •
edited
Loading

Could you share a run so I can see the logs produced by this?

log-test-ZUxVSC

dakinggg
dakinggg reviewed Apr 1, 2024
View reviewed changes
llmfoundry/data/packing.py Outdated Show resolved Hide resolved
@milocress milocress requested a review from dakinggg April 2, 2024 21:46
@dakinggg
Copy link
Collaborator

dakinggg commented Apr 2, 2024

@milocress I don't think this quite worked? Maybe it would be simpler to avoid trying to get tqdm to work fully here and just print out the packing ratios (or some subset) tried?

2024-04-02 21:45:51,223: rank0[1257][MainThread]: INFO: llmfoundry.data.packing: Profiling packing ratios
Profiling packing ratios: 0it [00:00, ?it/s]ead]: DEBUG: llmfoundry.data.packing:
Profiling packing ratio 1.0: 0it [00:00, ?it/s]]: DEBUG: llmfoundry.data.packing:
Profiling packing ratio 1.0: 1it [00:00,  3.96it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 2.0: 1it [00:00,  3.96it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 2.0: 2it [00:00,  4.29it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 3.1: 2it [00:00,  4.29it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 3.1: 3it [00:00,  4.53it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 4.1: 3it [00:00,  4.53it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 4.1: 4it [00:01,  3.55it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 5.1: 4it [00:01,  3.55it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 5.1: 5it [00:01,  4.03it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 6.1: 5it [00:01,  4.03it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 6.1: 6it [00:01,  4.40it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 7.2: 6it [00:01,  4.40it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 7.2: 7it [00:01,  4.70it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 8.2: 7it [00:01,  4.70it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 8.2: 8it [00:01,  4.94it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 9.2: 8it [00:01,  4.94it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 9.2: 9it [00:01,  5.11it/s]EBUG: llmfoundry.data.packing:
Profiling packing ratio 10.2: 9it [00:01,  5.11it/s]BUG: llmfoundry.data.packing:
Profiling packing ratio 10.2: 10it [00:02,  5.23it/s]UG: llmfoundry.data.packing:
Profiling packing ratio 11.3: 10it [00:02,  5.23it/s]UG: llmfoundry.data.packing:
Profiling packing ratio 11.3: 11it [00:02,  5.30it/s]UG: llmfoundry.data.packing:
Profiling packing ratio 12.3: 11it [00:02,  5.30it/s]UG: llmfoundry.data.packing:
2024-04-02 21:45:54,241: rank0[1257][MainThread]: DEBUG: llmfoundry.data.packing:
Profiling packing ratio 12.3: 11it [00:03,  3.65it/s]UG: llmfoundry.data.packing:
2024-04-02 21:45:54,241: rank0[1257][MainThread]: DEBUG: llmfoundry.data.packing:

2024-04-02 21:45:54,266: rank0[1257][MainThread]: INFO: llmfoundry.data.finetuning.dataloader: Using packing ratio 11.3

@milocress
Copy link
Contributor Author

@milocress I don't think this quite worked? Maybe it would be simpler to avoid trying to get tqdm to work fully here and just print out the packing ratios (or some subset) tried?

You're right, may have been overthinking it with the progress bar - I'll have it print the packing ratios and the progress percentage and call it a day

@milocress
Copy link
Contributor Author

@dakinggg here is the updated manual run: log-test-kvSMKt

dakinggg
dakinggg approved these changes Apr 3, 2024
View reviewed changes
llmfoundry/data/packing.py Outdated Show resolved Hide resolved
@milocress milocress enabled auto-merge (squash) April 4, 2024 15:16
@milocress milocress merged commit e70e424 into mosaicml:main Apr 4, 2024
9 checks passed
KuuCi pushed a commit that referenced this pull request Apr 18, 2024
@milocress @KuuCi
log packing ratio progress (#1070)
3481b1b 
* log packing ratio progress

* use tqdm to show progress of packing ratio profiling

* fix pbar

* Update packing.py

* added log to search for

* fix

* good old fashioned progress logging

* Update packing.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@irenedea irenedea irenedea left review comments

@dakinggg dakinggg dakinggg approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@milocress @dakinggg @irenedea