Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable flex attention support #2255

Draft
wants to merge 50 commits into
base: main
Choose a base branch
from
Draft

Enable flex attention support #2255

wants to merge 50 commits into from

Conversation

bursteratom
Copy link
Collaborator

Enable flex attention support

bursteratom and others added 15 commits January 5, 2025 23:27
@bursteratom
config validation for flex attention
61ad375
@bursteratom
flex attention support
bcd9ad4
@bursteratom
llama test
543daaf
@SalmanMohammadi @bursteratom
Fixing OSX installation (#2231)
0b47281 
* bumping version, removing non-osx compatible deps

* updating pylintrc

* fixing linters

* reverting changes
@winglian @bursteratom
Merge group queue (#2248)
2346f21 
* add support for merge groups

* also lint merge groups
@winglian @bursteratom
add hf cache caching for GHA (#2247)
9c9ac1c 
* add hf cache caching for GHA

* use modal volume to cache hf data

* make sure to update the cache as we add new fixtures in conftest
@NanoCode012 @bursteratom
fix: mistral nemo does not recognize token_type_ids in forward (#2233)
fac059a
@winglian @bursteratom
update modal version for ci (#2242)
e0d4b88
@v-dicicco @bursteratom
Use SequentialSampler if curriculum_sampling is enabled with sample_p…
da97a21 
…acking (#2235)
@winglian @bursteratom
update upstream HF deps (#2239)
b7d27bd 
* bump axolotl contribs for upstream main conflicts:

* bump datasets, tokenizer, trl

* remove log workarounds in trl

* bump lm-eval

* remove unsloth_ import from critical path

* remove llama fa2 from conftest

* unsloth breaks with latest upstream
@NanoCode012 @bursteratom
feat: add support for data_files in pretraining (#2238)
5eae134
@winglian @bursteratom
rename liger test so it properly runs in ci (#2246)
bd62d6e
@winglian @bursteratom
use 2.5.1 docker images as latest tag as it seems stable (#2198)
888cd94
@bursteratom
llama test
f99cae0
@bursteratom
Merge branch 'main' into flx_attn_support
2319ac7
@bursteratom bursteratom marked this pull request as draft January 13, 2025 14:43
@bursteratom bursteratom marked this pull request as ready for review January 13, 2025 15:27
@bursteratom bursteratom marked this pull request as draft January 16, 2025 21:41
@bursteratom bursteratom marked this pull request as ready for review January 28, 2025 19:21
@bursteratom bursteratom marked this pull request as draft January 28, 2025 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@bursteratom @SalmanMohammadi @winglian @NanoCode012 @v-dicicco