Phi2 rewrite #1058
Phi2 rewrite #1058
Conversation
|
Looks good to me. I would remove the code you commented out. You can come back to the PR if you need to look up what changed. Also, what are the possibilities of using sample packing with Phi2? |
|
@casper-hansen will clean up the commented out code As far as sample packing, it should be pretty straightforward. I started working on a fix for the previous implementation #877 but I may simply start over. You had mentioned last year figuring out a way to manage sample packing across all the architectures in a more manageable way. I'm happy to take a stab at it if you have a prrof of concept or anything. |
I had a branch going but didn't get to test and further implement it as I got busy with other stuff. The concept is to have one implementation that can be managed more easily managed in one module. https://github.com/OpenAccess-AI-Collective/axolotl/tree/refactor-flash-attention |
| @@ -843,7 +844,14 @@ def build_collator(self, training_args: AxolotlTrainingArguments, **kwargs): | |||
| if self.cfg.model_config_type == "mamba": | |||
| return MambaDataCollator(tokenizer=self.tokenizer) | |||
|
|
|||
| return BatchSamplerDataCollatorForSeq2Seq( | |||
| if training_args.sample_packing: | |||
There was a problem hiding this comment.
I would recommend maybe consolidate the class?
data_collator = BatchSamplerDataCollatorForSeq2Seq if training_args.sample_packing else DataCollatorForSeq2SeqThere was a problem hiding this comment.
my IDE doesn't like that 😭
|
alright, looks good on a single 4090 https://api.wandb.ai/links/oaaic/51qvcv4z |
|
Hi, do I need to change any configuration options or just use the default ones w/ Phi 2? |
* restore to current phi modeling code from phi-2 * enable gradient checkpointing * don't cast everything to float32 all the time * gradient checkpointing for phi2 ParallelBlock module too * fix enabling flash attn for phi2 * add comment about import * fix phi2 example * fix model type check for tokenizer * revert float32 -> bf16 casting changes * support fused dense flash attn * fix the repo for flash-attn * add package name for subdir pkg * fix the data collator when not using sample packing * install packaging for pytests in ci * also fix setup to not install flash attn fused dense subdir if not extras * split out the fused-dense-lib in extra requires * don't train w group_by_length for phi * update integration test to use phi2 * set max steps and save steps for phi e2e tests * try to workaround ssave issue in ci * skip phi2 e2e test for now
disable most of the upcasting to float32 in favor of bfloat16