diff --git a/README.md b/README.md index 75e7faa642..d4e48191e6 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,13 @@ GitHub License tests Releases +
+ contributors GitHub Repo stars -

-

+
+ discord + twitter +
tests-nightly multigpu-semi-weekly tests

@@ -42,7 +46,8 @@ Features: - [Axolotl](#axolotl) - [Table of Contents](#table-of-contents) - [Quickstart ⚡](#quickstart-) - - [Usage](#usage) + - [Edge Builds](#edge-builds-) + - [Axolotl CLI Usage](#axolotl-cli-usage) - [Badge ❤🏷️](#badge-️) - [Contributing 🤝](#contributing-) - [Sponsors 🤝❤](#sponsors-) @@ -107,58 +112,49 @@ Get started with Axolotl in just a few steps! This quickstart guide will walk yo **Requirements**: *Nvidia* GPU (Ampere architecture or newer for `bf16` and Flash Attention) or *AMD* GPU, Python >=3.10 and PyTorch >=2.3.1. ```bash -git clone https://github.com/axolotl-ai-cloud/axolotl -cd axolotl +pip3 install axolotl[flash-attn,deepspeed] -pip3 install packaging ninja -pip3 install -e '.[flash-attn,deepspeed]' +# download examples and optionally deepspeed configs to the local path +axolotl fetch examples +axolotl fetch deepspeed_configs # OPTIONAL + +# finetune using lora +axolotl train examples/llama-3/lora-1b.yml ``` -### Usage -```bash -# preprocess datasets - optional but recommended -CUDA_VISIBLE_DEVICES="0" python -m axolotl.cli.preprocess examples/openllama-3b/lora.yml +### Edge Builds 🏎️ -# finetune lora -accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml +If you're looking for the latest features and updates between releases, you'll need to install +from source. -# inference -accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \ - --lora_model_dir="./outputs/lora-out" - -# gradio -accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \ - --lora_model_dir="./outputs/lora-out" --gradio - -# remote yaml files - the yaml config can be hosted on a public URL -# Note: the yaml config must directly link to the **raw** yaml -accelerate launch -m axolotl.cli.train https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/openllama-3b/lora.yml +```bash +git clone https://github.com/axolotl-ai-cloud/axolotl.git +cd axolotl +pip3 install packaging ninja +pip3 install -e '.[flash-attn,deepspeed]' ``` -### Axolotl CLI - -If you've installed this package using `pip` from source, we now support a new, more -streamlined CLI using [click](https://click.palletsprojects.com/en/stable/). Rewriting -the above commands: +### Axolotl CLI Usage +We now support a new, more streamlined CLI using [click](https://click.palletsprojects.com/en/stable/). ```bash # preprocess datasets - optional but recommended -CUDA_VISIBLE_DEVICES="0" axolotl preprocess examples/openllama-3b/lora.yml +CUDA_VISIBLE_DEVICES="0" axolotl preprocess examples/llama-3/lora-1b.yml # finetune lora -axolotl train examples/openllama-3b/lora.yml +axolotl train examples/llama-3/lora-1b.yml # inference -axolotl inference examples/openllama-3b/lora.yml \ +axolotl inference examples/llama-3/lora-1b.yml \ --lora-model-dir="./outputs/lora-out" # gradio -axolotl inference examples/openllama-3b/lora.yml \ +axolotl inference examples/llama-3/lora-1b.yml \ --lora-model-dir="./outputs/lora-out" --gradio # remote yaml files - the yaml config can be hosted on a public URL # Note: the yaml config must directly link to the **raw** yaml -axolotl train https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/openllama-3b/lora.yml +axolotl train https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/llama-3/lora-1b.yml ``` We've also added a new command for fetching `examples` and `deepspeed_configs` to your @@ -175,6 +171,36 @@ axolotl fetch deepspeed_configs axolotl fetch examples --dest path/to/folder ``` +### Legacy Usage +
+ +Click to Expand + +While the Axolotl CLI is the preferred method for interacting with axolotl, we +still support the legacy `-m axolotl.cli.*` usage. + +```bash +# preprocess datasets - optional but recommended +CUDA_VISIBLE_DEVICES="0" python -m axolotl.cli.preprocess examples/llama-3/lora-1b.yml + +# finetune lora +accelerate launch -m axolotl.cli.train examples/llama-3/lora-1b.yml + +# inference +accelerate launch -m axolotl.cli.inference examples/llama-3/lora-1b.yml \ + --lora_model_dir="./outputs/lora-out" + +# gradio +accelerate launch -m axolotl.cli.inference examples/llama-3/lora-1b.yml \ + --lora_model_dir="./outputs/lora-out" --gradio + +# remote yaml files - the yaml config can be hosted on a public URL +# Note: the yaml config must directly link to the **raw** yaml +accelerate launch -m axolotl.cli.train https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/llama-3/lora-1b.yml +``` + +
+ ## Badge ❤🏷️ Building something cool with Axolotl? Consider adding a badge to your model card. diff --git a/examples/llama-3/lora-1b.yml b/examples/llama-3/lora-1b.yml new file mode 100644 index 0000000000..bdda2ed75e --- /dev/null +++ b/examples/llama-3/lora-1b.yml @@ -0,0 +1,74 @@ +base_model: NousResearch/Llama-3.2-1B + +load_in_8bit: false +load_in_4bit: false +strict: false + +datasets: + - path: teknium/GPT4-LLM-Cleaned + type: alpaca +dataset_prepared_path: last_run_prepared +val_set_size: 0.1 +output_dir: ./outputs/lora-out + +adapter: lora +lora_model_dir: + +sequence_len: 2048 +sample_packing: true +eval_sample_packing: true +pad_to_sequence_len: true + +lora_r: 16 +lora_alpha: 32 +lora_dropout: 0.05 +lora_fan_in_fan_out: +lora_target_modules: + - gate_proj + - down_proj + - up_proj + - q_proj + - v_proj + - k_proj + - o_proj + +wandb_project: +wandb_entity: +wandb_watch: +wandb_name: +wandb_log_model: + +gradient_accumulation_steps: 2 +micro_batch_size: 2 +num_epochs: 1 +optimizer: adamw_8bit +lr_scheduler: cosine +learning_rate: 0.0002 + +train_on_inputs: false +group_by_length: false +bf16: auto +fp16: +tf32: false + +gradient_checkpointing: true +early_stopping_patience: +resume_from_checkpoint: +local_rank: +logging_steps: 1 +xformers_attention: +flash_attention: true + +loss_watchdog_threshold: 5.0 +loss_watchdog_patience: 3 + +warmup_steps: 10 +evals_per_epoch: 4 +saves_per_epoch: 1 +debug: +deepspeed: +weight_decay: 0.0 +fsdp: +fsdp_config: +special_tokens: + pad_token: "<|end_of_text|>" diff --git a/examples/llama-3/qlora-1b.yml b/examples/llama-3/qlora-1b.yml index fdfe4aa7c8..be40db846e 100644 --- a/examples/llama-3/qlora-1b.yml +++ b/examples/llama-3/qlora-1b.yml @@ -1,4 +1,4 @@ -base_model: meta-llama/Llama-3.2-1B +base_model: NousResearch/Llama-3.2-1B load_in_8bit: false load_in_4bit: true @@ -22,7 +22,6 @@ pad_to_sequence_len: true lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 -lora_target_linear: true lora_fan_in_fan_out: lora_target_modules: - gate_proj diff --git a/src/axolotl/__init__.py b/src/axolotl/__init__.py index b4f1d06f7b..9a31745983 100644 --- a/src/axolotl/__init__.py +++ b/src/axolotl/__init__.py @@ -1,3 +1,3 @@ """Axolotl - Train and fine-tune large language models""" -__version__ = "0.5.3.dev0" +__version__ = "0.6.0"