Skip to content

Commit

Permalink
Update TUTORIAL.md
Browse files Browse the repository at this point in the history
fix indentation problem
  • Loading branch information
sdonoso authored Feb 7, 2024
1 parent 105f766 commit d597478
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions TUTORIAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,8 +371,7 @@ model:
target_modules:
- q_proj
- k_proj
target_modules:
- 'Wqkv'
- 'Wqkv'
```
- For efficiency, The MPT model concatenates the `Q`, `K`, and `V` matrices in each attention block into a single `Wqkv` matrix that is three times wider. Currently, LoRA supports a low-rank approximation to this `Wqkv` matrix.
- When evaluating with PEFT / LoRA separated weight, just set `pretrained_lora_id_or_path` in `model`(Find an example [here](scripts/eval/yamls/hf_lora_eval.yml#L19)).
Expand Down

0 comments on commit d597478

Please sign in to comment.