Is Llama 2 compatible for finetuning using lora?

[palash04]

I am trying to finetune llama2 using lora.

Following are the changes I made in yaml

model:
  name: hf_causal_lm
  pretrained_model_name_or_path: meta-llama/Llama-2-7b-hf
  init_device: mixed
  pretrained: true
  tie_word_embeddings: true
  use_auth_token: true 

# LORA
lora:
  args:
    r: 16
    lora_alpha: 32
    lora_dropout: 0.05
    target_modules: ['Wqkv']

# Tokenizer
tokenizer:
  name: meta-llama/Llama-2-7b-hf
  kwargs:
    model_max_length: ${max_seq_len}

Getting following error:
You are using a model of type llama to instantiate a model of type mpt. This is not supported for all configurations of models and can yield errors.

[creatorrr]

Not working for me :(

[creatorrr]

@danbider @dakinggg any pointers on how I can take a crack at implementing this?

[palash04]

@Creatorr, this is being fixed here - 435

[dakinggg]

Closed by #886, this should work now :)