Transformers version

[SummerFall1819]

Hello, I'm a follower of your work, what you did is amazing and I want to dive into further by first reproduce your work. However, I happened to meet some problems here, something about the version of transformer.

As I am running the first cell in demo.ipynb, an Error occured.

ValueError: `rope_scaling` must be a dictionary with with two fields, `type` and `factor`, got {'factor': 8.0, 
'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

After carefully examined my problem, the problem could be fixed by upgrading the version of transformers. But that would result in another error(let's leave out this one). Since in your readme you tested your code with transformers==4.34.0. I don't know how to deal with this problem.

I am new to this area, so sorry if this question is silly.

Environment

python==3.11.10
transformers==4.34.0
torch==2.1.1+cu121
torchaudio==2.1.1+cu121
torchvision==0.16.1+cu121
pandas==2.2.3
numpy==2.1.3

Model type: Llama-3.1-8B-Instruct (Since in your readme you claimed the library supports all LLaMA models.)

Origin Error Output

Thank you for your time and patience.

[tonychenxyz]

Hi! Thanks for your interest and sorry for the error.
Does llama2 or llama 3 (instead of 3.1) give this error? I was able to run llama 3 without error but I tried it a while ago and there wans't llama 3.1 at the time.
Thanks!

[SummerFall1819]

Hi! Thanks for your interest and sorry for the error. Does llama2 or llama 3 (instead of 3.1) give this error? I was able to run llama 3 without error but I tried it a while ago and there wans't llama 3.1 at the time. Thanks!

Thank you for your guidance! I switched to llama3 and it worked fine. You are doing a great job! Do you have any plans to support newer llamas since Meta just released 3.3 literally few days ago?

[tonychenxyz]

Thanks for trying! Support for llama 3 is on the todo list, but before that I was planning (and stuck on) finding a good way to get interpretations out of llama 3. How's your experience with outputs from llama 3? The last time I tried it I kept getting refusal from model maybe due to its safety mechanism.

[sev777]

For llama3, attention_mask = model.model._prepare_decoder_attention_mask( attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length ) in llama_forward_wrappers.py couldn't work.