Reproduce the result on llama3.

[SummerFall1819]

Sorry for another silly question. I am aware that the was an warning on your original demo.ipynb saying:

/proj/vondrick4/hc3295/selfie-dist/selfie/generate_wrappers.py:733: UserWarning: You are calling transformers.generation.generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('cuda') before running `.generate()`.

on cell In[4]. I wonder how that will affect the results.

So similar to issue #9. I couldn't reproduce a reasonable result. But I didn't change the InterpretationPrompt as issue #9 did. I changed the tokens_to_interpret. But the result seems very weird. Would you mind checking my modification and pointing a way for me?

I truly appreciate your exciting work, and am looking forward to your generous help.

Raw Code Blocks

[SummerFall1819]

Forget to mention, I am using llama-3-8b-instruct as the model. I wonder whether this is because the model is too weak to follow the instruction. Please let me know.

[tonychenxyz]

Hi! Sorry for the inconvenience.

1. I think the warning won't affect result.
2. The interpretation prompt in the notebook is with llama 2 instruct template (user instructions marked with [INST] [/INST]). llama 3 use different formats. You need to modify the interpretation prompt accordingly: https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/
3. Can you try to see if you can replicate results with llama2? I tested SelfIE on llama3, and for some reasons it doesn't work as well on llama3. I often get rejections. I suspected this is because of llama3's safety mechanisms and I am still looking into it.

[SummerFall1819]

Hi! Sorry for the inconvenience.

1. I think the warning won't affect result.
2. The interpretation prompt in the notebook is with llama 2 instruct template (user instructions marked with [INST] [/INST]). llama 3 use different formats. You need to modify the interpretation prompt accordingly: https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/
3. Can you try to see if you can replicate results with llama2? I tested SelfIE on llama3, and for some reasons it doesn't work as well on llama3. I often get rejections. I suspected this is because of llama3's safety mechanisms and I am still looking into it.

OK! Thanks for pointing that out! Unfortunately, there are some bad news.
So I tried to replace the token from llama2-style to llama3-style, and here is my raw output.

(I do add padding_side='left' when initializing tokenizer, but the warning persists.)

The result is that, the five blanks were never replaced, which actually is the result described in issue #9. As he also found there seems no difference for replacing 0,0,0,0,0 to _ _ _ _ _.

I'm not sure if the problem occurs because of llama3 architecture or something else. I use llama2-style token and the problem seems to be the same.

(I understand there should be some unreasonable sequence, but all of the results seems have [INST]_ _ _ _ _[/INST], which cast doubt whether the token was really replaced with the embedding.)

[SummerFall1819]

And by using llama2-7b-hf model, I failed to replicate reasonable results, Sorry.

[tonychenxyz]

Can you try llama2-7b-chat-hf? I just tried with chat model and here's what I got

Another thing to try is pip uninstall selfie, and just import from the project directory.

[SummerFall1819]

That's weird... Using llama2-7b-chat-hf seems not replicate the results either. Maybe that's issue related to dependency?

I established the environment using miniconda, and here is the result by running pip list.

Package            Version
------------------ -----------
accelerate         1.1.1
asttokens          3.0.0
Brotli             1.0.9
certifi            2024.8.30
charset-normalizer 3.3.2
comm               0.2.2
debugpy            1.8.9
decorator          5.1.1
exceptiongroup     1.2.2
executing          2.1.0
filelock           3.16.1
fsspec             2024.10.0
gmpy2              2.1.2
huggingface-hub    0.24.5
idna               3.7
importlib_metadata 8.5.0
ipykernel          6.29.5
ipython            8.30.0
jedi               0.19.2
Jinja2             3.1.4
jupyter_client     8.6.3
jupyter_core       5.7.2
MarkupSafe         2.1.3
matplotlib-inline  0.1.7
mkl_fft            1.3.11
mkl_random         1.2.8
mkl-service        2.4.0
mpmath             1.3.0
nest_asyncio       1.6.0
networkx           3.3
numpy              1.26.4
packaging          24.2
pandas             2.2.0
parso              0.8.4
pexpect            4.9.0
pickleshare        0.7.5
pillow             11.0.0
pip                24.2
platformdirs       4.3.6
prompt_toolkit     3.0.48
psutil             6.1.0
ptyprocess         0.7.0
pure_eval          0.2.3
Pygments           2.18.0
PySocks            1.7.1
python-dateutil    2.9.0.post0
pytz               2024.2
PyYAML             6.0.2
pyzmq              26.2.0
regex              2024.11.6
requests           2.32.3
safetensors        0.4.5
setuptools         75.1.0
six                1.17.0
stack-data         0.6.2
sympy              1.13.2
tokenizers         0.14.1
torch              2.1.1
torchaudio         2.1.1
torchvision        0.16.1
tornado            6.4.2
tqdm               4.67.1
traitlets          5.14.3
transformers       4.34.0
triton             2.1.0
typing_extensions  4.12.2
tzdata             2024.2
urllib3            2.2.3
wcwidth            0.2.13
wheel              0.44.0
zipp               3.21.0

For further reference:

• python==3.11.10
• cuda_version==12.1

(I should also mention that running pip install -e .) doesn't work well for me, I download related packages manually.

[tonychenxyz]

Hi! Here's the env I have

accelerate==1.2.0
anyio==4.7.0
argcomplete==3.2.3
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==3.0.0
async-lru==2.0.4
attrs==24.2.0
babel==2.16.0
beautifulsoup4==4.12.3
bleach==6.2.0
build==1.2.1
CacheControl==0.14.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
cleo==2.1.0
comm==0.2.2
crashtest==0.4.1
debugpy==1.8.9
decorator==5.1.1
defusedxml==0.7.1
distlib==0.3.8
dulwich==0.21.7
exceptiongroup==1.2.2
executing==2.1.0
fastjsonschema==2.21.1
filelock==3.13.1
fqdn==1.5.1
fsspec==2024.2.0
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
huggingface-hub==0.26.5
idna==3.10
importlib_metadata==7.1.0
installer==0.7.0
ipykernel==6.29.5
ipython==8.30.0
ipywidgets==8.1.5
isoduration==20.11.0
jaraco.classes==3.4.0
jedi==0.19.2
Jinja2==3.1.4
json5==0.10.0
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter==1.1.1
jupyter_client==8.6.3
jupyter-console==6.6.3
jupyter_core==5.7.2
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterlab==4.3.2
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.13
keyring==24.3.1
MarkupSafe==3.0.2
matplotlib-inline==0.1.7
mistune==3.0.2
mpmath==1.3.0
msgpack==1.0.8
nbclient==0.10.1
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.2.1
notebook==7.3.1
notebook_shim==0.2.4
numpy==1.26.3
nvidia-cublas-cu11==11.11.3.6
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu11==11.8.87
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu11==9.1.0.70
nvidia-cufft-cu11==10.9.0.58
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu11==10.3.0.86
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu11==11.4.1.48
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu11==11.7.5.86
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu11==2.21.5
nvidia-nvjitlink-cu12==12.4.99
nvidia-nvtx-cu11==11.8.86
nvidia-nvtx-cu12==12.1.105
overrides==7.7.0
packaging==24.2
pandas==2.2.3
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
pillow==10.2.0
pip==24.2
pipx==1.5.0
pkginfo==1.10.0
platformdirs==4.3.6
poetry==1.8.2
poetry-core==1.9.0
poetry-plugin-export==1.7.1
prometheus_client==0.21.1
prompt_toolkit==3.0.48
psutil==6.1.0
ptyprocess==0.7.0
pure_eval==0.2.3
pycparser==2.22
Pygments==2.18.0
pyproject_hooks==1.0.0
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytube==15.0.0
pytz==2024.2
PyYAML==6.0.2
pyzmq==26.2.0
referencing==0.35.1
regex==2024.11.6
requests==2.32.3
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.22.3
safetensors==0.4.5
Send2Trash==1.8.3
setuptools==75.6.0
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
soupsieve==2.6
stack-data==0.6.3
sympy==1.13.1
terminado==0.18.1
tinycss2==1.4.0
tokenizers==0.14.1
tomli==2.2.1
torch==2.5.1+cu118
torchaudio==2.5.1+cu118
torchvision==0.20.1+cu118
tornado==6.4.2
tqdm==4.67.1
traitlets==5.14.3
transformers==4.34.0
triton==3.1.0
trove-classifiers==2024.3.25
types-python-dateutil==2.9.0.20241206
typing_extensions==4.12.2
tzdata==2024.2
uri-template==1.3.0
urllib3==2.2.3
userpath==1.9.2
virtualenv==20.25.1
wcwidth==0.2.13
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0
wheel==0.44.0
widgetsnbextension==4.0.13

[tonychenxyz]

Can you also share the full code you are running?

[SummerFall1819]

Can you also share the full code you are running?

sorry for not being active these days, I'm using the updated code with the only modification on tokens_to_interpret. I'll try to use your environment and try again. Thank you for you hard work!