Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tiktoken chat format fix #893

Merged
merged 3 commits into from
Jan 30, 2024
Merged

Tiktoken chat format fix #893

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5971009
sys prompt fix
rajammanabrolu Jan 22, 2024
0af09e3
remove eos tokens
rajammanabrolu Jan 30, 2024
990771e
Merge branch 'main' into raj/tiktoken-fixes
rajammanabrolu Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions llmfoundry/tokenizers/tiktoken.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,16 +191,14 @@ def default_chat_template(self):
'{% for message in loop_messages %}'
'{% if loop.index0 == 0 %}'
'{% if system_message != false %}'
"{{ '<|im_start|>system\n' + system_message.strip() + '\n'}}"
"{{ '<|im_start|>system\n' + system_message.strip() + '<|im_end|>\n'}}"
'{% endif %}'
"{{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' }}"
'{% else %}'
"{{ '\n' + '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' }}"
'{% endif %}'
'{% if (add_generation_prompt == true and loop.last) %}'
"{{ '\n' + '<|im_start|>' + 'assistant' + '\n' }}"
"{% elif (message['role'] == 'assistant') %}"
'{{ eos_token }}'
'{% endif %}'
'{% endfor %}')
template = template.replace(
Expand Down
22 changes: 10 additions & 12 deletions tests/tokenizers/test_tiktoken.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
('gpt2', None),
]

DEFAULT_SYSTEM_PROMPT = """<|im_start|>system\nYou are a helpful, respectful and honest assistant. Always answer as helpfully as possible."""
DEFAULT_SYSTEM_PROMPT = """<|im_start|>system\nYou are a helpful, respectful and honest assistant. Always answer as helpfully as possible.<|im_end|>"""

MULTI_TURN_CHAT_ML = [
[{
Expand Down Expand Up @@ -73,34 +73,32 @@

Going outside has benefits include reducing stress and triggering the relaxation response, which can help us not only feel better mentally, but even heal faster from physical ailments.<|im_end|>
<|im_start|>assistant
You should go outside and touch grass.<|im_end|><|endoftext|>""",
"""<|im_start|>system
You are a honest and helpful AI language model. Tell the user the truth, the whole truth, and nothing but the truth.
You should go outside and touch grass.<|im_end|>""", """<|im_start|>system
You are a honest and helpful AI language model. Tell the user the truth, the whole truth, and nothing but the truth.<|im_end|>
<|im_start|>user
Please summarize the goals in this text:

Going outside has benefits include reducing stress and triggering the relaxation response, which can help us not only feel better mentally, but even heal faster from physical ailments.<|im_end|>
<|im_start|>assistant
You should go outside and touch grass.<|im_end|><|endoftext|>"""
You should go outside and touch grass.<|im_end|>"""
]

MULTI_TURN_CHAT_STRING_SYSTEM_PROMPT = [
"""<|im_start|>system
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible.
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible.<|im_end|>
<|im_start|>user
Please summarize the goals in this text:

Going outside has benefits include reducing stress and triggering the relaxation response, which can help us not only feel better mentally, but even heal faster from physical ailments.<|im_end|>
<|im_start|>assistant
You should go outside and touch grass.<|im_end|><|endoftext|>""",
"""<|im_start|>system
You are a honest and helpful AI language model. Tell the user the truth, the whole truth, and nothing but the truth.
You should go outside and touch grass.<|im_end|>""", """<|im_start|>system
You are a honest and helpful AI language model. Tell the user the truth, the whole truth, and nothing but the truth.<|im_end|>
<|im_start|>user
Please summarize the goals in this text:

Going outside has benefits include reducing stress and triggering the relaxation response, which can help us not only feel better mentally, but even heal faster from physical ailments.<|im_end|>
<|im_start|>assistant
You should go outside and touch grass.<|im_end|><|endoftext|>"""
You should go outside and touch grass.<|im_end|>"""
]

MULTI_TURN_GENERATE_CHAT_ML = [[{
Expand All @@ -118,13 +116,13 @@

MULTI_TURN_GENERATE_STRING = [
"""<|im_start|>system
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible.
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible.<|im_end|>
<|im_start|>user
Please summarize the goals in this text:

Going outside has benefits include reducing stress and triggering the relaxation response, which can help us not only feel better mentally, but even heal faster from physical ailments.<|im_end|>
<|im_start|>assistant
You should go outside and touch grass.<|im_end|><|endoftext|>
You should go outside and touch grass.<|im_end|>
<|im_start|>user
What else can I do?<|im_end|>
<|im_start|>assistant
Expand Down
Loading