some models fail to prepare tokens: No chat template

[davidkoski]

Model loaded -> id("mlx-community/phi-2-hf-4bit-mlx")
Error: chatTemplate("No chat template was specified")

For models that have a chat template this is fine, but for those that do not:

• I don't see a way to probe for the presence of a chat template
• nor a way to provide a default

I think we need to do the config-mutate path and inject a template where needed.

[awni]

Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers?

[davidkoski]

Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers?

No, nothing like that: https://github.com/huggingface/swift-transformers/blob/main/Sources/Tokenizers/Tokenizer.swift#L359

Perhaps there should be? @pcuenca @maiqingqiang

[johnmai-dev]

Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers?

No, nothing like that: https://github.com/huggingface/swift-transformers/blob/main/Sources/Tokenizers/Tokenizer.swift#L359

Perhaps there should be? @pcuenca @maiqingqiang

I agree. Similar to https://github.com/huggingface/transformers/blob/33868a057c02f0368ba63bd1edb746be38fe3d90/src/transformers/tokenization_utils_base.py#L1628-L1629

Alternatively, it could capture throw TokenizerError.chatTemplate and use tokenizer.encode instead.

[davidkoski]

Per above I added a workaround -- it will try to apply the template, but if that throws an error it will fall back to a built in template and tokenization.