[BUG/Help] <IndexError: Out of range: piece id is out of range.>

[Data2Me]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

File "/root/.cache/huggingface/modules/transformers_modules/THUDM/chatglm-6b/aa51e62ddc9c9f334858b0af44cf59b05c70148a/tokenization_chatglm.py", line 268, in _decode
return self.sp_tokenizer.decode(token_ids)
File "/root/.cache/huggingface/modules/transformers_modules/THUDM/chatglm-6b/aa51e62ddc9c9f334858b0af44cf59b05c70148a/tokenization_chatglm.py", line 117, in decode
text = self._get_text_tokenizer().decode(ids)
File "/root/.cache/huggingface/modules/transformers_modules/THUDM/chatglm-6b/aa51e62ddc9c9f334858b0af44cf59b05c70148a/tokenization_chatglm.py", line 29, in decode
return self.sp.DecodeIds(ids)
File "/opt/conda/lib/python3.10/site-packages/sentencepiece/init.py", line 837, in DecodeIds
return self.Decode(input=input, out_type=out_type, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/sentencepiece/init.py", line 780, in Decode
return self._DecodeIds(input)
File "/opt/conda/lib/python3.10/site-packages/sentencepiece/init.py", line 337, in _DecodeIds
return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
IndexError: Out of range: piece id is out of range.

Expected Behavior

No response

Steps To Reproduce

..

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response

[duzx16]

Duplicate of #438

[yecphaha]

参考 https://huggingface.co/THUDM/chatglm3-6b/commit/ea563876364622a0a5c24e6b71db0b93a9861ba0#d2h-069285
在 tokenization_chatglm.py 里新增两行代码