[Usability]: 运行LLaMa时报错'Config' object has no attribute 'define_bool_state'

[YefanWu]

Issue Type

Usability

Modules Involved

Documentation/Tutorial/Example

Have you reproduced the bug with SPU HEAD?

Yes

Have you searched existing issues?

Yes

SPU Version

spu 0.9.0

OS Platform and Distribution

Linux Ubuntu 20.04LTSC

Python Version

3.8.5

Compiler Version

GCC 11.2

Current Behavior?

按照CONTRIBUTING.md#build和issue#393，从源码编译spu为python包，然后按照这里运行LLaMa的spu推理报错。

一个终端在~/spu/下执行

bazel build //... -c opt
python3 setup.py bdist_wheel
pip3 install dist/*.whl

并拉起节点

bazel run -c opt //examples/python/utils:nodectl -- --config `pwd`/examples/python/ml/flax_llama7b/3pc.json up

然后另一个终端执行

cd examples/python/ml/flax_llama7b
python3 flax_llama7b.py --model_path ./open_llama_7b_EasyLM.msgpack --config ./3pc.json 
#文档中为 python3 flax_llama7b_split.py，可能是typo，也可能是我理解有误

P.S. EasyLM的作者在最近更新了convert_hf_to_easylm.py脚本，spu的文档中提到的需要修改的代码字段

    parser.add_argument(
        "--streaming",
        action="store_true",
        default=False,
        help="whether is model weight saved stream format",
    )

已经不存在了。

Standalone code to reproduce the issue

python3 flax_llama7b.py --model_path ./open_llama_7b_EasyLM.msgpack --config ./3pc.json

Relevant log output

python3 flax_llama7b.py --model_path ./open_llama_7b_EasyLM.msgpack --config ./3pc.json

Traceback (most recent call last):
  File "/home/user/spu/examples/python/ml/flax_llama7b/flax_llama7b.py", line 25, in <module>
    import flax.linen as nn
  File "/home/user/.local/lib/python3.9/site-packages/flax/__init__.py", line 18, in <module>
    from .configurations import (
  File "/home/user/.local/lib/python3.9/site-packages/flax/configurations.py", line 92, in <module>
    flax_filter_frames = define_bool_state(
  File "/home/user/.local/lib/python3.9/site-packages/flax/configurations.py", line 42, in define_bool_state
    return jax_config.define_bool_state('flax_' + name, default, help)
AttributeError: 'Config' object has no attribute 'define_bool_state'

[anakinxc]

@Ye-D can you take a look? Thanks

[Ye-D]

it seems that the flax or jax versions are incompatible with EasyLM.
Please check your environment following: https://github.com/young-geng/EasyLM/blob/main/scripts/gpu_environment.yml.
Note, you may first run the llama in plaintext compatible with spu, and then run secure llama.

[YefanWu]

@Ye-D Thanks for your reply. At this point, there are still some additional questions confusing me.

• I have little idea how to run the plaintext inference with spu. It sounds like running it directly without dependencies.
• After the dependencies' versions have been changed, should I re-build & re-produce spu for Python? It may require many attempts to ensure a compatible environment and the build process is time-consuming.

[Ye-D]

First, run the mode in plaintext means you do not need spu, just to test your python environment to satisfy the EasyLM requirements.

Secondly, yes, you should re-build spu.

[YefanWu]

在conda创建的虚拟环境里，似乎解决了依赖问题。
还是有些问题。运行结果如下：

python3 flax_llama7b.py --model_path ./open_llama_7b_EasyLM.msgpack --config ./3pc.json

terminal echo:
Traceback (most recent call last):
File "/home/user/spu/examples/python/ml/flax_llama7b/flax_llama7b.py", line 55, in
model_path = parser.model_path
AttributeError: 'ArgumentParser' object has no attribute 'model_path'

[Ye-D]

--model_path should be assigned the dir-to-flax-llama7b-EasyLM, not the .msgpack.

[YefanWu]

不好意思，我可能对教程前面的操作的理解有一些问题。
按照convert_hf_to_easyml.py的使用方式，模型被转换后应该是一个path-to-flax-llama7b-EasyLM.msgpack文件

python convert_hf_to_easylm.py  \
   --checkpoint_dir     path-to-flax-llama7b-dir    \
   --output_file path-to-flax-llama7b-EasyLM.msgpack  \
   --model_size 7b \
   --streaming false

如果只有模型的文件夹路径，那如何找到模型的文件本身呢？这里.msgpack文件的命名是固定的？

[Ye-D]

模型命名为flax_model.msgpack，放在一个目录里，path指向这个目录就行

[YefanWu]

将flax_llama7b.py中的model_path = parser.model_path改为model_path = args.model_path似乎能够解决问题。然后我得到了以下的输出。请问这里的脚本的功能是从huggingface还是从模型文件获取tokenizer？

报错如下：
/root/miniconda3/envs/EasyLM/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1743: FutureWarning: Calling LlamaTokenizer.from_pretrained() with the path to a single file or url is deprecated and won't be possible anymore in v5. Use a model identifier or the path to a directory instead.
warnings.warn(
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Traceback (most recent call last):
File "/home/admin/dev/examples/python/ml/flax_llama7b/flax_llama7b.py", line 57, in
tokenizer = LlamaTokenizer.from_pretrained(model_path)
File "/root/miniconda3/envs/EasyLM/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1841, in from_pretrained
return cls._from_pretrained(
File "/root/miniconda3/envs/EasyLM/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2004, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/root/miniconda3/envs/EasyLM/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 144, in init
self.sp_model.Load(vocab_file)
File "/root/miniconda3/envs/EasyLM/lib/python3.10/site-packages/sentencepiece/init.py", line 961, in Load
return self.LoadFromFile(model_file)
File "/root/miniconda3/envs/EasyLM/lib/python3.10/site-packages/sentencepiece/init.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: could not parse ModelProto from /home/admin/dev/flax_model.msgpack

[YefanWu]

好吧，我理解有误。我没有意识到转换后的.msgpack文件只是llama模型文件的一部分。.msgpack文件需要被移动到模型文件夹，才能找到tokenizer.model。

可以关掉这个issue了。
@anakinxc