A platform for training large language model based multi-turns chatbots.
Developed based on FastChat, trl, trlx, Huggingface transformers.
- Support Llama, Baichuan, Qwen seriels models.
- Characters awared templates.
- Long sequence length training with FlashAttention2.
- A stacked dataset class to support long chats training.
- Ghost Attention (GAtt) in templates.
pip3 install --no-cache-dir -i https://mirrors.aliyun.com/pypi/simple/ --upgrade -e ".[data]"
- Process data
python -m llm.data.process_chats --dataset PIPPA