Skip to content

congchan/llm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LLM(Large Language Modeling)

A platform for training large language model based multi-turns chatbots.

Developed based on FastChat, trl, trlx, Huggingface transformers.

Features

  • Support Llama, Baichuan, Qwen seriels models.
  • Characters awared templates.
  • Long sequence length training with FlashAttention2.
  • A stacked dataset class to support long chats training.
  • Ghost Attention (GAtt) in templates.

Installation

pip3 install --no-cache-dir -i https://mirrors.aliyun.com/pypi/simple/ --upgrade -e ".[data]"

Pre-training

Fine-tuning

  1. Process data
    python -m llm.data.process_chats --dataset PIPPA

Alignment

About

Codebase and experiments of LLM(Large Language Modeling)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published