Skip to content

Latest commit

 

History

History
63 lines (32 loc) · 4.15 KB

README.md

File metadata and controls

63 lines (32 loc) · 4.15 KB
Raw

LLM-Continual-Learning-Papers

Awesome License: MIT img

Must-read Papers on Large Language Model (LLM) Continual Learning

  1. Towards Continual Knowledge Learning of Language Models

    Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, Minjoon Seo. [abs]. ICLR 2022.

  2. Continual Pre-Training Mitigates Forgetting in Language and Vision

    Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Davide Bacci. [abs]. Preprint 2022.05.

  3. Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

    Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren. [abs]. NAACL 2022

  4. Continual Training of Language Models for Few-Shot Learning

    Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, Bing Liu. [abs]. EMNLP 2022.

  5. Continual Pre-training of Language Models

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, Bing Liu. [abs]. ICLR 2023.

  6. Progressive Prompts: Continual Learning for Language Models

    Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Amjad Almahairi. [abs]. ICLR 2023.

  7. A Unified Continual Learning Framework with General Parameter-Efficient Tuning

    Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, Jian Zhang. [abs]. ICCV 2023.

  8. Semiparametric Language Models Are Scalable Continual Learners

    Guangyue Peng, Tao Ge, Si-Qing Chen, Furu Wei, Houfeng Wang. [abs]. Preprint 2023.02.

  9. Continual Pre-Training of Large Language Models: How to (re)warm your model?

    Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, Timothée Lesort. [abs]. ICML 2023 Workshop.

  10. ConPET: Continual Parameter-Efficient Tuning for Large Language Models

    Chenyang Song, Xu Han, Zheni Zeng, Kuai Li, Chen Chen, Zhiyuan Liu, Maosong Sun, Tao Yang. [abs]. Preprint 2023.09.

  11. TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models

    Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xuanjing Huang. [abs]. Preprint 2023.10.

  12. A Study of Continual Learning Under Language Shift

    Evangelia Gogoulou, Timothée Lesort, Magnus Boman, Joakim Nivre. [abs]. Preprint 2023.11.

  13. Scalable Language Model with Generalized Continual Learning

    ICLR 2024 Conference Submission1284 Authors. [openreview]. Preprint 2023.

  14. Efficient Continual Pre-training for Building Domain Specific Large Language Models

    ICLR 2024 Conference Submission4091 Authors. [openreview]. Preprint 2023.