README.md

A Survey of LLM Surveys

Large language models (LLMs) are making sweeping advances across many fields of artificial intelligence. As a result, research interest and progress in LLMs have exploded. There are now hundreds of research papers on LLMs published in various conferences or posted to open-access archives every day. Given the significant growth in LLM-related papers, this work compiles surveys on LLMs to provide a comprehensive overview of the field. Most of these surveys have been published or posted in the past few years, so this collection is relatively new. We hope that our compilation can be helpful for people who want to get a quick understanding of the field.

Outline

• General Surveys
• Transformers
• Alignment
• Prompt Learning
  • In-context Learning
  • Chain of Thought
  • Prompt Engineering
  • Reasoning
• Data
• Evaluation
• Societal Issues
• Safety
  • Source Detection
  • Security
• Misinformation
  • Hallucinations
  • Factuality
• Attributes of LLMs
• Efficient LLMs
• Learning Methods for LLMs
• Multimodal LLMs
• Knowledge Based LLMs
  • Retrieval-Augmented LLMs
  • Knowledge Editing
• Extension of LLMs
  • LLMs with Tools
  • LLMs and Interactions
• Long Sequence LLMs
• LLMs Applications
  • Education
  • Law
  • Healthcare
  • Games
  • NLP Tasks
  • Software Engineering
  • Recommender Systems
  • Graphs
  • Other

Survey List

General Surveys

• Large Language Models: A Survey, arXiv 2024.02 [Paper]

• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT, arXiv 2023.03 [Paper]

• A Survey of Large Language Models, arXiv 2023.11 [Paper] [GitHub]

• Challenges and Applications of Large Language Models, arXiv 2023.07 [Paper]

• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond, arXiv 2023.04 [Paper] [GitHub]

• A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage, TechRxiv 2023.07 [Paper] [GitHub]

• A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT, arXiv 2023.05 [Paper]

• A Comprehensive Overview of Large Language Models, arXiv 2023.07 [Paper] [GitHub]

• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, ACM Computing Surveys 2023.01 [Paper]

Transformers

• A survey of transformers, arXiv 2022.10 [Paper]

• Introduction to Transformers: an NLP Perspective, arXiv 2023.11 [Paper] [GitHub]

• Efficient Transformers: A Survey, arXiv 2022.12 [Paper]

• A Practical Survey on Faster and Lighter Transformers, arXiv 2023.07 [Paper]

• Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey, arXiv 2020.12 [Paper]

Alignment

• Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation, arXiv 2023.06 [Paper]

• AI Alignment: A Comprehensive Survey, arXiv 2024.02 [Paper]

• Large Language Model Alignment: A Survey, arXiv 2023.09 [Paper]

• From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models, arXiv 2023.09 [Paper] [GitHub]

• Aligning Large Language Models with Human: A Survey, arXiv 2023.07 [Paper] [GitHub]

• Instruction Tuning for Large Language Models: A Survey, arXiv 2023.08 [Paper]

• A Comprehensive Survey on Instruction Following, arXiv 2024.01 [Paper] [GitHub]

Prompt Learning

In-context Learning

• A Practical Survey on Zero-shot Prompt Design for In-context Learning, ranlp 2023.09 [Paper]

• A Survey on In-context Learning, arXiv 2023.06 [Paper]

Chain of Thought

• A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future, arXiv 2023.10 [Paper] [GitHub]

• Towards Better Chain-of-Thought Prompting Strategies: A Survey, arXiv 2023.10 [Paper]

• Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents, arXiv 2023.11 [Paper] [GitHub]

Prompt Engineering

• Prompting Frameworks for Large Language Models: A Survey, arXiv 2023.11 [Paper] [GitHub]

• Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review, arXiv 2023.10 [Paper]

Reasoning

• Towards Reasoning in Large Language Models: A Survey, arXiv 2022.12 [Paper] [GitHub]

• A Survey of Reasoning with Foundation Models, arXiv 2023.12 [Paper] [GitHub]

Data

• Data Management For Large Language Models: A Survey, arXiv 2023.12 [Paper] [GitHub]

• A Survey on Data Selection for Language Models, arXiv 2024.02 [Paper]

• Datasets for Large Language Models: A Comprehensive Survey, arXiv 2024.02 [Paper] [GitHub]

• Large Language Models for Data Annotation: A Survey, arXiv 2024.02 [Paper] [GitHub]

• A Survey on Data Selection for LLM Instruction Tuning, arXiv 2024.02 [Paper]

• A Survey on Knowledge Distillation of Large Language Models, arXiv 2024.02 [Paper]

Evaluation

• Evaluating Large Language Models: A Comprehensive Survey, arXiv 2023.10 [Paper] [GitHub]

• A Survey on Evaluation of Large Language Models, arXiv 2023.07 [Paper] [GitHub]

• Baby steps in evaluating the capacities of large language models, arXiv 2023.06 [Paper]

Societal Issues

• A Survey on Fairness in Large Language Models, arXiv 2023.08 [Paper]

• Large Language Models as Subpopulation Representative Models: A Review, arXiv 2023.10 [Paper]

• Perception, performance, and detectability of conversational artificial intelligence across 32 university courses, SCI REP-UK 2023.08 [Paper]

• Should chatgpt be biased? challenges and risks of bias in large language models, arXiv 2023.04 [Paper]

• Bias and Fairness in Large Language Models: A Survey, arXiv 2023.09 [Paper] [GitHub]

Safety

Source Detection

• A Survey on Detection of LLMs-Generated Content, arXiv 2023.10 [Paper] [GitHub]

• A Survey on LLM-generated Text Detection: Necessity, Methods, and Future Directions, arXiv 2023.10 [Paper] [GitHub]

• Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated Text, arXiv 2023.09 [Paper]

• The Science of Detecting LLM-Generated Texts, arXiv 2023.02 [Paper]

Security

• Security and Privacy Challenges of Large Language Models: A Survey, arXiv 2024.01 [Paper]

• Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks, arXiv 2023.10 [Paper]

• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly, arXiv 2023.12 [Paper]

• Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks, arXiv 2023.05 [Paper]

• A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation, arXiv 2023.05 [Paper]

Misinformation

Hallucinations

• Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey, arXiv 2023.11 [Paper]

• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv 2023.11 [Paper] [GitHub]

• A Survey of Hallucination in “Large” Foundation Models, arXiv 2023.09 [Paper] [GitHub]

• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models, arXiv 2023.09 [Paper] [GitHub]

• Cognitive Mirage: A Review of Hallucinations in Large Language Models, arXiv 2023.09 [Paper] [GitHub]

• Augmenting LLMs with Knowledge: A survey on hallucination prevention, arXiv 2023.09 [Paper]

• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models, arXiv 2024.01 [Paper]

Factuality

• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment, arXiv 2023.08 [Paper]

• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity, arXiv 2023.10 [Paper] [GitHub]

• Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models, arXiv 2023.10 [Paper]

Attributes of LLMs

• Explainability for Large Language Models: A Survey, arXiv 2023.09 [Paper]

• The Mystery and Fascination of LLMs: A Comprehensive Survey on the Interpretation and Analysis of Emergent Abilitie, arXiv 2023.11 [Paper]

• From Understanding to Utilization: A Survey on Explainability for Large Language Models, arXiv 2024.01 [Paper]

• A Survey of Large Language Models Attribution, arXiv 2023.11 [Paper] [GitHub]

• A Survey of Language Model Confidence Estimation and Calibration, arXiv 2023.11 [Paper]

• Shortcut Learning of Large Language Models in Natural Language Understanding, COMMUN ACM 2023.12 [Paper]

• Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies, arXiv 2023.08 [Paper] [GitHub]

Efficient LLMs

• Efficient Large Language Models: A Survey, arXiv 2023.12 [Paper] [GitHub]

• LLM Inference Unveiled: Survey and Roofline Model Insights, arXiv 2024.03 [Paper]

• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems, arXiv 2023.12 [Paper]

• A Survey on Model Compression for Large Language Models, arXiv 2023.08 [Paper]

• A Comprehensive Survey of Compression Algorithms for Language Models, arXiv 2024.01 [Paper]

• The Efficiency Spectrum of Large Language Models: An Algorithmic Survey, arXiv 2023.10 [Paper] [GitHub]

• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment, arXiv 2023.12 [Paper]

• Model Compression and Efficient Inference for Large Language Models: A Survey, arXiv 2024.02 [Paper]

• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding, arXiv 2024.01 [Paper] [GitHub]

• A Survey on Hardware Accelerators for Large Language Models, arXiv 2024.01 [Paper]

Learning Methods for LLMs

• Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges, arXiv 2023.11 [Paper]

• Continual Learning with Pre-Trained Models: A Survey, arXiv 2024.01 [Paper] [GitHub]

• Continual Learning for Large Language Models: A Survey, arXiv 2024.02 [Paper]

• Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning, arXiv 2023.03 [Paper]

Multimodal LLMs

• Vision-Language Instruction Tuning: A Review and Analysis, arXiv 2023,11 [Paper] [GitHub]

• Large Language Models Meet Computer Vision: A Brief Survey, arXiv 2023.11 [Paper]

• Foundational Models Defining a New Era in Vision: A Survey and Outlook, arXiv 2023.07 [Paper] [GitHub]

• Video Understanding with Large Language Models: A Survey, arXiv 2023.12 [Paper] [GitHub]

• Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook, arXiv 2023.10 [Paper] [GitHub]

• Sparks of large audio models: A survey and outlook, arXiv 2023.08 [Paper] [GitHub]

• How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model, arXiv 2023.11 [Paper]

• A Survey on Multimodal Large Language Models, arXiv 2023.06 [Paper]

• Multimodal Large Language Models: A Survey, arXiv 2023.11 [Paper]

Knowledge Based LLMs

Retrieval-Augmented LLMs

• Building trust in conversational ai: A comprehensive review and solution architecture for explainable, privacy-aware systems using llms and knowledge graph, arXiv 2023.08 [Paper]

• A Survey on Retrieval-Augmented Text Generation, arXiv 2022.02 [Paper]

• Retrieval-Augmented Generation for Large Language Models: A Survey, arXiv 2023.12 [Paper] [GitHub]

Knowledge Editing

• Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications, arXiv 2023.11 [Paper]

• Knowledge Editing for Large Language Models: A Survey, arXiv 2023.10 [Paper]

• Editing Large Language Models: Problems, Methods, and Opportunities, arXiv 2023.05 [Paper]

Extension of LLMs

LLMs with Tools

• A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond, arXiv 2024.03 [Paper] [GitHub]

• Foundation Models for Decision Making: Problems, Methods, and Opportunities, arXiv 2023.03 [Paper]

• Augmented Language Models: a Survey, arXiv 2023.02 [Paper]

• Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey, arXiv 2023.10 [Paper] [GitHub]

• Large Language Models Meet NL2Code: A Survey, arXiv 2022.12 [Paper]

LLMs and Interactions

• Large Language Models for Robotics: A Survey, arXiv 2023.11 [Paper]

• A Survey on Multimodal Large Language Models for Autonomous Driving, WACV workshop 2023.11 [Paper]

• LLM4Drive: A Survey of Large Language Models for Autonomous Driving, arXiv 2023.11 [Paper] [GitHub]

• A Survey on Large Language Model based Autonomous Agents, arXiv 2023.08 [Paper] [GitHub]

• The Rise and Potential of Large Language Model Based Agents: A Survey, arXiv 2023.09 [Paper] [GitHub]

• Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives, arXiv 2023.12 [Paper]

• Large Multimodal Agents: A Survey, arXiv 2024.02 [Paper] [GitHub]

• Role play with large language models, arXiv 2023.11 [Paper]

Long Sequence LLMs

• Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey, arXiv 2023.11 [Paper]

• Length Extrapolation of Transformers: A Survey from the Perspective of Position Encoding, arXiv 2023.12 [Paper]

LLMs Applications

Education

• ChatGPT and Beyond: The Generative AI Revolution in Education, arXiv 2023.11 [Paper]

• ChatGPT and large language models in academia: opportunities and challenges, arXiv 2023.07 [Paper]

• ChatGPT for good? On opportunities and challenges of large language models for education, arXiv 2023.04 [Paper]

Law

• Large Language Models in Law: A Survey, arXiv 2023.11 [Paper]

• A short survey of viewing large language models in legal aspect, arXiv 2023.03 [Paper]

Healthcare

• A Survey of Large Language Models in Medicine: Progress, Application, and Challenge, arXiv 2023.11 [Paper] [GitHub]

• Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review, arXiv 2023.11 [Paper] [GitHub]

• Large AI Models in Health Informatics: Applications, Challenges, and the Future, arXiv 2023.03 [Paper] [GitHub]

• A SWOT (Strengths, Weaknesses, Opportunities, and Threats) Analysis of ChatGPT in the Medical Literature: Concise Review, JMIR 2023.11 [Paper]

• ChatGPT in Healthcare: A Taxonomy and Systematic Review, Computer Methods and Programs in Biomedicine 2024.01 [Paper]

• A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement, NCBI 2023.10 [Paper]

• Towards a Psychological Generalist AI: A Survey of Current Applications of Large Language Models and Future Prospects, arXiv 2023.12 [Paper]

• Large Language Models in Mental Health Care: a Scoping Review, arXiv 2024.01 [Paper]

• The utility of ChatGPT as an example of large language models in healthcare education, research and practice: Systematic review on the future perspectives and, arXiv 2023.12 [Paper]

• The imperative for regulatory oversight of large language models (or generative AI) in healthcare, arXiv 2023.07 [Paper]

• A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics, arXiv 2023.10 [Paper] [GitHub]

• The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs, arXiv 2023.03 [Paper]

Games

• Large Language Models and Games: A Survey and Roadmap, arXiv 2024.02 [Paper]

• Large Language Models and Video Games: A Preliminary Scoping Review, arXiv 2024.03 [Paper]

NLP Tasks

• Large Language Models for Information Retrieval: A Survey, arXiv 2023.08 [Paper] [GitHub]

• Large Language Models for Generative Information Extraction: A Survey, arXiv 2023.12 [Paper] [GitHub]

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey, arXiv 2021.11 [Paper]

• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents, arXiv 2024.01 [Paper]

Software Engineering

• Large Language Models for Software Engineering: Survey and Open Problems, arXiv 2023.10 [Paper]

• Large Language Models for Software Engineering: A Systematic Literature Review, arXiv 2023.08 [Paper]

• Software Testing with Large Language Models: Survey, Landscape, and Vision, arXiv 2023.07 [Paper]

• Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code, arXiv 2024.01 [Paper] [GitHub]

Recommender Systems

• Foundation Models for Recommender Systems: A Survey and New Perspectives, arXiv 2024.02 [Paper]

• User Modeling in the Era of Large Language Models: Current Research and Future Directions, arXiv 2023.11 [Paper] [GitHub]

• A Survey on Large Language Models for Personalized and Explainable Recommendations, arXiv 2023.11 [Paper]

• Large Language Models for Generative Recommendation: A Survey and Visionary Discussions, arXiv 2023.09 [Paper]

• A Survey on Large Language Models for Recommendation, arXiv 2023.05 [Paper] [GitHub]

• How Can Recommender Systems Benefit from Large Language Models: A Survey, arXiv 2023.06 [Paper] [GitHub]

Graphs

• A Survey of Graph Meets Large Language Model: Progress and Future Directions, arXiv 2023.11 [Paper]

• Large Language Models on Graphs: A Comprehensive Survey, arXiv 2023.12 [Paper] [GitHub]

• The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges, arXiv 2023.03 [Paper]

Other

• Large Language Models in Finance: A Survey, ICAIF 2023.11 [Paper]

• Mathematical Language Models: A Survey, arXiv 2023.12 [Paper]

• Recent applications of AI to environmental disciplines: A review, SCI TOTAL ENVIRON 2023.10 [Paper]

• Opportunities and Challenges of Applying Large Language Models in Building Energy Efficiency and Decarbonization Studies: An Exploratory Overview, arXiv 2023.12 [Paper]

• When Large Language Models Meet Citation: A Survey, arXiv 2023.09 [Paper]

• A Survey of Text Watermarking in the Era of Large Language Models, arXiv 2023.12 [Paper]

• The future of gpt: A taxonomy of existing chatgpt research, current challenges, and possible future directions, SSRN 2023.04 [Paper]

• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models, Meta-Radiology 2023.09 [Paper]

Acknowledgements

We would like to thank the people who have contributed to this project. The core contributors are

Junhao Ruan, Long Meng, Weiqiao Shan, Tong Xiao, Jingbo Zhu