In-context learning Secrets

Relevant papers:

1. In-context Learning and Induction Heads: This paper talks about an emerging phenomena in training transformers called induction heads. As the naming indicates, an induction heads refers to a special type of attention heads that is able to do "induction" reasoning.

2. In-Context Learning Creates Task Vectors

3. Transformers learn in-context by gradient descent

4. Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models

5. Looped Transformers as Programmable Computers

6. SUCCESSOR HEADS: RECURRING, INTERPRETABLE ATTENTION HEADS IN THE WILD

7. Transformers generalize differently from information stored in context vs in weights

8. Many-Shot In-Context Learning

9. Can language models learn from explanations in context?

10. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

11. The Developmental Landscape of In-Context Learning

12. IN-CONTEXT LANGUAGE LEARNING:ARCHITECTURES AND ALGORITHMS

13. [https://arxiv.org/abs/2402.11004](The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains)

14. [https://www.lesswrong.com/posts/j6s9H9SHrEhEfuJnq/causal-scrubbing-results-on-induction-heads](Causal scrubbing: results on induction heads)