Research literature notes 🤓

Notes from papers I'm reading, ordered by topic and chronologically.

• Research literature notes 🤓
  • NLP
    • Embeddings
    • Architectures
    • Frameworks
    • Datasets
    • NER
    • Sarcasm detection
    • Text summarization
    • Machine translation
    • Text generation
  • Reinforcement learning
  • Computer vision
  • Machine learning
  • Audio
  • Linguistics
  • Social sciences
  • Humanities
  • Economics
  • Physics
  • Neuroscience
  • Algorithms

NLP

1. What’s Going On in Neural Constituency Parsers? An Analysis, Gaddy et al., 2018 [Paper] [Notes] #nlp
2. Two Methods for Domain Adaptation of Bilingual Tasks: Delightfully Simple and Broadly Applicable, Hangya et al., 2018 [Paper] [Notes] #nlp
3. What do you learn from context? Probing for sentence structure in contextualized word representations, Tenney et al., 2019 [Paper] [Notes] #nlp
4. BPE-Dropout: simple and effective subword regularization, Provilkov et al., 2019 [Paper] [Notes] #nlp
5. From English To Foreign Languages: Transferring Pre-trained Language Models, Tran, 2020 [Paper] [Notes] #nlp
6. Evaluating NLP models via contrast sets, Gardner et al., 2020 [Paper] [Notes] #nlp
7. Byte Pair Encoding is Suboptimal for Language Model Pretraining, Bostrom et al., 2020 [Paper] [Notes] #nlp
8. Translation artifacts in cross-lingual transfer learning, Artetxe et al., 2020 [Paper] [Notes] #nlp
9. Weight poisoning attacks on pre-trained models, Kurita et al., 2020 [Paper] [Notes] #nlp
10. SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings, Sabet et al., 2020 [Paper] [Notes] #nlp
11. Experience Grounds Language, Bisk et al., 2020 [Paper] [Notes] #nlp #linguistics
12. Beyond accuracy: behavioral testing of NLP models with CheckList, Ribeiro et al., 2020 [Paper] [Notes] #nlp
13. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes, Kiela et al., 2020 [Paper] [Notes] #nlp
14. The Unstoppable Rise of Computational Linguistics in Deep Learning, Henderson, 2020 [Paper] [Notes] #nlp #linguistics
15. Language (Technology) is Power: A Critical Survey of "Bias" in NLP, Blodgett et al., 2020 [Paper] [Notes] #nlp
16. Representation Learning for Information Extraction from Form-like Documents, Majumder et al., 2020 [Paper] [Notes] #nlp
17. Learning to tag OOV tokens by integrating contextual representation and background knowledge, He et al., 2020 [Paper] [Notes] #nlp
18. It's not just size that matters, small language models are also few-shot learners, Schick and Schütze, 2020 [Paper] [Notes] #nlp
19. Did you read the next episode? Using textual cues for predicting podcast popularity, Joshi et al., 2020 [Paper] [Notes] #nlp
20. A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios, Hedderich et al., 2020 [Paper] [Notes] #nlp
21. Challenges in Deploying Machine Learning: a Survey of Case Studies, Paleyes et al., 2020 [Paper] [Notes] #nlp
22. Adapting Coreference Resolution to Twitter Conversations, Aktas et al., 2020 [Paper] [Notes] #nlp
23. Learning from others' mistakes: avoiding dataset biases without modeilng them, Sanh et al., 2020 [Paper] [Notes] #nlp
24. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation, Clark et al., 2021 [Paper] [Notes] #nlp

Embeddings

1. Semi-supervised sequence tagging with bidirectional language models, Peters et al., 2017 [Paper] [Notes] #nlp #embeddings
2. Mimicking Word Embeddings using Subword RNNs, Pinter et al., 2017 [Paper] [Notes] #nlp #embeddings
3. Deep contextualized word representations, Peters et al., 2018 [Paper] [Notes] #nlp #embeddings
4. Linguistic Knowledge and Transferability of Contextual Representations, Liu et al., 2019 [Paper] [Notes] #nlp #embeddings
5. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates, Kudo, 2018 [Paper] [Notes] #nlp #embeddings
6. Dissecting contextual word embeddings: architecture and representation, Peters et al., 2018 [Paper] [Notes] #nlp #embeddings
7. BERT: Pre-training of deep bidirectional transformers for language understanding, Devlin et al., 2018 [Paper] [Notes] #nlp #embeddings
8. Learning Semantic Representations for Novel Words: Leveraging Both Form and Context, Schick et al., 2018 [Paper] [Notes] #nlp #embeddings
9. Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia, Yamada et al., 2018 [Paper] [Notes] #nlp #embeddings
10. Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
11. Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
12. BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
13. BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA, Poerner et al., 2019 [Paper] [Notes] #nlp #embeddings

Architectures

1. Conditional Random Fields: probabilistic models for segmenting and labeling sequence data, Lafferty et al, 2001 [Paper] [Notes] #nlp #architectures
2. Bidirectional LSTM-CRF Models for sequence tagging, Huang et al., 2015 [Paper] [Notes] #nlp #architectures
3. Neural Architectures for Named Entity Recognition, Lample et al., 2016 [Paper] [Notes] #nlp #architectures #NER
4. Named Entity Recognition with Bidirectional LSTM-CNNs, Chiu et al., 2016 [Paper] [Notes] #nlp #architectures
5. Attention is all you need, Vaswani et al., 2018 [Paper] [Notes] #nlp #architectures
6. Reasoning with Sarcasm by Reading In-between, Tay et al., 2018 [Paper] [Notes] #sarcasm-detection #architectures
7. XLNet: generalized autoregressive pretraining for language understanding, Yang et al., 2019 [Paper] [Notes] #nlp #architectures
8. R-Transformer: Recurrent Neural Network Enhanced Transformer, Wang et al., 2019 [Paper] [Notes] #nlp #architectures
9. Generalization through Memorization: Nearest Neighbor Language Models, Khandelwal et al., 2019 [Paper] [Notes] #nlp #architectures
10. Single Headed Attention RNN: Stop Thinking With Your Head, Merity, 2019 [Paper] [Notes] #nlp #architectures
11. A Transformer-based approach to Irony and Sarcasm detection, Potamias et al., 2019 [Paper] [Notes] #sarcasm-detection #architecture
12. ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, Qi et al., 2020 [Paper] [Notes] #nlp #architectures
13. Pre-trained Models for Natural Language Processing: A Survey, Qiu et al., 2020 [Paper] [Notes] #nlp #architectures
14. SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, Iandola et al., 2020 [Paper] [Notes] #nlp #architectures #computer-vision
15. A comparison of LSTM and BERT for small corpus, Ezen-Can, 2020 [Paper] [Notes] #nlp #architectures

Frameworks

1. Flair: an easy-to-use framework for stat-of-the-art NLP [Paper] [Notes] #nlp #frameworks
2. HuggingFace's Transformers: State-of-the-art Natural Language Processing, Wolf et al., 2019 [Paper] [Notes] #nlp #frameworks
3. Selective Brain Damage: Measuring the Disparate Impact of Model Pruning, Hooker et al., 2019 [Paper] [Notes] #frameworks
4. Why should we add early exits to neural networks?, Scardapane et al., 2020 [Paper] [Notes] #frameworks

Datasets

1. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition, Sang et al., 2003 [Paper] [Notes] #nlp #datasets
2. Datasheets for datasets, Gebru et al., 2018 [Paper] [Notes] #nlp #datasets
3. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference, Zellers et al., 2018 [Paper] [Notes] #nlp #datasets
4. A Named Entity Recognition Shootout for German, Riedl and Padó, 2018 [Paper] [Notes] #nlp #NER #datasets
5. Probing Neural Network Comprehension of Natural Language Arguments, Nivel and Kao, 2019 [Paper] [Notes] #nlp #datasets
6. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference., McCoy et al., 2019 [Paper] [Notes] #nlp #linguistics #datasets
7. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor, Hasan et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
8. HellaSwag: Can a Machine Really Finish Your Sentence?, Zellers et al., 2019 [Paper] [Notes] #nlp #datasets
9. Sentiment analysis is not solved! Assessing and probing sentiment classification, Barnes et al., 2019 [Paper] [Notes] #nlp #datasets
10. Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model, Cai et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
11. Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper), Castro et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
12. iSarcasm: A Dataset of Intended Sarcasm, Oprea et al., 2019 [Paper] [Notes] #datasets #sarcasm-detection
13. Lessons from archives: strategies for collecting sociocultural data in machine learning, Seo Jo and Gebru, 2019 [Paper] [Notes] #nlp #datasets
14. BERTweet: A pre-trained language model for English Tweets, Nguyen et al., 2020 [Paper] [Notes] #nlp #datasets
15. GAIA: a fine-grained multimedia knowlege extraction system, Li et al., 2020 [Paper, [Notes] #nlp #datasets
16. It's morphin' time! Combating linguistic discrimination with inflectional perturbations, Tan et al., 2020 [Paper, [Notes] #nlp #datasets
17. Reactive Supervision: A New method for Collecting Sarcasm Data, Shmueli et al, 2020 [Paper] [Notes] #datasets #sarcasm-detection

NER

1. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition, Sang et al., 2003 [Paper] [Notes] #nlp #datasets #NER
2. Neural Architectures for Named Entity Recognition, Lample et al., 2016 [Paper] [Notes] #nlp #architectures #NER
3. Named Entity Recognition with Bidirectional LSTM-CNNs, Chiu et al., 2016 [Paper] [Notes] #nlp #architectures #NER
4. Towards Robust Named Entity Recognition for Historic German, Schweter et al., 2019 [Paper] [Notes] #nlp #NER
5. A Named Entity Recognition Shootout for German, Riedl and Padó, 2018 [Paper] [Notes] #nlp #NER #datasets

Sarcasm detection

summary

1. Sarcasm Detection on Twitter: A Behavioral Modeling Approach, Rajadesingan et al., 2015 [Paper] [Notes] #sarcasm-detection
2. Contextualized Sarcasm Detection on Twitter, Bamman and Smith, 2015 [Paper] [Notes] #sarcasm-detection
3. Harnessing Context Incongruity for Sarcasm Detection, Joshi et al., 2015 [Paper] [Notes] #linguistics #sarcasm-detection
4. Automatic Sarcasm Detection: A Survey, Joshi et al., 2017 [Paper] [Notes] #sarcasm-detection
5. Detecting Sarcasm is Extremely Easy ;-), Parde and Nielsen, 2018 [Paper] [Notes] #sarcasm-detection
6. CASCADE: Contextual Sarcasm Detection in Online Discussion Forums, Hazarika et al., 2018 [Paper] [Notes] #sarcasm-detection
7. Reasoning with Sarcasm by Reading In-between, Tay et al., 2018 [Paper] [Notes] #sarcasm-detection #architectures
8. Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning, Wu et al., 2018 [Paper] [Notes] #sarcasm-detection
9. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor, Hasan et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
10. Exploring Author Context for Detecting Intended vs Perceived Sarcasm, Oprea and Magdy, 2019 [Paper] [Notes] #sarcasm-detection
11. Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper), Castro et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
12. Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model, Cai et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
13. A2Text-Net: A Novel Deep Neural Network for Sarcasm Detection, Liu et al., 2019 [Paper] [Notes] #sarcasm-detection
14. Sarcasm detection in tweets, Rajagopalan et al., 2019 [Paper] [Notes] #sarcasm-detection
15. A Transformer-based approach to Irony and Sarcasm detection, Potamias et al., 2019 [Paper] [Notes] #sarcasm-detection #architecture
16. Deep and dense sarcasm detection, Pelser et al., 2019 [Paper] [Notes] #sarcasm-detection
17. iSarcasm: A Dataset of Intended Sarcasm, Oprea et al., 2019 [Paper] [Notes] #datasets #sarcasm-detection
18. Reactive Supervision: A New method for Collecting Sarcasm Data, Shmueli et al, 2020 [Paper] [Notes] #datasets #sarcasm-detection

Text summarization

1. Evaluating the Factual Consistency of Abstractive Text Summarization, Kryscinski et al., 2019 [Paper] [Notes] #nlp #text-summarization
2. TLDR: extreme summarization of scientific documents, Cachola et al, 2020 [Paper] [Notes] #nlp #text-summarization
3. A survey on text simplification, Sikka and Mago, 2020 [Paper] [Notes] #nlp #text-summarization

Machine translation

1. Unsupervised Tokenization for Machine Translation, Chung and Gildea, 2009 [Paper] [Notes] #nlp #machine-translation
2. Neural Machine Translation of Rare Words with Subword Units, Sennrich et al., 2015 [Paper] [Notes] #nlp #machine-translation
3. Unsupervised neural machine translation, Artetxe et al., 2017 [Paper] [Notes] #nlp #machine-translation
4. How Much Does Tokenization Affect Neural Machine Translation? Domingo et al., 2018 [Paper] [Notes] #nlp #machine-translation
5. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT, Chronopoulou et al., 2020 [Paper] [Notes] #nlp #machine-translation

Text generation

• Article on different types of NLG

---

Reinforcement learning

1. Theory of Minds: Understanding Behavior in Groups Through Inverse Planning, Shum et al., 2019 [Paper] [Notes] #reinforcement-learning #social-sciences
2. The Hanabi Challenge: A New Frontier for AI Research, Bard et al., 2019 [Paper] [Notes] #reinforcement-learning
3. Mastering Atari, Go, Chess and Shogi by Planning with a learned model, Schrittwieser et al., 2019 [Paper] [Notes] #reinforcement-learning
4. Language as a cognitive tool to imagine goals in curiosity-driven exploration, Colas et al., 2020 [Paper] [Notes] #reinforcement-learning
5. Planning to Explore via Self-Supervised World Models, Sekar et al., 2020 [Paper] [Notes] #reinforcement-learning

---

Computer vision

1. Cubic Stylization, Derek Liu and Jacobson, 2019 [Paper] [Notes] #computer-vision
2. SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, Iandola et al., 2020 [Paper] [Notes] #nlp #computer-vision

---

Machine learning

1. Gender shades: intersectional accuracy disparities in commercial gender classification, Buolamwini and Gebru, 2018 [Paper] [Notes] #machine-learning
2. Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges, Molnar et al., 2020 [Paper] [Notes] #machine-learning

Audio

1. End-to-End Adversarial Text-to-Speech, Donahue et al., 2020 [Paper] [Notes] #audio
2. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, Baevski et al., 2020 [Paper] [Notes] #audio
3. Large-scale multilingual audio visual dubbing, Yang et al., 2020 [Paper] [Notes] #audio

---

Linguistics

1. Moving beyond the plateau: from lower-intermediate to upper-intermediate, Richards, 2015 [Paper] [Notes] #linguistics
2. Harnessing Context Incongruity for Sarcasm Detection, Joshi et al., 2015 [Paper] [Notes] #linguistics #sarcasm-detection
3. A Trainable Spaced Repetition Model for Language Learning, Settles and Meeder, 2016 [Paper] [Notes] #linguistics
4. Targeted synctactic evaluation of language models, Marvin and Linzen, 2018 [Paper] [Notes] #nlp #linguistics
5. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference., McCoy et al., 2019 [Paper] [Notes] #nlp #linguistics #datasets
6. Language Models as Knowledge Bases?, Petroni et al., 2019 [Paper] [Notes] #nlp #linguistics
7. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche, Coupé et al., 2019 [Paper] [Notes] #linguistics #social-sciences
8. My English sounds better than yours: Second language learners perceive their own accent as better than that of their peers, Mittlerer et al., 2020 [Paper] [Notes] #linguistics
9. Experience Grounds Language, Bisk et al., 2020 [Paper] [Notes] #nlp #linguistics
10. The Unstoppable Rise of Computational Linguistics in Deep Learning, Henderson, 2020 [Paper] [Notes] #nlp #linguistics
11. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data, Bender et al., 2020 [Paper] [Notes] #nlp #linguistics

---

Social sciences

1. Antisocial Behavior in Online Discussion Communities, Cheng et al., 2015 [Paper] [Notes] #social-sciences
2. How much does education improve intelligence? A meta-analysis, Ritchie et al., 2017 [Paper] [Notes] #social-sciences
3. Theory of Minds: Understanding Behavior in Groups Through Inverse Planning, Shum et al., 2019 [Paper] [Notes] #reinforcement-learning #social-sciences
4. Fake news game confers psychological resistance against online misinformation, Roozenbeek and van der Linden, 2019 [Paper] [Notes] #social-sciences #humanities
5. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche, Coupé et al., 2019 [Paper] [Notes] #linguistics #social-sciences
6. Kids these days: Why the youth of today seem lacking, Protzko and Schooler, 2019 [Paper] [Notes] #social-sciences

---

Humanities

1. Fake news game confers psychological resistance against online misinformation, Roozenbeek and van der Linden, 2019 [Paper] [Notes] #social-sciences #humanities

---

Economics

1. Why do people stay poor? Balboni et al., 2020 [Paper] [Notes] #economics

---

Physics

1. First-order transition in a model of prestige bias, Skinner, 2019 [Paper] [Notes] #physics

---

Neuroscience

1. A deep learning framework for neuroscience, Richard et al., 2019 [Paper] [Notes] #neuroscience

---

Algorithms

1. Replace or Retrieve Keywords In Documents At Scale, Singh, 2017 [Paper] [Notes] #algorithms