NLP-Models-Tensorflow, Gathers machine learning and tensorflow deep learning models for NLP problems, code simplify inside Jupyter Notebooks 100%.
- Abstractive Summarization
- Chatbot
- Dependency Parser
- Entity Tagging
- Extractive Summarization
- Generator
- Language Detection
- Neural Machine Translation
- OCR
- POS Tagging
- Question-Answers
- Sentence pairs
- Speech-to-Text
- Spelling correction
- SQUAD Question-Answers
- Stemming
- Text Augmentation
- Text Classification
- Text Similarity
- Text-to-Speech
- Topic Generator
- Topic Modeling
- Unsupervised Extractive Summarization
- Vectorizer
- Old-to-Young Vocoder
- Visualization
- Attention
Original implementations are quite complex and not really beginner friendly. So I tried to simplify most of it. Also, there are tons of not-yet release papers implementation. So feel free to use it for your own research!
I will attached github repositories for models that I not implemented from scratch, basically I copy, paste and fix those code for deprecated issues.
Tensorflow version 1.10 and above only, not included 2.X version.
pip install -r requirements.txt
Trained on India news.
Accuracy based on 10 epochs only, calculated using word positions.
- LSTM Seq2Seq using topic modelling, test accuracy 13.22%
- LSTM Seq2Seq + Luong Attention using topic modelling, test accuracy 12.39%
- LSTM Seq2Seq + Beam Decoder using topic modelling, test accuracy 10.67%
- LSTM Bidirectional + Luong Attention + Beam Decoder using topic modelling, test accuracy 8.29%
- Pointer-Generator + Bahdanau, https://github.com/xueyouluo/my_seq2seq, test accuracy 15.51%
- Copynet, test accuracy 11.15%
- Pointer-Generator + Luong, https://github.com/xueyouluo/my_seq2seq, test accuracy 16.51%
- Dilated Seq2Seq, test accuracy 10.88%
- Dilated Seq2Seq + Self Attention, test accuracy 11.54%
- BERT + Dilated CNN Seq2seq, test accuracy 13.5%
- self-attention + Pointer-Generator, test accuracy 4.34%
- Dilated-CNN Seq2seq + Pointer-Generator, test accuracy 5.57%
Trained on Cornell Movie Dialog corpus, accuracy table in chatbot.
- Seq2Seq-manual
- Seq2Seq-API Greedy
- Bidirectional Seq2Seq-manual
- Bidirectional Seq2Seq-API Greedy
- Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
- Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
- Bytenet
- Capsule layers + LSTM Seq2Seq-API + Luong Attention + Beam Decoder
- End-to-End Memory Network
- Attention is All you need
- Transformer-XL + LSTM
- GPT-2 + LSTM
- Tacotron + Beam decoder
Complete list (54 notebooks)
- Basic cell Seq2Seq-manual
- LSTM Seq2Seq-manual
- GRU Seq2Seq-manual
- Basic cell Seq2Seq-API Greedy
- LSTM Seq2Seq-API Greedy
- GRU Seq2Seq-API Greedy
- Basic cell Bidirectional Seq2Seq-manual
- LSTM Bidirectional Seq2Seq-manual
- GRU Bidirectional Seq2Seq-manual
- Basic cell Bidirectional Seq2Seq-API Greedy
- LSTM Bidirectional Seq2Seq-API Greedy
- GRU Bidirectional Seq2Seq-API Greedy
- Basic cell Seq2Seq-manual + Luong Attention
- LSTM Seq2Seq-manual + Luong Attention
- GRU Seq2Seq-manual + Luong Attention
- Basic cell Seq2Seq-manual + Bahdanau Attention
- LSTM Seq2Seq-manual + Bahdanau Attention
- GRU Seq2Seq-manual + Bahdanau Attention
- LSTM Bidirectional Seq2Seq-manual + Luong Attention
- GRU Bidirectional Seq2Seq-manual + Luong Attention
- LSTM Bidirectional Seq2Seq-manual + Bahdanau Attention
- GRU Bidirectional Seq2Seq-manual + Bahdanau Attention
- LSTM Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
- GRU Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
- LSTM Seq2Seq-API Greedy + Luong Attention
- GRU Seq2Seq-API Greedy + Luong Attention
- LSTM Seq2Seq-API Greedy + Bahdanau Attention
- GRU Seq2Seq-API Greedy + Bahdanau Attention
- LSTM Seq2Seq-API Beam Decoder
- GRU Seq2Seq-API Beam Decoder
- LSTM Bidirectional Seq2Seq-API + Luong Attention + Beam Decoder
- GRU Bidirectional Seq2Seq-API + Luong Attention + Beam Decoder
- LSTM Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
- GRU Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
- Bytenet
- LSTM Seq2Seq + tf.estimator
- Capsule layers + LSTM Seq2Seq-API Greedy
- Capsule layers + LSTM Seq2Seq-API + Luong Attention + Beam Decoder
- LSTM Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder + Dropout + L2
- DNC Seq2Seq
- LSTM Bidirectional Seq2Seq-API + Luong Monotic Attention + Beam Decoder
- LSTM Bidirectional Seq2Seq-API + Bahdanau Monotic Attention + Beam Decoder
- End-to-End Memory Network + Basic cell
- End-to-End Memory Network + LSTM cell
- Attention is all you need
- Transformer-XL
- Attention is all you need + Beam Search
- Transformer-XL + LSTM
- GPT-2 + LSTM
- CNN Seq2seq
- Conv-Encoder + LSTM
- Tacotron + Greedy decoder
- Tacotron + Beam decoder
- Google NMT
Trained on CONLL English Dependency. Train set to train, dev and test sets to test.
Stackpointer and Biaffine-attention originally from https://github.com/XuezheMax/NeuroNLP2 written in Pytorch.
Accuracy based on arc, types and root accuracies after 15 epochs only.
- Bidirectional RNN + CRF + Biaffine, arc accuracy 70.48%, types accuracy 65.18%, root accuracy 66.4%
- Bidirectional RNN + Bahdanau + CRF + Biaffine, arc accuracy 70.82%, types accuracy 65.33%, root accuracy 66.77%
- Bidirectional RNN + Luong + CRF + Biaffine, arc accuracy 71.22%, types accuracy 65.73%, root accuracy 67.23%
- BERT Base + CRF + Biaffine, arc accuracy 64.30%, types accuracy 62.89%, root accuracy 74.19%
- Bidirectional RNN + Biaffine Attention + Cross Entropy, arc accuracy 72.42%, types accuracy 63.53%, root accuracy 68.51%
- BERT Base + Biaffine Attention + Cross Entropy, arc accuracy 72.85%, types accuracy 67.11%, root accuracy 73.93%
- Bidirectional RNN + Stackpointer, arc accuracy 61.88%, types accuracy 48.20%, root accuracy 89.39%
- XLNET Base + Biaffine Attention + Cross Entropy, arc accuracy 74.41%, types accuracy 71.37%, root accuracy 73.17%
Trained on CONLL NER.
- Bidirectional RNN + CRF, test accuracy 96%
- Bidirectional RNN + Luong Attention + CRF, test accuracy 93%
- Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 95%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 96%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 96%
- Char Ngrams + Residual Network + Bahdanau Attention + CRF, test accuracy 69%
- Char Ngrams + Attention is you all Need + CRF, test accuracy 90%
- BERT, test accuracy 99%
- XLNET-Base, test accuracy 99%
Trained on CNN News dataset.
Accuracy based on ROUGE-2.
- LSTM RNN, test accuracy 16.13%
- Dilated-CNN, test accuracy 15.54%
- Multihead Attention, test accuracy 26.33%
- BERT-Base
Trained on Shakespeare dataset.
- Character-wise RNN + LSTM
- Character-wise RNN + Beam search
- Character-wise RNN + LSTM + Embedding
- Word-wise RNN + LSTM
- Word-wise RNN + LSTM + Embedding
- Character-wise + Seq2Seq + GRU
- Word-wise + Seq2Seq + GRU
- Character-wise RNN + LSTM + Bahdanau Attention
- Character-wise RNN + LSTM + Luong Attention
- Word-wise + Seq2Seq + GRU + Beam
- Character-wise + Seq2Seq + GRU + Bahdanau Attention
- Word-wise + Seq2Seq + GRU + Bahdanau Attention
- Character-wise Dilated CNN + Beam search
- Transformer + Beam search
- Transformer XL + Beam search
Trained on Tatoeba dataset.
- Fast-text Char N-Grams
Trained on English-Vietnam, accuracy table in neural-machine-translation.
- Bytenet
- Capsule layers + LSTM Seq2Seq-API + Luong Attention + Beam Decoder
- End-to-End Memory Network
- Attention is All you need
- Conv Seq2Seq
- BERT + Transformer Decoder
- XLNET + Transformer Decoder
Complete list (50 notebooks)
- basic-seq2seq-manual
- lstm-seq2seq-manual
- gru-seq2seq-manual
- basic-seq2seq-api-greedy
- lstm-seq2seq-api-greedy
- gru-seq2seq-greedy
- basic-birnn-seq2seq-manual
- lstm-birnn-seq2seq-manual
- gru-birnn-seq2seq-manual
- basic-birnn-seq2seq-greedy
- lstm-birnn-seq2seq-greedy
- gru-birnn-seq2seq-greedy
- basic-seq2seq-luong
- lstm-seq2seq-luong
- gru-seq2seq-luong
- basic-seq2seq-bahdanau
- lstm-seq2seq-bahdanau
- gru-seq2seq-bahdanau
- basic-birnn-seq2seq-bahdanau
- lstm-birnn-seq2seq-bahdanau
- gru-birnn-seq2seq-bahdanau
- basic-birnn-seq2seq-luong
- lstm-birnn-seq2seq-luong
- gru-birnn-seq2seq-luong
- lstm-seq2seq-contrib-greedy-luong
- gru-seq2seq-contrib-greedy-luong
- lstm-seq2seq-contrib-greedy-bahdanau
- gru-seq2seq-contrib-greedy-bahdanau
- lstm-seq2seq-contrib-beam-bahdanau
- gru-seq2seq-contrib-beam-bahdanau
- lstm-birnn-seq2seq-contrib-beam-luong
- gru-birnn-seq2seq-contrib-beam-luong
- lstm-birnn-seq2seq-contrib-luong-bahdanau-beam
- gru-birnn-seq2seq-contrib-luong-bahdanau-beam
- bytenet-greedy
- capsule-lstm-seq2seq-contrib-greedy
- capsule-gru-seq2seq-contrib-greedy
- dnc-seq2seq-bahdanau-greedy
- dnc-seq2seq-luong-greedy
- lstm-birnn-seq2seq-beam-luongmonotic
- lstm-birnn-seq2seq-beam-bahdanaumonotic
- memory-network-lstm-seq2seq-contrib
- attention-is-all-you-need-beam
- conv-seq2seq
- conv-encoder-lstm-decoder
- dilated-conv-seq2seq
- gru-birnn-seq2seq-greedy-residual
- google-nmt
- bert-transformer-decoder-beam
- xlnet-base-transformer-decoder-beam
- CNN + LSTM RNN, test accuracy 100%
- Im2Latex, test accuracy 100%
Trained on CONLL POS.
- Bidirectional RNN + CRF, test accuracy 92%
- Bidirectional RNN + Luong Attention + CRF, test accuracy 91%
- Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
- Char Ngrams + Residual Network + Bahdanau Attention + CRF, test accuracy 3%
- Char Ngrams + Attention is you all Need + CRF, test accuracy 89%
- BERT, test accuracy 99%
Trained on bAbI Dataset.
- End-to-End Memory Network + Basic cell
- End-to-End Memory Network + GRU cell
- End-to-End Memory Network + LSTM cell
- Dynamic Memory
Trained on Cornell Movie--Dialogs Corpus
- BERT
Trained on Toronto speech dataset.
- Tacotron, https://github.com/Kyubyong/tacotron_asr, test accuracy 77.09%
- BiRNN LSTM, test accuracy 84.66%
- BiRNN Seq2Seq + Luong Attention + Cross Entropy, test accuracy 87.86%
- BiRNN Seq2Seq + Bahdanau Attention + Cross Entropy, test accuracy 89.28%
- BiRNN Seq2Seq + Bahdanau Attention + CTC, test accuracy 86.35%
- BiRNN Seq2Seq + Luong Attention + CTC, test accuracy 80.30%
- CNN RNN + Bahdanau Attention, test accuracy 80.23%
- Dilated CNN RNN, test accuracy 31.60%
- Wavenet, test accuracy 75.11%
- Deep Speech 2, test accuracy 81.40%
- Wav2Vec Transfer learning BiRNN LSTM, test accuracy 83.24%
- BERT-Base
- XLNET-Base
- BERT-Base Fast
- BERT-Base accurate
Trained on SQUAD Dataset.
- BERT,
{"exact_match": 77.57805108798486, "f1": 86.18327335287402}
Trained on English Lemmatization.
- LSTM + Seq2Seq + Beam
- GRU + Seq2Seq + Beam
- LSTM + BiRNN + Seq2Seq + Beam
- GRU + BiRNN + Seq2Seq + Beam
- DNC + Seq2Seq + Greedy
- BiRNN + Bahdanau + Copynet
- Pretrained Glove
- GRU VAE-seq2seq-beam TF-probability
- LSTM VAE-seq2seq-beam TF-probability
- GRU VAE-seq2seq-beam + Bahdanau Attention TF-probability
- VAE + Deterministic Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
- VAE + VAE Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
- BERT-Base + Nucleus Sampling
- XLNET-Base + Nucleus Sampling
Trained on English sentiment dataset, accuracy table in text-classification.
- Basic cell RNN
- Bidirectional RNN
- LSTM cell RNN
- GRU cell RNN
- LSTM RNN + Conv2D
- K-max Conv1d
- LSTM RNN + Conv1D + Highway
- LSTM RNN with Attention
- Neural Turing Machine
- BERT
- Dynamic Memory Network
- XLNET
- ALBERT
- GPT-2
Complete list (77 notebooks)
- Basic cell RNN
- Basic cell RNN + Hinge
- Basic cell RNN + Huber
- Basic cell Bidirectional RNN
- Basic cell Bidirectional RNN + Hinge
- Basic cell Bidirectional RNN + Huber
- LSTM cell RNN
- LSTM cell RNN + Hinge
- LSTM cell RNN + Huber
- LSTM cell Bidirectional RNN
- LSTM cell Bidirectional RNN + Huber
- LSTM cell RNN + Dropout + L2
- GRU cell RNN
- GRU cell RNN + Hinge
- GRU cell RNN + Huber
- GRU cell Bidirectional RNN
- GRU cell Bidirectional RNN + Hinge
- GRU cell Bidirectional RNN + Huber
- LSTM RNN + Conv2D
- K-max Conv1d
- LSTM RNN + Conv1D + Highway
- LSTM RNN + Basic Attention
- LSTM Dilated RNN
- Layer-Norm LSTM cell RNN
- Only Attention Neural Network
- Multihead-Attention Neural Network
- Neural Turing Machine
- LSTM Seq2Seq
- LSTM Seq2Seq + Luong Attention
- LSTM Seq2Seq + Bahdanau Attention
- LSTM Seq2Seq + Beam Decoder
- LSTM Bidirectional Seq2Seq
- Pointer Net
- LSTM cell RNN + Bahdanau Attention
- LSTM cell RNN + Luong Attention
- LSTM cell RNN + Stack Bahdanau Luong Attention
- LSTM cell Bidirectional RNN + backward Bahdanau + forward Luong
- Bytenet
- Fast-slow LSTM
- Siamese Network
- LSTM Seq2Seq + tf.estimator
- Capsule layers + RNN LSTM
- Capsule layers + LSTM Seq2Seq
- Capsule layers + LSTM Bidirectional Seq2Seq
- Nested LSTM
- LSTM Seq2Seq + Highway
- Triplet loss + LSTM
- DNC (Differentiable Neural Computer)
- ConvLSTM
- Temporal Convd Net
- Batch-all Triplet-loss + LSTM
- Fast-text
- Gated Convolution Network
- Simple Recurrent Unit
- LSTM Hierarchical Attention Network
- Bidirectional Transformers
- Dynamic Memory Network
- Entity Network
- End-to-End Memory Network
- BOW-Chars Deep sparse Network
- Residual Network using Atrous CNN
- Residual Network using Atrous CNN + Bahdanau Attention
- Deep pyramid CNN
- Transformer-XL
- Transfer learning GPT-2 345M
- Quasi-RNN
- Tacotron
- Slice GRU
- Slice GRU + Bahdanau
- Wavenet
- Transfer learning BERT Base
- Transfer learning XL-net Large
- LSTM BiRNN global Max and average pooling
- Transfer learning BERT Base drop 6 layers
- Transfer learning BERT Large drop 12 layers
- Transfer learning XL-net Base
- Transfer learning ALBERT
Trained on First Quora Dataset Release: Question Pairs.
- BiRNN + Contrastive loss, test accuracy 76.50%
- Dilated CNN + Contrastive loss, test accuracy 72.98%
- Transformer + Contrastive loss, test accuracy 73.48%
- Dilated CNN + Cross entropy, test accuracy 72.27%
- Transformer + Cross entropy, test accuracy 71.1%
- Transfer learning BERT base + Cross entropy, test accuracy 90%
- Transfer learning XLNET base + Cross entropy, test accuracy 77.39%
Trained on Toronto speech dataset.
- Tacotron, https://github.com/Kyubyong/tacotron
- CNN Seq2seq + Dilated CNN vocoder
- Seq2Seq + Bahdanau Attention
- Seq2Seq + Luong Attention
- Dilated CNN + Monothonic Attention + Dilated CNN vocoder
- Dilated CNN + Self Attention + Dilated CNN vocoder
- Deep CNN + Monothonic Attention + Dilated CNN vocoder
- Deep CNN + Self Attention + Dilated CNN vocoder
Trained on Malaysia news.
- TAT-LSTM
- TAV-LSTM
- MTA-LSTM
- Dilated CNN Seq2seq
Trained on English sentiment dataset.
- LDA2Vec
- BERT Attention
- XLNET Attention
Trained on random books.
- Skip-thought Vector
- Residual Network using Atrous CNN
- Residual Network using Atrous CNN + Bahdanau Attention
Trained on English sentiment dataset.
- Word Vector using CBOW sample softmax
- Word Vector using CBOW noise contrastive estimation
- Word Vector using skipgram sample softmax
- Word Vector using skipgram noise contrastive estimation
- Supervised Embedded
- Triplet-loss + LSTM
- LSTM Auto-Encoder
- Batch-All Triplet-loss LSTM
- Fast-text
- ELMO (biLM)
- Triplet-loss + BERT
- Attention heatmap on Bahdanau Attention
- Attention heatmap on Luong Attention
- BERT attention, https://github.com/hsm207/bert_attn_viz
- XLNET attention
Trained on Toronto speech dataset.
- Dilated CNN
- Bahdanau
- Luong
- Hierarchical
- Additive
- Soft
- Attention-over-Attention
- Bahdanau API
- Luong API
- Markov chatbot
- Decomposition summarization (3 notebooks)