README.md

Transformer NLP

Demos

Here are a few info:

In Natural Language Processing:

Encoder Only

• Transformer Utilization
• Roberta Base Implementation
• Masked language modeling with DebertaV3
• Training any NLP Transformer with Native Pytorch
• Roberta Distilation classification
• Knowledge distilation from large language model

Decoder Only

• GPT-2 Finetunning for text genration
• GPT-2 implementation from scratch for text genration

Encoder Decoder

• Roberta Distilation Seq2Seq Summarization
• Seq2Seq Translation

Knowledge Distillation

• Roberta distilation
• Roberta Distilation classification
• Knowledge distilation from large language model
• Initializing a model with different embedding size and layer from large model through Teacher-Student pretraining

Model architectures

1. RoBERTa (from Facebook)