2020 CCNS A Survey of Text Data Augmentation

[DelaramRajaei]

Title	A Survey of Text Data Augmentation
Year	2022
Venue	CCNS
Paper's Link	https://ieeexplore.ieee.org/document/9240734

This research paper served as a survey on data augmentation and was mentioned in the footnote of "A Survey of Data Augmentation Approaches for NLP" paper. I specifically selected and initiated my literature review with this paper to gain insights into the hierarchy of data augmentation.

What is data augmentation?

• Generating new data, equivalent to the training sample without needing to significantly increase the amount of data.
• Data augmentation = Data enhancement
• Data augmentation was first used in computer vision and speech recognition.
• The method of text data enhancement includes:
  • back translation
  • random word replacement
  • non-core word replacement
  • data enhancement based on text generation model

Why is data augmentation needed?

• The number of training samples is limited and the model faces the possibility of underfitting.
• High cost of manual annotation
• Data augmentation is also used to balance the data. Imbalance data may lead to low accuracy and low recall rate.
• Improve the model effect which depends on the quality of data.

Data augmentation can divide into 3 categories:

• Supervised data enhancement: Using data transformation rules on available data. Transformation rules are divided into 2 categories.

Categories	Description
Single sample data enhancement	Transforming individual samples
Multi sample data enhancement	Transforming multiple samples together. A few examples of methods or approaches: 1. Synthetic minority over-sampling technique, 2. SamplePairing, 3. mixup

• Semi-supervised data enhancement

• Unsupervised data enhancement: Unsupervised data enhancement can be separated into two main areas: creating new data and learning how to improve the existing data.

Unsupervised data enhancement methods	Description
Method 1	Involves using a model to learn the pattern of data distribution. It generates new images that match the patterns found in the training dataset. This approach is commonly known as GAN.
Method 2	Utilize a model to learn a data enhancement technique, suitable for the specific task at hand. Example: AutoAugment.

Data augmentation in NLP

• Adding noise - supervised method
• Back-translation - supervised method
• Challenges
  • In NLP data is discrete -> makes it impossible to convert input data directly
  • It may change the meaning or the tag (label) of the sentence

Classification of Data augmentation

• Unconditional Data augmentation

  • Back-translation

    • Translate the original data into other languages and then back to the original language
    • Inter-language can be more than one(>=1)
    • Both the translation model and tool can be used in this method.
    • Translation models:
      => produce text with richer language
      => seq2seq, NMT, transformer
    • Translation tools:
      => generate text that is more accurate and coherent in terms of meaning.
      => Google translation, baidu translation, Youdao translation
    • The quality of data generated through back translation relies on translation accuracy, which may not always be high.

  • Lexical Subsituaiton

    • Divided into two categories: 1. random word substitution, 2. non-core word substitution

Random word substitution	Selects random words and performs synonym replacement or deletion
Easy Data Augmentation (EDA)	In 2019 a new method proposed which contains 4 operations
1. Synonym replacement	Select non-stop words randomly from sentences and replace these words with randomly selected synonyms.
2. Random insertion	Random insertion involves randomly selecting a non-stop-word from the sentence and replacing it with a randomly chosen synonym, which is then inserted at a random position in the sentence. Repeat n times.
3. Random exchange	Random swap is a random choice of two words in a sentence and exchanging their positions.
4. Random deletion	Random deletion is to delete every word in a sentence with probability P

Random word substitution	3 ways to obtain words in random word replacement
synonym dictionary	Using character-level replacement in text classification
word vector	Finding adjacent words in the embedding space to replace. Some famous pre-trained word embeddings include Word2Vec, GloVe, FastText, Sent2Vec
language model	Using a language model like BERT, substitution words are obtained by masking vocabulary and predicting replacements. This method generates contextually-aware text with improved semantic coherence. However, there are risks involved, such as potential changes in the original text's semantics or category label. For reducing the risk of changing the semantics, words with lower TF-IDF scores can be replaced.

• Conditional Data augmentation: Involves adding "text label" information to the model and using it to generate data.
  • Deep Generative Mode
    • This method needs a large quantity of high-quality text, however, data augmentation is employed because of limited data availability. So the practicability of deep generative model is poor
  • Pre-trained Models
    • Pre-trained models, such as Contextual Augment, utilize tag information and abundant unlabeled data for pre-training to improve results.