Deep Latent Sequence Model

This is PyTorch implementation of the paper:

A Probabilistic Formulation of Unsupervised Text Style Transfer
Junxian He*, Xinyi Wang*, Graham Neubig, Taylor Berg-Kirkpatrick
ICLR 2020

The code implements a deep latent sequence model that performs unsupervised style transfer task. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion.

Please contact [email protected] if you have any quesitons.

Requirements

• Python 3
• PyTorch >= 1.0

Data

Datasets used in sentiment transfer, decipherment, author imitation, and related language translation tasks can be downloaded with (this repo only supports yelp and shakespeare experiments right now, we plan to add the other two tasks before 2/14):

python prepare_data.py --dataset [yelp|decipher|shakespeare|src_bos]

Downloaded data is located in ./data folder. For the formality transfer dataset please see its official repo.

Pretrained LMs and Classifiers

Our approach requires pretrained LMs as priors for each domain during trainining, and an oracle classifier is required at test time to compute the accuracy for sentiment, author imitation, and formality transfer tasks. Here we provide our pretrained LMs and classifiers to reproduce the reported results. (Note that we pretrain these models in PyTorch 1.4, inconsistent PyTorch versions might cause loading problems.)

Download pretrained lms (located in folder ./pretrained_lm):

python prepare_lm.py --dataset [yelp|shakespeare|decipher|sr_bos]

Download pretrained classifiers (located in folder ./pretrained_classifer):

python prepare_classifier --dataset [yelp|shakespeare|decipher|sr_bos]

Usage

Training:

CUDA_VISIBLE_DEVICES=xx bash scripts/[dataset]/train_[dataset].sh

Eval:

CUDA_VISIBLE_DEVICES=xx bash scripts/[dataset]/eval_all.sh [model dir]

The evaluation command will report several evaluation metrics (e.g. accuracy, self-bleu, reference bleu, and ppl for sentiment transfer task) and also transfer the test sentences to another domain, saved in [model dir].

Train your own LMs and Classifiers

Train LMs:

CUDA_VISIBLE_DEVICES=xx bash scripts/train_lm.sh [dataset] [domain_id (0 or 1)]

To run the code on your own text/image dataset, you need to create a new configuration file in ./config/ folder to specifiy network hyperparameters and datapath. If the new config file is ./config/config_abc.py, then [dataset] needs to be set as abc accordingly. Pretrained LMs are saved in ./pretrained_lm/[dataset] folder.

Train LSTM classifiers: see ./scripts/train_classifer.sh for an example to tain an LSTM classifer on shakespeare dataset.

Acknowledgement

We appreciate the following efforts to publish data that we used in this repo:

• Wei Xu, Alan Ritter, William B. Dolan, Ralph Grishman, and Cherry Colin. Paraphrasing for style. COLING 2012.

• Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text

  by cross-alignment. NeurIPS 2017.

• Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discriminators. NeurIPS 2018.

• Delete, retrieve, generate: A simple approach to sentiment and style transfer. NAACL 2018

Reference

@inproceedings{he2020a,
title={A Probabilistic Formulation of Unsupervised Text Style Transfer},
author={Junxian He and Xinyi Wang and Graham Neubig and Taylor Berg-Kirkpatrick},
booktitle={Proceedings of ICLR},
year={2020}
}