7.1 Training RNN for music generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating music with RNN's the simple way\n",
    "\n",
    "Whilst most of the RNN applications required large amount of training time and resources, music generation is surprisingly easy to achieve *good* results. In this tutorial heavily based on \\cite{blog} we will apply simple RNN's networks to produce traditional folk tunes.\n",
    "\n",
    "## The data\n",
    "\n",
    "Perhaps the most important question we must answer before getting in greater detail is which data representation we will use, since this will determine the complexity of the model required. Here, since we don't want to waste weeks training our models we will use a very simple approach: The ABC notation.\n",
    "\n",
    "### ABC notation\n",
    "\n",
    "Citing from the wikipedia \\cite{wikipedia}:\n",
    "\n",
    ">ABC notation is a shorthand form of musical notation. In basic form it uses the letters A through G to represent the given notes, with other elements used to place added value on these - sharp, flat, the length of the note, key, ornamentation. Lines in the first part of the tune notation, beginning with a letter followed by a colon, indicate various aspects of the tune such as the index, when there are more than one tune in a file (X:), the title (T:), the time signature (M:), the default note length (L:), the type of tune (R:) and the key (K:). Lines following the key designation represent the tune. It can be translated into traditional music notation using one of the abc conversion tools.\n",
    "\n",
    "For instance here we have an example of \"Greensleves\" written in ABC notation\n",
    "\n",
    "``` abc\n",
    "X:870\n",
    "T:Greensleeves\n",
    "C:anon.\n",
    "O:England\n",
    "R:Broadside ballad\n",
    "Z:Transcribed by Frank Nordberg - http://www.musicaviva.com\n",
    "F:http://abc.musicaviva.com/tunes/england/greensleeves-dorian.abc\n",
    "M:6/4\n",
    "L:1/4\n",
    "Q:1/2=110\n",
    "K:Gdor\n",
    "G|\"Gm\"B2c d>ed|\"F\"c2A F>GA|\"Gm\"B2A G>^FG|\"Dm\"A2^F D2G|\n",
    "\"Gm\"B2c d>ed|\"F\"c2A F>GA|\"Gm\"B>AG \"D\"^F>EF|\"Gm\"G3 G2z|\n",
    "\"Bb\"f3 f>ed|\"F\"c2A \"Dm\"F>GA|\"Gm\"B2G G>^FG|\"Dm\"A2^F D2z|\n",
    "\"Bb\"f3 f>ed|\"F\"c2A \"Dm\"F>GA|\"Gm\"B>AG \"D\"^F>EF|\"G\"G3 G2|]\n",
    "```\n",
    "\n",
    "This notation is great because it enables to generate music only with the common alphabet set, without complicated notations and symbols. Also, a vast amount of tunes is available in the internets, which is important for learning.\n",
    "\n",
    "## The model\n",
    "\n",
    "For this exercise we will be using a simple implementation of rnn for text called char-rnn, originally developed by the omnipresent Karpathy. An implementation written in tersorflow is available at github.\n",
    "\n",
    "It has two scripts, one for training and one for sampling. We can see the many options available calling them with *-h*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/sherjilozair/char-rnn-tensorflow.git\n",
    "!python ./char-rnn-tensorflow/train.py -h\n",
    "!python ./char-rnn-tensorflow/sample.py -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With regard to the hyperparameters by default, if we take a look into train.py we can see which ones it uses.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    " parser.add_argument('--data_dir', type=str, default='data/tinyshakespeare',\n",
    "                       help='data directory containing input.txt')\n",
    "    parser.add_argument('--save_dir', type=str, default='save',\n",
    "                       help='directory to store checkpointed models')\n",
    "    parser.add_argument('--rnn_size', type=int, default=128,\n",
    "                       help='size of RNN hidden state')\n",
    "    parser.add_argument('--num_layers', type=int, default=2,\n",
    "                       help='number of layers in the RNN')\n",
    "    parser.add_argument('--model', type=str, default='lstm',\n",
    "                       help='rnn, gru, or lstm')\n",
    "    parser.add_argument('--batch_size', type=int, default=50,\n",
    "                       help='minibatch size')\n",
    "    parser.add_argument('--seq_length', type=int, default=50,\n",
    "                       help='RNN sequence length')\n",
    "    parser.add_argument('--num_epochs', type=int, default=50,\n",
    "                       help='number of epochs')\n",
    "    parser.add_argument('--save_every', type=int, default=1000,\n",
    "                       help='save frequency')\n",
    "    parser.add_argument('--grad_clip', type=float, default=5.,\n",
    "                       help='clip gradients at this value')\n",
    "    parser.add_argument('--learning_rate', type=float, default=0.002,\n",
    "                       help='learning rate')\n",
    "    parser.add_argument('--decay_rate', type=float, default=0.97,\n",
    "                       help='decay rate for rmsprop')                       \n",
    "    parser.add_argument('--init_from', type=str, default=None,\n",
    "                       help=\"\"\"continue training from saved model at this path. Path must contain files saved by previous training process: \n",
    "                            'config.pkl'        : configuration;\n",
    "                            'chars_vocab.pkl'   : vocabulary definitions;\n",
    "                            'checkpoint'        : paths to model file(s) (created by tf).\n",
    "                                                  Note: this file contains absolute paths, be careful when moving files around;\n",
    "                            'model.ckpt-*'      : file(s) with model definition (created by tf)\n",
    "                        \"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how we can especify the size of the hidden state, the number of layers, the number of steps used for training and the typical stuff related to SGD, batch size, learning rate and others. Those parameters are used later to build the network inside model.py. The relevant part is shown commented below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First we chose the type of RNN according the parameters provided. The default value is lstm, yet we will use gru.\n",
    "if args.model == 'rnn':\n",
    "    cell_fn = rnn_cell.BasicRNNCell\n",
    "elif args.model == 'gru':\n",
    "    cell_fn = rnn_cell.GRUCell\n",
    "elif args.model == 'lstm':\n",
    "    cell_fn = rnn_cell.BasicLSTMCell\n",
    "else:\n",
    "    raise Exception(\"model type not supported: {}\".format(args.model))\n",
    "\n",
    "# The cell is initialied and replicated as layers have been determined. This value is 128 x 2 layers in the defaults\n",
    "cell = cell_fn(args.rnn_size)\n",
    "\n",
    "self.cell = cell = rnn_cell.MultiRNNCell([cell] * args.num_layers)\n",
    "\n",
    "# The placeholders of the network are initialized, along with the initial state.\n",
    "self.input_data = tf.placeholder(tf.int32, [args.batch_size, args.seq_length])\n",
    "self.targets = tf.placeholder(tf.int32, [args.batch_size, args.seq_length])\n",
    "self.initial_state = cell.zero_state(args.batch_size, tf.float32)\n",
    "\n",
    "# The output softmax weights and biases are built here\n",
    "with tf.variable_scope('rnnlm'):\n",
    "    softmax_w = tf.get_variable(\"softmax_w\", [args.rnn_size, args.vocab_size])\n",
    "    softmax_b = tf.get_variable(\"softmax_b\", [args.vocab_size])\n",
    "    with tf.device(\"/cpu:0\"):\n",
    "        # The embedding is created here, mapping the inputs to the state of the rnn\n",
    "        embedding = tf.get_variable(\"embedding\", [args.vocab_size, args.rnn_size])\n",
    "        inputs = tf.split(1, args.seq_length, tf.nn.embedding_lookup(embedding, self.input_data))\n",
    "        inputs = [tf.squeeze(input_, [1]) for input_ in inputs]\n",
    "\n",
    "# The loop function passes does a forward pass of the data through the RNN cells        \n",
    "def loop(prev, _):\n",
    "    prev = tf.matmul(prev, softmax_w) + softmax_b\n",
    "    prev_symbol = tf.stop_gradient(tf.argmax(prev, 1))\n",
    "    return tf.nn.embedding_lookup(embedding, prev_symbol)\n",
    "\n",
    "# We get the final output of the network\n",
    "outputs, last_state = seq2seq.rnn_decoder(inputs, self.initial_state, cell, loop_function=loop if infer else None, scope='rnnlm')\n",
    "output = tf.reshape(tf.concat(1, outputs), [-1, args.rnn_size])\n",
    "# We input the output of the cell to the softmax\n",
    "self.logits = tf.matmul(output, softmax_w) + softmax_b\n",
    "self.probs = tf.nn.softmax(self.logits)\n",
    "# We set the loss\n",
    "loss = seq2seq.sequence_loss_by_example([self.logits],\n",
    "        [tf.reshape(self.targets, [-1])],\n",
    "        [tf.ones([args.batch_size * args.seq_length])],\n",
    "        args.vocab_size)\n",
    "# The loss is averaged\n",
    "self.cost = tf.reduce_sum(loss) / args.batch_size / args.seq_length\n",
    "self.final_state = last_state\n",
    "# Learning rate variable\n",
    "self.lr = tf.Variable(0.0, trainable=False)\n",
    "tvars = tf.trainable_variables()\n",
    "# Gradient clipping\n",
    "grads, _ = tf.clip_by_global_norm(tf.gradients(self.cost, tvars),\n",
    "        args.grad_clip)\n",
    "# Optimize using Adam\n",
    "optimizer = tf.train.AdamOptimizer(self.lr)\n",
    "self.train_op = optimizer.apply_gradients(zip(grads, tvars))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we don't want to mess too much with the hyperparameters, we will take the default ones as good enough, with the exception of using GRU instead of LTSM since the amount of data is not large. First we need the data. We will use this (http://www.norbeck.nu/abc/hn201602.zip), but any abc tune collection is equally good. After concat all the files in one we are ready to train. Further cleaning is possible, and it will improve the resulting generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!wget -N http://www.norbeck.nu/abc/hn201602.zip\n",
    "!unzip hn201602.zip\n",
    "!mv ./s/* ./i\n",
    "!cat ./i/*abc > ./char-rnn-tensorflow/data/input.txt\n",
    "!tail -n 100 ./char-rnn-tensorflow/data/input.txt\n",
    "!ls ./char-rnn-tensorflow/data\n",
    "!rm -R ./s ./i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "\n",
    "We are ready to start training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!python char-rnn-tensorflow/train.py --model gru --data_dir ./char-rnn-tensorflow/data --save_dir ./char-rnn-tensorflow/save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}