5.10. 批量归一化.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "5.10. 批量归一化.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chongzicbo/Dive-into-Deep-Learning-tf.keras/blob/master/5.10.%20%E6%89%B9%E9%87%8F%E5%BD%92%E4%B8%80%E5%8C%96.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fq_Z_3XihSko",
        "colab_type": "text"
      },
      "source": [
        "##5.10. 批量归一化\n",
        "本节我们介绍批量归一化（batch normalization）层，它能让较深的神经网络的训练变得更加容易 。标准化处理输入数据使各个特征的分布相近：这往往更容易训练出有效的模型。\n",
        "\n",
        "通常来说，数据标准化预处理对于浅层模型就足够有效了。随着模型训练的进行，当每层中参数更新时，靠近输出层的输出较难出现剧烈变化。但对深层神经网络来说，即使输入数据已做标准化，训练中模型参数的更新依然很容易造成靠近输出层输出的剧烈变化。这种计算数值的不稳定性通常令我们难以训练出有效的深度模型。\n",
        "\n",
        "批量归一化的提出正是为了应对深度模型训练的挑战。在模型训练时，批量归一化利用小批量上的均值和标准差，不断调整神经网络中间输出，从而使整个神经网络在各层的中间输出的数值更稳定。批量归一化和下一节将要介绍的残差网络为训练和设计深度模型提供了两类重要思路。\n",
        "\n",
        "###5.10.1. 批量归一化层\n",
        "对全连接层和卷积层做批量归一化的方法稍有不同。下面我们将分别介绍这两种情况下的批量归一化。\n",
        "\n",
        "####5.10.1.1. 对全连接层做批量归一化\n",
        "我们先考虑如何对全连接层做批量归一化。通常，我们将批量归一化层置于全连接层中的仿射变换和激活函数之间。设全连接层的输入为 $u$ ，权重参数和偏差参数分别为 $W$ 和 $b$ ，激活函数为 $ϕ$ 。设批量归一化的运算符为 $BN$ 。那么，使用批量归一化的全连接层的输出为\n",
        "$$\\phi(\\text{BN}(\\boldsymbol{x})),$$\n",
        "其中批量归一化输入 $x$ 由仿射变换\n",
        "$$\\boldsymbol{x} = \\boldsymbol{W\\boldsymbol{u} + \\boldsymbol{b}}$$\n",
        "得到。考虑一个由 m 个样本组成的小批量，仿射变换的输出为一个新的小批量$\\mathcal{B} = \\{\\boldsymbol{x}^{(1)}, \\ldots, \\boldsymbol{x}^{(m)} \\}$。它们正是批量归一化层的输入。对于小批量 $B$ 中任意样本$\\boldsymbol{x}^{(i)} \\in \\mathbb{R}^d, 1 \\leq i \\leq m$，批量归一化层的输出同样是d维向量\n",
        "$$\\boldsymbol{y}^{(i)} = \\text{BN}(\\boldsymbol{x}^{(i)}),$$\n",
        "并由以下几步求得。首先，对小批量$ B$ 求均值和方差：\n",
        "$$\\boldsymbol{\\mu}_\\mathcal{B} \\leftarrow \\frac{1}{m}\\sum_{i = 1}^{m} \\boldsymbol{x}^{(i)},$$\n",
        "$$\\boldsymbol{\\sigma}_\\mathcal{B}^2 \\leftarrow \\frac{1}{m} \\sum_{i=1}^{m}(\\boldsymbol{x}^{(i)} - \\boldsymbol{\\mu}_\\mathcal{B})^2,$$\n",
        "其中的平方计算是按元素求平方。接下来，使用按元素开方和按元素除法对 $x^{(i)}$ 标准化：\n",
        "$$\\hat{\\boldsymbol{x}}^{(i)} \\leftarrow \\frac{\\boldsymbol{x}^{(i)} - \\boldsymbol{\\mu}_\\mathcal{B}}{\\sqrt{\\boldsymbol{\\sigma}_\\mathcal{B}^2 + \\epsilon}},$$\n",
        "\n",
        "这里 $\\epsilon>0$ 是一个很小的常数，保证分母大于0。在上面标准化的基础上，批量归一化层引入了两个可以学习的模型参数，拉伸（scale）参数  $\\gamma$  和偏移（shift）参数  $\\beta$。这两个参数和 $x^{(i)}$ 形状相同，皆为 d 维向量。它们与 $\\hat{x}^{(i)}$ 分别做按元素乘法（符号 ⊙ ）和加法计算：\n",
        "$${\\boldsymbol{y}}^{(i)} \\leftarrow \\boldsymbol{\\gamma} \\odot \\hat{\\boldsymbol{x}}^{(i)} + \\boldsymbol{\\beta}.$$\n",
        "\n",
        "至此，我们得到了 $x^{(i)}$ 的批量归一化的输出 $y^{(i)}$ 。 值得注意的是，可学习的拉伸和偏移参数保留了不对 $x^{(i)}$ 做批量归一化的可能：此时只需学出 $\\boldsymbol{\\gamma} = \\sqrt{\\boldsymbol{\\sigma}_\\mathcal{B}^2 + \\epsilon}$和$\\boldsymbol{\\beta} = \\boldsymbol{\\mu}_\\mathcal{B}$。我们可以对此这样理解：如果批量归一化无益，理论上，学出的模型可以不使用批量归一化。\n",
        "\n",
        "####5.10.1.2. 对卷积层做批量归一化\n",
        "对卷积层来说，批量归一化发生在卷积计算之后、应用激活函数之前。如果卷积计算输出多个通道，我们需要对这些通道的输出分别做批量归一化，且每个通道都拥有独立的拉伸和偏移参数，并均为标量。设小批量中有 $m$ 个样本。在单个通道上，假设卷积计算输出的高和宽分别为 $p$ 和 $q$ 。我们需要对该通道中 $m \\times p \\times q$ 个元素同时做批量归一化。对这些元素做标准化计算时，我们使用相同的均值和方差，即该通道中 $m \\times p \\times q$ 个元素的均值和方差。\n",
        "\n",
        "####5.10.1.3. 预测时的批量归一化\n",
        "使用批量归一化训练时，我们可以将批量大小设得大一点，从而使批量内样本的均值和方差的计算都较为准确。将训练好的模型用于预测时，我们希望模型对于任意输入都有确定的输出。因此，单个样本的输出不应取决于批量归一化所需要的随机小批量中的均值和方差。一种常用的方法是通过移动平均估算整个训练数据集的样本均值和方差，并在预测时使用它们得到确定的输出。可见，和丢弃层一样，批量归一化层在训练模式和预测模式下的计算结果也是不一样的。\n",
        "\n",
        "###5.10.2. 从零开始实现\n",
        "下面我们来实现批量归一化层。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PJYSYoCkDDgE",
        "colab_type": "code",
        "outputId": "6817f4f8-d255-445d-fd35-65b16416e045",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 62
        }
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers,Sequential\n",
        "from tensorflow.data import Dataset\n",
        "from tensorflow.keras import losses,optimizers\n",
        "import time\n",
        "import numpy as np\n",
        "from tensorflow import image\n",
        "import os\n",
        "import sys"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "koL80ypgEWMU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.enable_eager_execution() #启用动态图计算，tf2.x不用调用该函数"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XMgwVUUMEZf5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def batch_norm(X,gamma,beta,moving_mean,moving_var,eps,momentum,is_training=True):\n",
        "  if not is_training:\n",
        "    #如果是预测模式下，直接使用传入的移动平均所得的均值和方差\n",
        "    X_hat=(X-moving_mean)/tf.sqrt(moving_var+eps)\n",
        "  else:\n",
        "    assert len(X.shape) in (2,4)\n",
        "\n",
        "    if len(X.shape)==2:\n",
        "      #使用全连接层的情况，计算特征维上的均值和方差\n",
        "      mean=tf.reduce_mean(X,axis=0)\n",
        "      var=tf.reduce_mean(tf.square(X-mean),axis=0)\n",
        "    else:\n",
        "      #使用二维卷积层的情况，计算通道维上(axis=-1)的均值和方差，这里我们需要保持X的形状以便后面可以做广播运算\n",
        "      mean=tf.reduce_mean(X,axis=(0,1,2),keepdims=True)\n",
        "      var=tf.reduce_mean(tf.square((X-mean)),axis=(0,1,2),keepdims=True)\n",
        "    #训练模式下调用当前的均值和方差做标准化\n",
        "    X_hat=(X-mean)/tf.sqrt(var+eps)\n",
        "    #更新移动平均的均值和方差\n",
        "    moving_mean=momentum*moving_mean+(1.0-momentum)*mean\n",
        "    moving_var=momentum*moving_var+(1.0-momentum)*var\n",
        "  Y=gamma*X_hat+beta #拉伸和偏移\n",
        "  return Y,moving_mean,moving_var"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a4ytHsAJk-Wp",
        "colab_type": "text"
      },
      "source": [
        "接下来，我们自定义一个BatchNorm层。它保存参与求梯度和迭代的拉伸参数gamma和偏移参数beta，同时也维护移动平均得到的均值和方差，以便能够在模型预测时被使用。BatchNorm实例所需指定的num_features参数对于全连接层来说应为输出个数，对于卷积层来说则为输出通道数。该实例所需指定的num_dims参数对于全连接层和卷积层来说分别为2和4。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jGye2dVNI3_y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class BatchNorm(layers.Layer):\n",
        "  def __init__(self,num_features,num_dims):\n",
        "    super(BatchNorm,self).__init__()\n",
        "    if num_dims==2:\n",
        "      shape=(1,num_features)\n",
        "    else:\n",
        "      shape=(1,1,1,num_features)\n",
        "    #参与求梯度和迭代的拉伸和偏移参数，分布初始化为1和0\n",
        "    self.gamma=tf.Variable(tf.constant(1,shape=shape,dtype=tf.float32))\n",
        "    self.beta=tf.Variable(tf.zeros(shape=shape,dtype=tf.float32))\n",
        "\n",
        "    #不参与求梯度和迭代的变量，初始化成0\n",
        "    self.moving_mean=tf.zeros(shape=shape)\n",
        "    self.moving_var=tf.zeros(shape=shape)\n",
        "\n",
        "  def call(self,X):\n",
        "    Y,self.moving_mean,self.moving_var=batch_norm(X,self.gamma,self.beta,self.moving_mean,self.moving_var,eps=1e-5,momentum=0.8)\n",
        "    return Y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ro-UypNlJ3s",
        "colab_type": "text"
      },
      "source": [
        "####5.10.2.1. 使用批量归一化层的LeNet\n",
        "下面我们修改“卷积神经网络（LeNet）”这一节介绍的LeNet模型，从而应用批量归一化层。我们在所有的卷积层或全连接层之后、激活层之前加入批量归一化层。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_z9dyfhMLgKj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "net=Sequential([\n",
        "  layers.Conv2D(6,kernel_size=4),\n",
        "  BatchNorm(6,num_dims=4),\n",
        "  layers.Activation('sigmoid'),\n",
        "  layers.MaxPool2D(pool_size=2,strides=2),\n",
        "  layers.Conv2D(16,kernel_size=5),\n",
        "  BatchNorm(16,num_dims=4),\n",
        "  layers.Activation('sigmoid'),\n",
        "  layers.MaxPool2D(pool_size=2,strides=2), \n",
        "  layers.Flatten(),\n",
        "  layers.Dense(120),\n",
        "  BatchNorm(120,num_dims=2),\n",
        "  layers.Activation('sigmoid'),\n",
        "  layers.Dense(84),\n",
        "  BatchNorm(84,num_dims=2),\n",
        "  layers.Activation('sigmoid'),\n",
        "  layers.Dense(10)\n",
        "]\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T4OEIv7GcLan",
        "colab_type": "code",
        "outputId": "143e64c0-0b73-4adc-f07b-b1189b5d69c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 622
        }
      },
      "source": [
        "X=tf.random.uniform(shape=(1,224,224,3))\n",
        "for layer in net.layers:\n",
        "  # print(type(layer))\n",
        "  # print(layer.name)\n",
        "  X=layer(X)\n",
        "  print(layer.name,'output shape:\\t',X.shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "conv2d\n",
            "conv2d output shape:\t (1, 221, 221, 6)\n",
            "batch_norm\n",
            "(1, 221, 221, 6)\n",
            "batch_norm output shape:\t (1, 221, 221, 6)\n",
            "activation\n",
            "activation output shape:\t (1, 221, 221, 6)\n",
            "max_pooling2d\n",
            "max_pooling2d output shape:\t (1, 110, 110, 6)\n",
            "conv2d_1\n",
            "conv2d_1 output shape:\t (1, 106, 106, 16)\n",
            "batch_norm_1\n",
            "(1, 106, 106, 16)\n",
            "batch_norm_1 output shape:\t (1, 106, 106, 16)\n",
            "activation_1\n",
            "activation_1 output shape:\t (1, 106, 106, 16)\n",
            "max_pooling2d_1\n",
            "max_pooling2d_1 output shape:\t (1, 53, 53, 16)\n",
            "flatten\n",
            "flatten output shape:\t (1, 44944)\n",
            "dense\n",
            "dense output shape:\t (1, 120)\n",
            "batch_norm_2\n",
            "(1, 120)\n",
            "batch_norm_2 output shape:\t (1, 120)\n",
            "activation_2\n",
            "activation_2 output shape:\t (1, 120)\n",
            "dense_1\n",
            "dense_1 output shape:\t (1, 84)\n",
            "batch_norm_3\n",
            "(1, 84)\n",
            "batch_norm_3 output shape:\t (1, 84)\n",
            "activation_3\n",
            "activation_3 output shape:\t (1, 84)\n",
            "dense_2\n",
            "dense_2 output shape:\t (1, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_giQUnzicg3S",
        "colab_type": "code",
        "outputId": "66d3363d-2568-4280-fe05-265ae9da14df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        }
      },
      "source": [
        "y=net(tf.random.uniform(shape=(1,224,224,3)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1, 221, 221, 6)\n",
            "(1, 106, 106, 16)\n",
            "(1, 120)\n",
            "(1, 84)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_cCF--8lSBe",
        "colab_type": "text"
      },
      "source": [
        "下面我们训练修改后的模型。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V-_vl1KDX9-a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr,num_epochs,batch_size=0.05,5,128\n",
        "optimizer=optimizers.SGD(learning_rate=lr)\n",
        "loss=losses.SparseCategoricalCrossentropy(from_logits=True)\n",
        "buffer_size=1000\n",
        "def load_data_fashion_mnist(batch_size,buffer_size):\n",
        "  (x_train,y_train),(x_test,y_test)=keras.datasets.fashion_mnist.load_data()\n",
        "  x_train=x_train[:,:,:,np.newaxis]#将三维张量增加一个channel维\n",
        "  x_test=x_test[:,:,:,np.newaxis]\n",
        "  train_iter=Dataset.from_tensor_slices((x_train,y_train)).map(lambda x,y:(x/255,y)).shuffle(buffer_size).batch(batch_size).map(lambda x,y:(image.resize(images=x,size=(224,224)),y))\n",
        "  test_iter=Dataset.from_tensor_slices((x_test,y_test)).map(lambda x,y:(x/255,y)).batch(batch_size).map(lambda x,y:(image.resize(images=x,size=(224,224)),y))\n",
        "  return train_iter,test_iter\n",
        "train_iter,test_iter=load_data_fashion_mnist(batch_size=batch_size,buffer_size=buffer_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zzRtz2JvadRk",
        "colab_type": "code",
        "outputId": "31948b31-10a7-4894-d1c6-6986414d01c6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "net.compile(optimizer=optimizer,loss=loss,metrics=['accuracy'])\n",
        "history=net.fit_generator(train_iter,validation_data=test_iter,epochs=num_epochs)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "468/469 [============================>.] - ETA: 0s - loss: 0.6898 - acc: 0.7783Epoch 1/5\n",
            "469/469 [==============================] - 170s 362ms/step - loss: 0.6893 - acc: 0.7785 - val_loss: 0.4985 - val_acc: 0.8527\n",
            "Epoch 2/5\n",
            "468/469 [============================>.] - ETA: 0s - loss: 0.4316 - acc: 0.8640Epoch 1/5\n",
            "469/469 [==============================] - 163s 349ms/step - loss: 0.4315 - acc: 0.8640 - val_loss: 0.4048 - val_acc: 0.8715\n",
            "Epoch 3/5\n",
            "468/469 [============================>.] - ETA: 0s - loss: 0.3536 - acc: 0.8830Epoch 1/5\n",
            "469/469 [==============================] - 164s 349ms/step - loss: 0.3536 - acc: 0.8830 - val_loss: 0.3602 - val_acc: 0.8795\n",
            "Epoch 4/5\n",
            "468/469 [============================>.] - ETA: 0s - loss: 0.3088 - acc: 0.8966Epoch 1/5\n",
            "469/469 [==============================] - 163s 349ms/step - loss: 0.3088 - acc: 0.8966 - val_loss: 0.3359 - val_acc: 0.8849\n",
            "Epoch 5/5\n",
            "468/469 [============================>.] - ETA: 0s - loss: 0.2746 - acc: 0.9067Epoch 1/5\n",
            "469/469 [==============================] - 164s 349ms/step - loss: 0.2746 - acc: 0.9067 - val_loss: 0.3152 - val_acc: 0.8912\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ITZRJ5IPafY5",
        "colab_type": "text"
      },
      "source": [
        "###5.10.3. 小结\n",
        "* 在模型训练时，批量归一化利用小批量上的均值和标准差，不断调整神经网络的中间输出，从而使整个神经网络在各层的中间输出的数值更稳定。\n",
        "* 对全连接层和卷积层做批量归一化的方法稍有不同。\n",
        "* 批量归一化层和丢弃层一样，在训练模式和预测模式的计算结果是不一样的。"
      ]
    }
  ]
}