"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "train_concise(3)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ffb438b9",
+ "metadata": {},
+ "source": [
+ "#### 练习（答案仅供参考）\n",
+ "1. 在本节的估计问题中使用$\\lambda$的值进行实验。绘制训练和测试精度关于$\\lambda$的函数。你观察到了什么？\n",
+ "随lambda增大精度先降低后不收敛，"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5f48dcd",
+ "metadata": {},
+ "source": [
+ "[评论区](#https://zh-v2.d2l.ai/chapter_multilayer-perceptrons/weight-decay.html)有其他同学发布的图片"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f6f883dd",
+ "metadata": {},
+ "source": [
+ "2. 使用验证集来找到最佳值$\\lambda$。它真的是最优值吗？这有关系吗？\n",
+ "
不能确定是否是最优值，但没关系，只要足够优即可\n",
+ "1. 如果我们使用$\\sum_i |w_i|$作为我们选择的惩罚（$L_1$正则化），那么更新方程会是什么样子？\n",
+ "
$\\frac{\\partial{}}{\\partial{\\mathbf{w}}}(\\ell(\\mathbf{w},b)+\\lambda||\\mathbf{w}||_1)=\\frac{\\partial{\\ell(\\mathbf{w},b)}}{{\\partial{\\mathbf{w}}}}+\\lambda I'$\n",
+ "
其中$I'=(a_1,...,a_n)$,当$\\mathbf{w}$中第$i$个元素为正时$a_i=1$，反之$a_i=-1$.（=0时随意）
代入公式化简得$\\mathbf{w}_{t+1}=\\mathbf{w}_{t}-\\eta\\frac{\\partial{\\ell(\\mathbf{w}_t,b_t)}}{{\\partial{\\mathbf{w}_{t}}}}-\\eta\\lambda I'$\n",
+ "
从这个式子可以看出使用L1正则化时只能对所有同号的参数施加一个相同大小的正则项（增减一个定值），而反观L2正则化对参数的影响是与参数本身的值有关的（乘上一个系数）似乎是更好的选择。不过L1正则化在特征提取上会有用处。\n",
+ "1. 我们知道$\\|\\mathbf{w}\\|^2 = \\mathbf{w}^\\top \\mathbf{w}$。你能找到类似的矩阵方程吗（见 :numref:`subsec_lin-algebra-norms` 中的Frobenius范数）？
这个不太理解题意，可能是说对称矩阵？\n",
+ "1. 回顾训练误差和泛化误差之间的关系。除了权重衰减、增加训练数据、使用适当复杂度的模型之外，你还能想出其他什么方法来处理过拟合？
EarlyStopping等\n",
+ "1. 在贝叶斯统计中，我们使用先验和似然的乘积，通过公式$P(w \\mid x) \\propto P(x \\mid w) P(w)$得到后验。如何得到带正则化的$P(w)$？\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6d395f2b",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git "a/code/13-\344\270\242\345\274\203\346\263\225.ipynb" "b/code/13-\344\270\242\345\274\203\346\263\225.ipynb"
new file mode 100644
index 0000000..356599e
--- /dev/null
+++ "b/code/13-\344\270\242\345\274\203\346\263\225.ipynb"
@@ -0,0 +1,76 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "d5195177",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from d2l import torch as d2l\n",
+ "\n",
+ "def dropout_layer (X,dropout): #X为dropout层的输入，dropout为设置的丢弃概率\n",
+ " assert 0<=dropout<=1 #丢弃概率介于0，1之间\n",
+ " if dropout == 1:\n",
+ " return torch.zeros_like(X) #若丢弃概率为1，则X的全部项均被置0\n",
+ " if dropout == 0:\n",
+ " return X #若丢弃概率为0，不对X作丢弃操作，直接返回X\n",
+ " mask=(torch.Tensor(X.shape).uniform_(0,1)>dropout).float() #用uniform函数生成0-1间的随机实数，利用”>\"，将大于dropout的记为1，小于dropout的记为0，实现丢弃操作\n",
+ " return mask*X/(1-dropout) #将mask与X相乘实现丢弃操作，并除以(1-dropout)，这里不使用选中X中元素置0的原因是相乘操作相比选中操作更快"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "8ff5ba10",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensor([[ 0., 1., 2., 3., 4., 5., 6., 7.],\n",
+ " [ 8., 9., 10., 11., 12., 13., 14., 15.]])\n",
+ "tensor([[ 0., 1., 2., 3., 4., 5., 6., 7.],\n",
+ " [ 8., 9., 10., 11., 12., 13., 14., 15.]])\n",
+ "tensor([[ 0., 0., 4., 0., 0., 10., 0., 14.],\n",
+ " [16., 18., 20., 0., 0., 26., 0., 30.]])\n",
+ "tensor([[0., 0., 0., 0., 0., 0., 0., 0.],\n",
+ " [0., 0., 0., 0., 0., 0., 0., 0.]])\n"
+ ]
+ }
+ ],
+ "source": [
+ "#丢弃法测试\n",
+ "X=torch.arange(16,dtype=torch.float32).reshape((2,8))\n",
+ "print(X)\n",
+ "print(dropout_layer (X,0.)) #丢弃率设置为0\n",
+ "print(dropout_layer (X,0.5)) #丢弃率设置为0.5\n",
+ "print(dropout_layer (X,1)) #丢弃率设置为1"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python [conda env:py36torch040]",
+ "language": "python",
+ "name": "conda-env-py36torch040-py"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git "a/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/custom-layer.ipynb" "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/custom-layer.ipynb"
new file mode 100644
index 0000000..2fbfe73
--- /dev/null
+++ "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/custom-layer.ipynb"
@@ -0,0 +1,352 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 自定义层\n",
+ "\n",
+ "深度学习成功背后的一个因素是神经网络的灵活性：\n",
+ "我们可以用创造性的方式组合不同的层，从而设计出适用于各种任务的架构。\n",
+ "例如，研究人员发明了专门用于处理图像、文本、序列数据和执行动态规划的层。\n",
+ "未来，你会遇到或要自己发明一个现在在深度学习框架中还不存在的层。\n",
+ "在这些情况下，你必须构建自定义层。在本节中，我们将向你展示如何构建。\n",
+ "\n",
+ "## 不带参数的层\n",
+ "\n",
+ "首先，我们(**构造一个没有任何参数的自定义层**)。\n",
+ "如果你还记得我们在 :numref:`sec_model_construction`对块的介绍，\n",
+ "这应该看起来很眼熟。\n",
+ "下面的`CenteredLayer`类要从其输入中减去均值。\n",
+ "要构建它，我们只需继承基础层类并实现前向传播功能。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "import torch.nn.functional as F\n",
+ "from torch import nn\n",
+ "\n",
+ "\n",
+ "class CenteredLayer(nn.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ "\n",
+ " def forward(self, X):\n",
+ " return X - X.mean()#使用了广播机制"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 4
+ },
+ "source": [
+ "让我们向该层提供一些数据，验证它是否能按预期工作。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([-2., -1., 0., 1., 2.])"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "layer = CenteredLayer()\n",
+ "layer(torch.FloatTensor([1, 2, 3, 4, 5]))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 8
+ },
+ "source": [
+ "现在，我们可以[**将层作为组件合并到更复杂的模型中**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 10,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "net = nn.Sequential(nn.Linear(8, 128), CenteredLayer())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 12
+ },
+ "source": [
+ "作为额外的健全性检查，我们可以在向该网络发送随机数据后，检查均值是否为0。\n",
+ "由于我们处理的是浮点数，因为存储精度的原因，我们仍然可能会看到一个非常小的非零数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "origin_pos": 14,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor(0., grad_fn=)"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Y = net(torch.rand(4, 8))\n",
+ "Y.mean()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 16
+ },
+ "source": [
+ "## [**带参数的层**]\n",
+ "\n",
+ "以上我们知道了如何定义简单的层，下面我们继续定义具有参数的层，\n",
+ "这些参数可以通过训练进行调整。\n",
+ "我们可以使用内置函数来创建参数，这些函数提供一些基本的管理功能。\n",
+ "比如管理访问、初始化、共享、保存和加载模型参数。\n",
+ "这样做的好处之一是：我们不需要为每个自定义层编写自定义的序列化程序。\n",
+ "\n",
+ "现在，让我们实现自定义版本的全连接层。\n",
+ "回想一下，该层需要两个参数，一个用于表示权重，另一个用于表示偏置项。\n",
+ "在此实现中，我们使用修正线性单元作为激活函数。\n",
+ "该层需要输入参数：`in_units`和`units`，分别表示输入数和输出数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "origin_pos": 18,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "class MyLinear(nn.Module):\n",
+ " def __init__(self, in_units, units):\n",
+ " super().__init__()\n",
+ " #实例化Parameter的同时传入随机值作为初始值\n",
+ " self.weight = nn.Parameter(torch.randn(in_units, units))\n",
+ " self.bias = nn.Parameter(torch.randn(units,))\n",
+ " def forward(self, X):\n",
+ " linear = torch.matmul(X, self.weight.data) + self.bias.data#通过.data获取tensor形式的参数\n",
+ " return F.relu(linear)#在前向传播中使用ReLU激活"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 21,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "接下来，我们实例化`MyLinear`类并访问其模型参数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "origin_pos": 23,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Parameter containing:\n",
+ "tensor([[ 2.2578, -0.4571, -1.4185],\n",
+ " [ 1.6469, -0.4779, 1.0448],\n",
+ " [ 0.9764, 0.4048, 0.6039],\n",
+ " [ 0.9259, 1.2831, -0.4907],\n",
+ " [ 1.1768, -0.2245, 1.2883]], requires_grad=True)"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "linear = MyLinear(5, 3)\n",
+ "linear.weight"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 25
+ },
+ "source": [
+ "我们可以[**使用自定义层直接执行前向传播计算**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "origin_pos": 27,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[0.9983, 1.5834, 0.1999],\n",
+ " [2.7293, 0.3060, 1.1866]])"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "linear(torch.rand(2, 5))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 29
+ },
+ "source": [
+ "我们还可以(**使用自定义层构建模型**)，就像使用内置的全连接层一样使用自定义层。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "origin_pos": 31,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[0.],\n",
+ " [0.]])"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net = nn.Sequential(MyLinear(64, 8), MyLinear(8, 1))\n",
+ "net(torch.rand(2, 64))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 33
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* 我们可以通过基本层类设计自定义层。这允许我们定义灵活的新层，其行为与深度学习框架中的任何现有层不同。\n",
+ "* 在自定义层定义完成后，我们就可以在任意环境和网络架构中调用该自定义层。\n",
+ "* 层可以有局部参数，这些参数可以通过内置函数创建。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 设计一个接受输入并计算张量降维的层，它返回$y_k = \\sum_{i, j} W_{ijk} x_i x_j$。\n",
+ "1. 设计一个返回输入数据的傅立叶系数前半部分的层。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 35,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1835)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git "a/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/model-construction.ipynb" "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/model-construction.ipynb"
new file mode 100644
index 0000000..393d0f4
--- /dev/null
+++ "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/model-construction.ipynb"
@@ -0,0 +1,643 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 层和块\n",
+ ":label:`sec_model_construction`\n",
+ "\n",
+ "之前首次介绍神经网络时，我们关注的是具有单一输出的线性模型。\n",
+ "在这里，整个模型只由一个输出。\n",
+ "注意，单个神经网络\n",
+ "（1）接受一些输入；\n",
+ "（2）生成相应的标量输出；\n",
+ "（3）具有一组相关 *参数*（parameters），更新这些参数可以优化某目标函数。\n",
+ "\n",
+ "然后，当考虑具有多个输出的网络时，\n",
+ "我们利用矢量化算法来描述整层神经元。\n",
+ "像单个神经元一样，层（1）接受一组输入，\n",
+ "（2）生成相应的输出，\n",
+ "（3）由一组可调整参数描述。\n",
+ "当我们使用softmax回归时，一个单层本身就是模型。\n",
+ "然而，即使我们随后引入了多层感知机，我们仍然可以认为该模型保留了上面所说的基本架构。\n",
+ "\n",
+ "对于多层感知机而言，整个模型及其组成层都是这种架构。\n",
+ "整个模型接受原始输入（特征），生成输出（预测），\n",
+ "并包含一些参数（所有组成层的参数集合）。\n",
+ "同样，每个单独的层接收输入（由前一层提供），\n",
+ "生成输出（到下一层的输入），并且具有一组可调参数，\n",
+ "这些参数根据从下一层反向传播的信号进行更新。\n",
+ "\n",
+ "事实证明，研究讨论“比单个层大”但“比整个模型小”的组件更有价值。\n",
+ "例如，在计算机视觉中广泛流行的ResNet-152架构就有数百层，\n",
+ "这些层是由*层组*（groups of layers）的重复模式组成。\n",
+ "这个ResNet架构赢得了2015年ImageNet和COCO计算机视觉比赛\n",
+ "的识别和检测任务 :cite:`He.Zhang.Ren.ea.2016`。\n",
+ "目前ResNet架构仍然是许多视觉任务的首选架构。\n",
+ "在其他的领域，如自然语言处理和语音，\n",
+ "层组以各种重复模式排列的类似架构现在也是普遍存在。\n",
+ "\n",
+ "为了实现这些复杂的网络，我们引入了神经网络*块*的概念。\n",
+ "*块*（block）可以描述单个层、由多个层组成的组件或整个模型本身。\n",
+ "使用块进行抽象的一个好处是可以将一些块组合成更大的组件，\n",
+ "这一过程通常是递归的，如 :numref:`fig_blocks`所示。\n",
+ "通过定义代码来按需生成任意复杂度的块，\n",
+ "我们可以通过简洁的代码实现复杂的神经网络。\n",
+ "\n",
+ "\n",
+ ":label:`fig_blocks`\n",
+ "\n",
+ "从编程的角度来看，块由*类*（class）表示。\n",
+ "它的任何子类都必须定义一个将其输入转换为输出的前向传播函数，\n",
+ "并且必须存储任何必需的参数。\n",
+ "注意，有些块不需要任何参数。\n",
+ "最后，为了计算梯度，块必须具有反向传播函数。\n",
+ "在定义我们自己的块时，由于自动微分（在 :numref:`sec_autograd` 中引入）\n",
+ "提供了一些后端实现，我们只需要考虑前向传播函数和必需的参数。\n",
+ "\n",
+ "在构造自定义块之前，(**我们先回顾一下多层感知机**)\n",
+ "（ :numref:`sec_mlp_concise` ）的代码。\n",
+ "下面的代码生成一个网络，其中包含一个具有256个单元和ReLU激活函数的全连接隐藏层，\n",
+ "然后是一个具有10个隐藏单元且不带激活函数的全连接输出层。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[ 0.0557, -0.2469, 0.0089, 0.0078, 0.1052, 0.1832, -0.0727, -0.4046,\n",
+ " 0.2297, -0.0177],\n",
+ " [ 0.1933, -0.1326, 0.1534, 0.1434, -0.0235, 0.1264, -0.1059, -0.3951,\n",
+ " 0.2697, -0.0394]], grad_fn=)"
+ ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from torch.nn import functional as F\n",
+ "\n",
+ "net = nn.Sequential(nn.Linear(20, 256), nn.ReLU(), nn.Linear(256, 10))\n",
+ "\n",
+ "X = torch.rand(2, 20)# 随机初始化X，X大小2 *20\n",
+ "net(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 5,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "在这个例子中，我们通过实例化`nn.Sequential`来构建我们的模型，\n",
+ "层的执行顺序是作为参数传递的。\n",
+ "简而言之，(**`nn.Sequential`定义了一种特殊的`Module`**)，\n",
+ "即在PyTorch中表示一个块的类，\n",
+ "它维护了一个由`Module`组成的有序列表。\n",
+ "注意，两个全连接层都是`Linear`类的实例，\n",
+ "`Linear`类本身就是`Module`的子类。\n",
+ "另外，到目前为止，我们一直在通过`net(X)`调用我们的模型来获得模型的输出。\n",
+ "这实际上是`net.__call__(X)`的简写。\n",
+ "这个前向传播函数非常简单：\n",
+ "它将列表中的每个块连接在一起，将每个块的输出作为下一个块的输入。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 7
+ },
+ "source": [
+ "## [**自定义块**]\n",
+ "\n",
+ "要想直观地了解块是如何工作的，最简单的方法就是自己实现一个。\n",
+ "在实现我们自定义块之前，我们简要总结一下每个块必须提供的基本功能：\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 9,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "1. 将输入数据作为其前向传播函数的参数。\n",
+ "1. 通过前向传播函数来生成输出。请注意，输出的形状可能与输入的形状不同。例如，我们上面模型中的第一个全连接的层接收一个20维的输入，但是返回一个维度为256的输出。\n",
+ "1. 计算其输出关于输入的梯度，可通过其反向传播函数进行访问。通常这是自动发生的。\n",
+ "1. 存储和访问前向传播计算所需的参数。\n",
+ "1. 根据需要初始化模型参数。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 10
+ },
+ "source": [
+ "在下面的代码片段中，我们从零开始编写一个块。\n",
+ "它包含一个多层感知机，其具有256个隐藏单元的隐藏层和一个10维输出层。\n",
+ "注意，下面的`MLP`类继承了表示块的类。\n",
+ "我们的实现只需要提供我们自己的构造函数（Python中的`__init__`函数）和前向传播函数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 12,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "class MLP(nn.Module):\n",
+ " # 重写__init__方法，用模型参数声明层，我们声明两个全连接的层\n",
+ " def __init__(self):\n",
+ " # 调用MLP的父类Module的构造函数来执行必要的初始化。\n",
+ " # 这样，在类实例化时也可以指定其他函数参数，例如模型参数params（稍后将介绍）\n",
+ " super().__init__()# 父类构造函数\n",
+ " self.hidden = nn.Linear(20, 256) # 隐藏层\n",
+ " self.out = nn.Linear(256, 10) # 输出层\n",
+ "\n",
+ " # 定义模型的前向传播，即如何根据输入X返回所需的模型输出\n",
+ " def forward(self, X):\n",
+ " # 注意，这里我们使用ReLU的函数版本，其在nn.functional模块中定义。\n",
+ " return self.out(F.relu(self.hidden(X)))# 构建网络"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 14
+ },
+ "source": [
+ "我们首先看一下前向传播函数，它以`X`作为输入，\n",
+ "计算带有激活函数的隐藏表示，并输出其未规范化的输出值。\n",
+ "在这个`MLP`实现中，两个层都是实例变量。\n",
+ "要了解这为什么是合理的，可以想象实例化两个多层感知机（`net1`和`net2`），\n",
+ "并根据不同的数据对它们进行训练。\n",
+ "当然，我们希望它们学到两种不同的模型。\n",
+ "\n",
+ "接着我们[**实例化多层感知机的层，然后在每次调用前向传播函数时调用这些层**]。\n",
+ "注意一些关键细节：\n",
+ "首先，我们定制的`__init__`函数通过`super().__init__()`\n",
+ "调用父类的`__init__`函数，\n",
+ "省去了重复编写模版代码的痛苦。\n",
+ "然后，我们实例化两个全连接层，\n",
+ "分别为`self.hidden`和`self.out`。\n",
+ "注意，除非我们实现一个新的运算符，\n",
+ "否则我们不必担心反向传播函数或参数初始化，\n",
+ "系统将自动生成这些。\n",
+ "\n",
+ "我们来试一下这个函数：\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 16,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[-0.1515, 0.0797, 0.0325, -0.1259, -0.1020, -0.0129, 0.0294, 0.0134,\n",
+ " -0.1724, 0.1959],\n",
+ " [-0.0858, 0.1295, 0.0952, -0.0216, 0.0174, 0.0534, 0.0705, 0.1240,\n",
+ " -0.1096, -0.0396]], grad_fn=)"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net = MLP() # 实例化MLP\n",
+ "net(X) # 将X输入网络进行运算"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 18
+ },
+ "source": [
+ "块的一个主要优点是它的多功能性。\n",
+ "我们可以子类化块以创建层（如全连接层的类）、\n",
+ "整个模型（如上面的`MLP`类）或具有中等复杂度的各种组件。\n",
+ "我们在接下来的章节中充分利用了这种多功能性，\n",
+ "比如在处理卷积神经网络时。\n",
+ "\n",
+ "## [**顺序块**]\n",
+ "\n",
+ "现在我们可以更仔细地看看`Sequential`类是如何工作的，\n",
+ "回想一下`Sequential`的设计是为了把其他模块串起来。\n",
+ "为了构建我们自己的简化的`MySequential`，\n",
+ "我们只需要定义两个关键函数：\n",
+ "\n",
+ "1. 一种将块逐个追加到列表中的函数。\n",
+ "1. 一种前向传播函数，用于将输入按追加块的顺序传递给块组成的“链条”。\n",
+ "\n",
+ "下面的`MySequential`类提供了与默认`Sequential`类相同的功能。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "origin_pos": 20,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "class MySequential(nn.Module):\n",
+ " # 重写__init__方法，将模型名称的序列作为参数传入\n",
+ " def __init__(self, *args):\n",
+ " super().__init__()# 父类构造函数\n",
+ " for idx, module in enumerate(args):\n",
+ " # 这里，module是Module子类的一个实例。我们把它保存在'Module'类的成员变量_modules中。module的类型是OrderedDict\n",
+ " self._modules[str(idx)] = module # 按顺序将module插入\n",
+ "\n",
+ " def forward(self, X):\n",
+ " # OrderedDict保证了按照成员添加的顺序遍历它们\n",
+ " for block in self._modules.values():# 遍历_modules的成员，顺序调用每个模块\n",
+ " X = block(X)# 迭代处理X\n",
+ " return X"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 23,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "`__init__`函数将每个模块逐个添加到有序字典`_modules`中。\n",
+ "你可能会好奇为什么每个`Module`都有一个`_modules`属性？\n",
+ "以及为什么我们使用它而不是自己定义一个Python列表？\n",
+ "简而言之，`_modules`的主要优点是：\n",
+ "在模块的参数初始化过程中，\n",
+ "系统知道在`_modules`字典中查找需要初始化参数的子块。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#测试如果改为用list存储各个层\n",
+ "class MySequential2(nn.Module):\n",
+ " def __init__(self,*args):\n",
+ " super().__init__()\n",
+ " self.ms=[]#存储所用List\n",
+ " for idx,module in enumerate(args):\n",
+ " self.ms.append(module)\n",
+ " def forward(self,X):\n",
+ " for block in self.ms:\n",
+ " X=block(X)\n",
+ " return X"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "MySequential(\n",
+ " (0): Linear(in_features=20, out_features=256, bias=True)\n",
+ " (1): ReLU()\n",
+ " (2): Linear(in_features=256, out_features=10, bias=True)\n",
+ ")\n",
+ "MySequential2()\n",
+ "MLP(\n",
+ " (hidden): Linear(in_features=20, out_features=256, bias=True)\n",
+ " (out): Linear(in_features=256, out_features=10, bias=True)\n",
+ ")\n",
+ "OrderedDict([('hidden', Linear(in_features=20, out_features=256, bias=True)), ('out', Linear(in_features=256, out_features=10, bias=True))])\n"
+ ]
+ }
+ ],
+ "source": [
+ "net=MySequential(nn.Linear(20,256),nn.ReLU(),nn.Linear(256,10))\n",
+ "net2=MySequential2(nn.Linear(20,256),nn.ReLU(),nn.Linear(256,10))\n",
+ "print(net)#能输出\n",
+ "print(net2)#输出为空\n",
+ "net=MLP()\n",
+ "print(net)#输出成功\n",
+ "print(net._modules)#自动生成了_modules有序字典"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 24
+ },
+ "source": [
+ "当`MySequential`的前向传播函数被调用时，\n",
+ "每个添加的块都按照它们被添加的顺序执行。\n",
+ "现在可以使用我们的`MySequential`类重新实现多层感知机。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "origin_pos": 26,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[ 0.2252, 0.0363, -0.1693, 0.0870, 0.0077, -0.0833, 0.0181, 0.0321,\n",
+ " 0.2099, -0.2846],\n",
+ " [ 0.0971, 0.0553, -0.1123, 0.1500, 0.0153, -0.0618, -0.0417, -0.0683,\n",
+ " 0.1763, -0.1938]], grad_fn=)"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net = MySequential(nn.Linear(20, 256), nn.ReLU(), nn.Linear(256, 10)) # 实例化MySequential\n",
+ "net(X) # 将X输入网络进行运算"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 28
+ },
+ "source": [
+ "请注意，`MySequential`的用法与之前为`Sequential`类编写的代码相同\n",
+ "（如 :numref:`sec_mlp_concise` 中所述）。\n",
+ "\n",
+ "## [**在前向传播函数中执行代码**]\n",
+ "\n",
+ "`Sequential`类使模型构造变得简单，\n",
+ "允许我们组合新的架构，而不必定义自己的类。\n",
+ "然而，并不是所有的架构都是简单的顺序架构。\n",
+ "当需要更强的灵活性时，我们需要定义自己的块。\n",
+ "例如，我们可能希望在前向传播函数中执行Python的控制流。\n",
+ "此外，我们可能希望执行任意的数学运算，\n",
+ "而不是简单地依赖预定义的神经网络层。\n",
+ "\n",
+ "到目前为止，\n",
+ "我们网络中的所有操作都对网络的激活值及网络的参数起作用。\n",
+ "然而，有时我们可能希望合并既不是上一层的结果也不是可更新参数的项，\n",
+ "我们称之为*常数参数*（constant parameter）。\n",
+ "例如，我们需要一个计算函数\n",
+ "$f(\\mathbf{x},\\mathbf{w}) = c \\cdot \\mathbf{w}^\\top \\mathbf{x}$的层，\n",
+ "其中$\\mathbf{x}$是输入，\n",
+ "$\\mathbf{w}$是参数，\n",
+ "$c$是某个在优化过程中没有更新的指定常量。\n",
+ "因此我们实现了一个`FixedHiddenMLP`类，如下所示：\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "origin_pos": 30,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "# 在__init__和forward中进行自定义计算\n",
+ "class FixedHiddenMLP(nn.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " # 不计算梯度的随机权重参数。因此其在训练期间保持不变\n",
+ " self.rand_weight = torch.rand((20, 20), requires_grad=False)\n",
+ " self.linear = nn.Linear(20, 20)\n",
+ "\n",
+ " def forward(self, X):\n",
+ " X = self.linear(X)\n",
+ " # 使用创建的常量参数以及relu和mm函数\n",
+ " X = F.relu(torch.mm(X, self.rand_weight) + 1)\n",
+ " # 复用全连接层。这相当于两个全连接层共享参数\n",
+ " X = self.linear(X)\n",
+ " # 控制流\n",
+ " while X.abs().sum() > 1:\n",
+ " X /= 2\n",
+ " return X.sum()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 32
+ },
+ "source": [
+ "在这个`FixedHiddenMLP`模型中，我们实现了一个隐藏层，\n",
+ "其权重（`self.rand_weight`）在实例化时被随机初始化，之后为常量。\n",
+ "这个权重不是一个模型参数，因此它永远不会被反向传播更新。\n",
+ "然后，神经网络将这个固定层的输出通过一个全连接层。\n",
+ "\n",
+ "注意，在返回输出之前，模型做了一些不寻常的事情：\n",
+ "它运行了一个while循环，在$L_1$范数大于$1$的条件下，\n",
+ "将输出向量除以$2$，直到它满足条件为止。\n",
+ "最后，模型返回了`X`中所有项的和。\n",
+ "注意，此操作可能不会常用于在任何实际任务中，\n",
+ "我们只是向你展示如何将任意代码集成到神经网络计算的流程中。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "origin_pos": 34,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor(-0.0123, grad_fn=)"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net = FixedHiddenMLP()\n",
+ "net(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 35
+ },
+ "source": [
+ "我们可以[**混合搭配各种组合块的方法**]。\n",
+ "在下面的例子中，我们以一些想到的方法嵌套块。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "origin_pos": 37,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor(-0.3015, grad_fn=)"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# 定义嵌套的MLP\n",
+ "class NestMLP(nn.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " # 将Linear, Sequnential嵌套入网络\n",
+ " self.net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(),\n",
+ " nn.Linear(64, 32), nn.ReLU())\n",
+ " self.linear = nn.Linear(32, 16)\n",
+ "\n",
+ " def forward(self, X):\n",
+ " return self.linear(self.net(X))\n",
+ "\n",
+ "# Sequnential的输入可以是任何nn.Module的子类\n",
+ "# 将NestMLP, Linear, FixedHiddenMLP嵌套入Sequnential\n",
+ "chimera = nn.Sequential(NestMLP(), nn.Linear(16, 20), FixedHiddenMLP())\n",
+ "chimera(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 39
+ },
+ "source": [
+ "## 效率\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 41,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "你可能会开始担心操作效率的问题。\n",
+ "毕竟，我们在一个高性能的深度学习库中进行了大量的字典查找、\n",
+ "代码执行和许多其他的Python代码。\n",
+ "Python的问题[全局解释器锁](https://wiki.python.org/moin/GlobalInterpreterLock)\n",
+ "是众所周知的。\n",
+ "在深度学习环境中，我们担心速度极快的GPU可能要等到CPU运行Python代码后才能运行另一个作业。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 43
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* 一个块可以由许多层组成；一个块可以由许多块组成。\n",
+ "* 块可以包含代码。\n",
+ "* 块负责大量的内部处理，包括参数初始化和反向传播。\n",
+ "* 层和块的顺序连接由`Sequential`块处理。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 如果将`MySequential`中存储块的方式更改为Python列表，会出现什么样的问题？\n",
+ "1. 实现一个块，它以两个块为参数，例如`net1`和`net2`，并返回前向传播中两个网络的串联输出。这也被称为平行块。\n",
+ "1. 假设你想要连接同一网络的多个实例。实现一个函数，该函数生成同一个块的多个实例，并在此基础上构建更大的网络。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 45,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1827)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python [conda env:pytorch]",
+ "language": "python",
+ "name": "conda-env-pytorch-py"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git "a/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/parameters.ipynb" "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/parameters.ipynb"
new file mode 100644
index 0000000..4e1007b
--- /dev/null
+++ "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/parameters.ipynb"
@@ -0,0 +1,787 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 参数管理\n",
+ "\n",
+ "在选择了架构并设置了超参数后，我们就进入了训练阶段。\n",
+ "此时，我们的目标是找到使损失函数最小化的模型参数值。\n",
+ "经过训练后，我们将需要使用这些参数来做出未来的预测。\n",
+ "此外，有时我们希望提取参数，以便在其他环境中复用它们，\n",
+ "将模型保存下来，以便它可以在其他软件中执行，\n",
+ "或者为了获得科学的理解而进行检查。\n",
+ "\n",
+ "之前的介绍中，我们只依靠深度学习框架来完成训练的工作，\n",
+ "而忽略了操作参数的具体细节。\n",
+ "本节，我们将介绍以下内容：\n",
+ "\n",
+ "* 访问参数，用于调试、诊断和可视化。\n",
+ "* 参数初始化。\n",
+ "* 在不同模型组件间共享参数。\n",
+ "\n",
+ "(**我们首先看一下具有单隐藏层的多层感知机。**)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[0.3913],\n",
+ " [0.3100]], grad_fn=)"
+ ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "\n",
+ "net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1))#使用Sequential定义网络\n",
+ "X = torch.rand(size=(2, 4))#随机生成输出\n",
+ "net(X)#通过网络计算输出"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 4
+ },
+ "source": [
+ "## [**参数访问**]\n",
+ "\n",
+ "我们从已有模型中访问参数。\n",
+ "当通过`Sequential`类定义模型时，\n",
+ "我们可以通过索引来访问模型的任意层。\n",
+ "这就像模型是一个列表一样，每层的参数都在其属性中。\n",
+ "如下所示，我们可以检查第二个全连接层的参数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "OrderedDict([('weight', tensor([[-0.2680, -0.3387, 0.0259, -0.0591, 0.1884, 0.2721, 0.1892, 0.2496]])), ('bias', tensor([0.3248]))])\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(net[2].state_dict())# 取得net第三模块参数"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 8
+ },
+ "source": [
+ "输出的结果告诉我们一些重要的事情：\n",
+ "首先，这个全连接层包含两个参数，分别是该层的权重和偏置。\n",
+ "两者都存储为单精度浮点数（float32）。\n",
+ "注意，参数名称允许唯一标识每个参数，即使在包含数百个层的网络中也是如此。\n",
+ "\n",
+ "### [**目标参数**]\n",
+ "\n",
+ "注意，每个参数都表示为参数类的一个实例。\n",
+ "要对参数执行任何操作，首先我们需要访问底层的数值。\n",
+ "有几种方法可以做到这一点。有些比较简单，而另一些则比较通用。\n",
+ "下面的代码从第二个全连接层（即第三个神经网络层）提取偏置，\n",
+ "提取后返回的是一个参数类实例，并进一步访问该参数的值。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 10,
+ "scrolled": false,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Parameter containing:\n",
+ "tensor([0.3248], requires_grad=True)\n",
+ "tensor([0.3248])\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(type(net[2].bias))# 第三层偏置的数据类型：torch.nn.parameter.Parameter\n",
+ "print(net[2].bias)#一个requires_grad为True的tensor\n",
+ "print(net[2].bias.data)#一个Tensor"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 12,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "参数是复合的对象，包含值、梯度和额外信息。\n",
+ "这就是我们需要显式参数值的原因。\n",
+ "除了值之外，我们还可以访问每个参数的梯度。\n",
+ "在上面这个网络中，由于我们还没有调用反向传播，所以参数的梯度处于初始状态。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "origin_pos": 14,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net[2].weight.grad == None"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 15
+ },
+ "source": [
+ "### [**一次性访问所有参数**]\n",
+ "\n",
+ "当我们需要对所有参数执行操作时，逐个访问它们可能会很麻烦。\n",
+ "当我们处理更复杂的块（例如，嵌套块）时，情况可能会变得特别复杂，\n",
+ "因为我们需要递归整个树来提取每个子块的参数。\n",
+ "下面，我们将通过演示来比较访问第一个全连接层的参数和访问所有层。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "origin_pos": 17,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "('weight', torch.Size([8, 4])) ('bias', torch.Size([8]))\n",
+ "('0.weight', torch.Size([8, 4])) ('0.bias', torch.Size([8])) ('2.weight', torch.Size([1, 8])) ('2.bias', torch.Size([1]))\n"
+ ]
+ }
+ ],
+ "source": [
+ "#*list可以在函数传参时将列表中的元素作为函数的参数传递\n",
+ "#函数named_parameters()可迭代，迭代器返回参数名和参水对象构成的元组\n",
+ "#此处迭代的访问每个参数，得到参数名和参数，组合成列表，再使用*逐一传入print函数输出\n",
+ "print(*[(name, param.shape) for name, param in net[0].named_parameters()])\n",
+ "print(*[(name, param.shape) for name, param in net.named_parameters()])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 19
+ },
+ "source": [
+ "这为我们提供了另一种访问网络参数的方式，如下所示。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "origin_pos": 21,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([0.3248])"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net.state_dict()['2.bias'].data# 指定层数和参数名，在字典中得到相应的值，关键字值为层数.参数名"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 23
+ },
+ "source": [
+ "### [**从嵌套块收集参数**]\n",
+ "\n",
+ "让我们看看，如果我们将多个块相互嵌套，参数命名约定是如何工作的。\n",
+ "我们首先定义一个生成块的函数（可以说是“块工厂”），然后将这些块组合到更大的块中。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "origin_pos": 25,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[-0.2676],\n",
+ " [-0.2679]], grad_fn=)"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "def block1():\n",
+ " return nn.Sequential(nn.Linear(4, 8), nn.ReLU(),\n",
+ " nn.Linear(8, 4), nn.ReLU())\n",
+ "\n",
+ "def block2():\n",
+ " net = nn.Sequential()\n",
+ " for i in range(4):\n",
+ " # 在这里嵌套\n",
+ " net.add_module(f'block {i}', block1())#f''包起来的部分中的{}被视作表达式\n",
+ " return net\n",
+ "\n",
+ "rgnet = nn.Sequential(block2(), nn.Linear(4, 1))#再嵌套一层\n",
+ "rgnet(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 27
+ },
+ "source": [
+ "[**设计了网络后，我们看看它是如何工作的。**]\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "origin_pos": 29,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Sequential(\n",
+ " (0): Sequential(\n",
+ " (block 0): Sequential(\n",
+ " (0): Linear(in_features=4, out_features=8, bias=True)\n",
+ " (1): ReLU()\n",
+ " (2): Linear(in_features=8, out_features=4, bias=True)\n",
+ " (3): ReLU()\n",
+ " )\n",
+ " (block 1): Sequential(\n",
+ " (0): Linear(in_features=4, out_features=8, bias=True)\n",
+ " (1): ReLU()\n",
+ " (2): Linear(in_features=8, out_features=4, bias=True)\n",
+ " (3): ReLU()\n",
+ " )\n",
+ " (block 2): Sequential(\n",
+ " (0): Linear(in_features=4, out_features=8, bias=True)\n",
+ " (1): ReLU()\n",
+ " (2): Linear(in_features=8, out_features=4, bias=True)\n",
+ " (3): ReLU()\n",
+ " )\n",
+ " (block 3): Sequential(\n",
+ " (0): Linear(in_features=4, out_features=8, bias=True)\n",
+ " (1): ReLU()\n",
+ " (2): Linear(in_features=8, out_features=4, bias=True)\n",
+ " (3): ReLU()\n",
+ " )\n",
+ " )\n",
+ " (1): Linear(in_features=4, out_features=1, bias=True)\n",
+ ")\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(rgnet)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 31
+ },
+ "source": [
+ "因为层是分层嵌套的，所以我们也可以像通过嵌套列表索引一样访问它们。\n",
+ "下面，我们访问第一个主要的块中、第二个子块的第一层的偏置项。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "origin_pos": 33,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([-0.3981, -0.4271, -0.3701, 0.1086, -0.1413, 0.3649, 0.0478, -0.4952])"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "rgnet[0][1][0].bias.data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 35
+ },
+ "source": [
+ "## 参数初始化\n",
+ "\n",
+ "知道了如何访问参数后，现在我们看看如何正确地初始化参数。\n",
+ "我们在 :numref:`sec_numerical_stability`中讨论了良好初始化的必要性。\n",
+ "深度学习框架提供默认随机初始化，\n",
+ "也允许我们创建自定义初始化方法，\n",
+ "满足我们通过其他规则实现初始化权重。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 37,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "默认情况下，PyTorch会根据一个范围均匀地初始化权重和偏置矩阵，\n",
+ "这个范围是根据输入和输出维度计算出的。\n",
+ "PyTorch的`nn.init`模块提供了多种预置初始化方法。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 39
+ },
+ "source": [
+ "### [**内置初始化**]\n",
+ "\n",
+ "让我们首先调用内置的初始化器。\n",
+ "下面的代码将所有权重参数初始化为标准差为0.01的高斯随机变量，\n",
+ "且将偏置参数设置为0。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "origin_pos": 41,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(tensor([-0.0053, 0.0067, 0.0123, 0.0004]), tensor(0.))"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "def init_normal(m):#输入为一个层，类型为Module的子类\n",
+ " if type(m) == nn.Linear:#对于所有线性层\n",
+ " nn.init.normal_(m.weight, mean=0, std=0.01)#期望0，方差0.01的正态分布\n",
+ " nn.init.zeros_(m.bias)#全部置零\n",
+ "net.apply(init_normal)#递归的对于每个模块调用函数\n",
+ "net[0].weight.data[0], net[0].bias.data[0]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 43
+ },
+ "source": [
+ "我们还可以将所有参数初始化为给定的常数，比如初始化为1。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "origin_pos": 45,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(tensor([1., 1., 1., 1.]), tensor(0.))"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "def init_constant(m):\n",
+ " if type(m) == nn.Linear:\n",
+ " nn.init.constant_(m.weight, 1)#初始化为常数1\n",
+ " nn.init.zeros_(m.bias)\n",
+ "net.apply(init_constant)\n",
+ "net[0].weight.data[0], net[0].bias.data[0]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 47
+ },
+ "source": [
+ "我们还可以[**对某些块应用不同的初始化方法**]。\n",
+ "例如，下面我们使用Xavier初始化方法初始化第一个神经网络层，\n",
+ "然后将第三个神经网络层初始化为常量值42。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "origin_pos": 49,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensor([-0.4398, -0.2092, 0.2993, 0.2878])\n",
+ "tensor([[42., 42., 42., 42., 42., 42., 42., 42.]])\n"
+ ]
+ }
+ ],
+ "source": [
+ "def xavier(m):\n",
+ " if type(m) == nn.Linear:\n",
+ " nn.init.xavier_uniform_(m.weight)#使用均匀分布进行xavier初始化\n",
+ "def init_42(m):\n",
+ " if type(m) == nn.Linear:\n",
+ " nn.init.constant_(m.weight, 42)#初始化为常数42\n",
+ "\n",
+ "net[0].apply(xavier)\n",
+ "net[2].apply(init_42)\n",
+ "print(net[0].weight.data[0])\n",
+ "print(net[2].weight.data)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 51
+ },
+ "source": [
+ "### [**自定义初始化**]\n",
+ "\n",
+ "有时，深度学习框架没有提供我们需要的初始化方法。\n",
+ "在下面的例子中，我们使用以下的分布为任意权重参数$w$定义初始化方法：\n",
+ "\n",
+ "$$\n",
+ "\\begin{aligned}\n",
+ " w \\sim \\begin{cases}\n",
+ " U(5, 10) & \\text{ 可能性 } \\frac{1}{4} \\\\\n",
+ " 0 & \\text{ 可能性 } \\frac{1}{2} \\\\\n",
+ " U(-10, -5) & \\text{ 可能性 } \\frac{1}{4}\n",
+ " \\end{cases}\n",
+ "\\end{aligned}\n",
+ "$$\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 53,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "同样，我们实现了一个`my_init`函数来应用到`net`。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "origin_pos": 56,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Init weight torch.Size([8, 4])\n",
+ "Init weight torch.Size([1, 8])\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[-0.0000, -8.5068, 0.0000, 6.6374],\n",
+ " [ 0.0000, 5.2751, -7.4942, 5.9510]], grad_fn=)"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "def my_init(m):\n",
+ " if type(m) == nn.Linear:\n",
+ " #取所有参数的(名字，参数)元组构成列表的第一项，将元组元素分别作为参数传入print函数\n",
+ " print(\"Init\", *[(name, param.shape) for name, param in m.named_parameters()][0])\n",
+ " nn.init.uniform_(m.weight, -10, 10)# 权重在-10，10上服从均匀分布\n",
+ " m.weight.data *= m.weight.data.abs()>=5 # 保留权重绝对值大于5的值\n",
+ "\n",
+ "net.apply(my_init)\n",
+ "net[0].weight[:2]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 58
+ },
+ "source": [
+ "注意，我们始终可以直接设置参数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {
+ "origin_pos": 60,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([42.0000, -7.5068, 1.0000, 7.6374])"
+ ]
+ },
+ "execution_count": 14,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net[0].weight.data[:] += 1\n",
+ "net[0].weight.data[0, 0] = 42\n",
+ "net[0].weight.data[0]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 63
+ },
+ "source": [
+ "## [**参数绑定**]\n",
+ "\n",
+ "有时我们希望在多个层间共享参数：\n",
+ "我们可以定义一个稠密层，然后使用它的参数来设置另一个层的参数。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "origin_pos": 65,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensor([True, True, True, True, True, True, True, True])\n",
+ "tensor([True, True, True, True, True, True, True, True])\n"
+ ]
+ }
+ ],
+ "source": [
+ "# 我们需要给共享层一个名称，以便可以引用它的参数\n",
+ "shared = nn.Linear(8, 8)\n",
+ "net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(),\n",
+ " shared, nn.ReLU(),\n",
+ " shared, nn.ReLU(),\n",
+ " nn.Linear(8, 1))\n",
+ "net(X)\n",
+ "# 检查参数是否相同\n",
+ "print(net[2].weight.data[0] == net[4].weight.data[0])\n",
+ "net[2].weight.data[0, 0] = 100#修改其中一个的一个位置，如果二者实际上是同一对象，则修改一个另一个也会改变\n",
+ "# 确保它们实际上是同一个对象，而不只是有相同的值\n",
+ "print(net[2].weight.data[0] == net[4].weight.data[0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 68,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "这个例子表明第三个和第五个神经网络层的参数是绑定的。\n",
+ "它们不仅值相等，而且由相同的张量表示。\n",
+ "因此，如果我们改变其中一个参数，另一个参数也会改变。\n",
+ "你可能会思考：当参数绑定时，梯度会发生什么情况？\n",
+ "答案是由于模型参数包含梯度，因此在反向传播期间第二个隐藏层\n",
+ "（即第三个神经网络层）和第三个隐藏层（即第五个神经网络层）的梯度会加在一起。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 69
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* 我们有几种方法可以访问、初始化和绑定模型参数。\n",
+ "* 我们可以使用自定义初始化方法。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 使用 :numref:`sec_model_construction` 中定义的`FancyMLP`模型，访问各个层的参数。\n",
+ "1. 查看初始化模块文档以了解不同的初始化方法。\n",
+ "1. 构建包含共享参数层的多层感知机并对其进行训练。在训练过程中，观察模型各层的参数和梯度。\n",
+ "1. 为什么共享参数是个好主意？\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 71,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1829)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git "a/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/read-write.ipynb" "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/read-write.ipynb"
new file mode 100644
index 0000000..8e5c639
--- /dev/null
+++ "b/code/16-Pytorch\347\245\236\347\273\217\347\275\221\347\273\234\345\237\272\347\241\200/read-write.ipynb"
@@ -0,0 +1,356 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 读写文件\n",
+ "\n",
+ "到目前为止，我们讨论了如何处理数据，\n",
+ "以及如何构建、训练和测试深度学习模型。\n",
+ "然而，有时我们希望保存训练的模型，\n",
+ "以备将来在各种环境中使用（比如在部署中进行预测）。\n",
+ "此外，当运行一个耗时较长的训练过程时，\n",
+ "最佳的做法是定期保存中间结果，\n",
+ "以确保在服务器电源被不小心断掉时，我们不会损失几天的计算结果。\n",
+ "因此，现在是时候学习如何加载和存储权重向量和整个模型了。\n",
+ "\n",
+ "## (**加载和保存张量**)\n",
+ "\n",
+ "对于单个张量，我们可以直接调用`load`和`save`函数分别读写它们。\n",
+ "这两个函数都要求我们提供一个名称，`save`要求将要保存的变量作为输入。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from torch.nn import functional as F\n",
+ "\n",
+ "x = torch.arange(4)\n",
+ "torch.save(x, 'x-file')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 4
+ },
+ "source": [
+ "我们现在可以将存储在文件中的数据读回内存。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([0, 1, 2, 3])"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "x2 = torch.load('x-file')\n",
+ "x2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 8
+ },
+ "source": [
+ "我们可以[**存储一个张量列表，然后把它们读回内存。**]\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 10,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(tensor([0, 1, 2, 3]), tensor([0., 0., 0., 0.]))"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "y = torch.zeros(4)\n",
+ "torch.save([x, y],'x-files')\n",
+ "x2, y2 = torch.load('x-files')\n",
+ "(x2, y2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 12
+ },
+ "source": [
+ "我们甚至可以(**写入或读取从字符串映射到张量的字典**)。\n",
+ "当我们要读取或写入模型中的所有权重时，这很方便。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "origin_pos": 14,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'x': tensor([0, 1, 2, 3]), 'y': tensor([0., 0., 0., 0.])}"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "mydict = {'x': x, 'y': y}\n",
+ "torch.save(mydict, 'mydict')\n",
+ "mydict2 = torch.load('mydict')\n",
+ "mydict2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 16
+ },
+ "source": [
+ "## [**加载和保存模型参数**]\n",
+ "\n",
+ "保存单个权重向量（或其他张量）确实有用，\n",
+ "但是如果我们想保存整个模型，并在以后加载它们，\n",
+ "单独保存每个向量则会变得很麻烦。\n",
+ "毕竟，我们可能有数百个参数散布在各处。\n",
+ "因此，深度学习框架提供了内置函数来保存和加载整个网络。\n",
+ "需要注意的一个重要细节是，这将保存模型的参数而不是保存整个模型。\n",
+ "例如，如果我们有一个3层多层感知机，我们需要单独指定架构。\n",
+ "因为模型本身可以包含任意代码，所以模型本身难以序列化。\n",
+ "因此，为了恢复模型，我们需要用代码生成架构，\n",
+ "然后从磁盘加载参数。\n",
+ "让我们从熟悉的多层感知机开始尝试一下。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "origin_pos": 18,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "#定义多层感知机\n",
+ "class MLP(nn.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.hidden = nn.Linear(20, 256)\n",
+ " self.output = nn.Linear(256, 10)\n",
+ "\n",
+ " def forward(self, x):\n",
+ " return self.output(F.relu(self.hidden(x)))\n",
+ "\n",
+ "net = MLP()#实例化\n",
+ "X = torch.randn(size=(2, 20))#随机输入\n",
+ "Y = net(X)#得到对应输出"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 20
+ },
+ "source": [
+ "接下来，我们[**将模型的参数存储在一个叫做“mlp.params”的文件中。**]\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "origin_pos": 22,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "torch.save(net.state_dict(), 'mlp.params')#将所有参数写入字典并保存为文件"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 24
+ },
+ "source": [
+ "为了恢复模型，我们[**实例化了原始多层感知机模型的一个备份。**]\n",
+ "这里我们不需要随机初始化模型参数，而是(**直接读取文件中存储的参数。**)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "origin_pos": 26,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MLP(\n",
+ " (hidden): Linear(in_features=20, out_features=256, bias=True)\n",
+ " (output): Linear(in_features=256, out_features=10, bias=True)\n",
+ ")"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "clone = MLP()#另一个多层感知机，参数并未学习\n",
+ "clone.load_state_dict(torch.load('mlp.params'))#读取文件并写入模型（覆盖初始化的参数）\n",
+ "clone.eval()#设置为评估模式，保持参数不变"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 28
+ },
+ "source": [
+ "由于两个实例具有相同的模型参数，在输入相同的`X`时，\n",
+ "两个实例的计算结果应该相同。\n",
+ "让我们来验证一下。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "origin_pos": 30,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[True, True, True, True, True, True, True, True, True, True],\n",
+ " [True, True, True, True, True, True, True, True, True, True]])"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Y_clone = clone(X)\n",
+ "Y_clone == Y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 32
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* `save`和`load`函数可用于张量对象的文件读写。\n",
+ "* 我们可以通过参数字典保存和加载网络的全部参数。\n",
+ "* 保存架构必须在代码中完成，而不是在参数中完成。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 即使不需要将经过训练的模型部署到不同的设备上，存储模型参数还有什么实际的好处？\n",
+ "1. 假设我们只想复用网络的一部分，以将其合并到不同的网络架构中。比如说，如果你想在一个新的网络中使用之前网络的前两层，你该怎么做？\n",
+ "1. 如何同时保存网络架构和参数？你会对架构加上什么限制？\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 34,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1839)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git "a/code/17-\344\275\277\347\224\250\345\222\214\350\264\255\344\271\260GPU.ipynb" "b/code/17-\344\275\277\347\224\250\345\222\214\350\264\255\344\271\260GPU.ipynb"
new file mode 100644
index 0000000..244bd65
--- /dev/null
+++ "b/code/17-\344\275\277\347\224\250\345\222\214\350\264\255\344\271\260GPU.ipynb"
@@ -0,0 +1,221 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "79fae070",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "527e2f88",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "94442ac9",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "\n",
+ "\"\"\"指定cpu, gpu设备\"\"\"\n",
+ "torch.device('cpu'), torch.device('cuda'), torch.device('cuda:1')\n",
+ "# cpu, gpu0, gpu1\n",
+ "\n",
+ "\"\"\"查询可用gpu数量\"\"\"\n",
+ "torch.cuda.device_count()\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "85a5d238",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(device(type='cpu'), device(type='cpu'), [device(type='cpu')])"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "\"\"\"定义了两个方便的函数， 这两个函数允许我们在不存在所需所有GPU的情况下运行代码\"\"\"\n",
+ "\n",
+ "\n",
+ "\"\"\"如果存在，则返回gpu(i)，否则返回cpu()\"\"\"\n",
+ "def try_gpu(i=0):\n",
+ "# 不输入参数则默认i = 0 \n",
+ " if torch.cuda.device_count() >= i + 1:\n",
+ " return torch.device(f'cuda:{i}')\n",
+ " # 如果当前可用gpu的总数大于等于i+1，则返回第i个gpu（从0计数）\t\t\n",
+ " return torch.device('cpu')\n",
+ "\t# 否则证明当前没有更多可用gpu，则返回cpu\n",
+ "\n",
+ "\n",
+ "\"\"\"返回所有可用的GPU，如果没有GPU，则返回[cpu(),]\"\"\"\n",
+ "def try_all_gpus():\n",
+ " devices = [torch.device(f'cuda:{i}')\n",
+ " for i in range(torch.cuda.device_count())]\n",
+ " # 所有可用gpu设备序号组成的列表devices\n",
+ " \n",
+ " return devices if devices else [torch.device('cpu')]\n",
+ "\t# 如果列表devices不为空则证明此时有可用的gpu，则返回可用gpu序号列表；否则证明没有可用gpu，则返回cpu\n",
+ "\n",
+ "try_gpu(), try_gpu(10), try_all_gpus()\n",
+ "# 测试函数功能\n",
+ "# try_gpu():检测是否有第i=0号gpu\n",
+ "# try_gpu(10):检测是否有第i=10号gpu\n",
+ "# try_all_gpus():返回所有可用gpu序号列表，如果没有gpu则返回cpu\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "295a6578",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "device(type='cpu')"
+ ]
+ },
+ "execution_count": 19,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "\"\"\"我们可以查询张量所在的设备。 默认情况下，张量是在CPU上创建的。\"\"\"\n",
+ "x = torch.tensor([1, 2, 3])\n",
+ "x.device"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "255462a4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\"\"\"我们在第一个gpu上创建张量变量X\"\"\" \n",
+ "X = torch.ones(2, 3, device=try_gpu())\n",
+ "\n",
+ "\"\"\"假设你至少有两个GPU，下面的代码将在第二个GPU上创建一个随机张量\"\"\"\n",
+ "Y = torch.rand(2, 3, device=try_gpu(1))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8f72119d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\"\"\"将gpu(0)中的X复制到gpu(1)中的Z\"\"\"\n",
+ "Z = X.cuda(1)\n",
+ "print(X)\n",
+ "print(Z)\n",
+ "\n",
+ "\n",
+ "\"\"\"现在数据在同一个GPU上（Z和Y都在），我们可以将它们相加。\"\"\"\n",
+ "Y + Z\n",
+ "\n",
+ "\n",
+ "\"\"\"如果变量Z已经存在于第i个GPU上，再调用Z.cuda(i)只会返回Z并不会复制并分配新内存\"\"\"\n",
+ "Z.cuda(1) is Z\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "e9861ae0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\"\"\"\n",
+ "类似地，神经网络模型可以指定设备。 下面的代码将模型参数放在GPU上。\n",
+ "\"\"\"\n",
+ "\n",
+ "net = nn.Sequential(nn.Linear(3, 1))\n",
+ "net = net.to(device=try_gpu())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "id": "0b17de5d",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "device(type='cpu')"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net(X)\n",
+ "\n",
+ "net[0].weight.data.device\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "26d4de02",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.7"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git "a/code/20-\345\241\253\345\205\205\345\222\214\346\255\245\345\271\205.ipynb" "b/code/20-\345\241\253\345\205\205\345\222\214\346\255\245\345\271\205.ipynb"
new file mode 100644
index 0000000..330c905
--- /dev/null
+++ "b/code/20-\345\241\253\345\205\205\345\222\214\346\255\245\345\271\205.ipynb"
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "### 填充和步幅"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "-"
+ }
+ },
+ "source": [
+ "导入包，定义comp_conv2d函数(进行卷积操作, 输出后两维，便于观察高宽的维度变化)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "\n",
+ "def comp_conv2d(conv2d, X):\n",
+ " X = X.reshape((1, 1) + X.shape) #X的维度之前加入批量大小数(batch_size)和输入通道数(channel_in)\n",
+ " Y = conv2d(X) \n",
+ " return Y.reshape(Y.shape[2:]) #去掉前面的两维后(原来四维) 进行输出\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "### padding\n",
+ "在所有侧边填充1个像素(padding=1(即(1,1)))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "torch.Size([8, 8])"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "conv2d = nn.Conv2d(1, 1, kernel_size=3, padding=1) #输入输出通道数为1, 卷积核大小3x3, 填充为1(上下左右各填充一行)\n",
+ "X = torch.rand(size=(8, 8)) \n",
+ "comp_conv2d(conv2d, X).shape"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "填充不同的高度和宽度(padding=(2,1))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "torch.Size([8, 8])"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "conv2d = nn.Conv2d(1, 1, kernel_size=(5, 3), padding=(2, 1))\n",
+ "comp_conv2d(conv2d, X).shape"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "### stride\n",
+ "将高度和宽度的步幅设置为2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "torch.Size([4, 4])"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "conv2d = nn.Conv2d(1, 1, kernel_size=3, padding=1, stride=2)\n",
+ "comp_conv2d(conv2d, X).shape"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "一个稍微复杂的例子"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "torch.Size([2, 2])"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "conv2d = nn.Conv2d(1, 1, kernel_size=(3, 5), padding=(0, 1), stride=(3, 4))\n",
+ "comp_conv2d(conv2d, X).shape"
+ ]
+ }
+ ],
+ "metadata": {
+ "celltoolbar": "幻灯片",
+ "kernelspec": {
+ "display_name": "Python [conda env:root] *",
+ "language": "python",
+ "name": "conda-root-py"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git "a/code/21-\345\244\232\350\276\223\345\205\245\350\276\223\345\207\272\351\200\232\351\201\223.ipynb" "b/code/21-\345\244\232\350\276\223\345\205\245\350\276\223\345\207\272\351\200\232\351\201\223.ipynb"
new file mode 100644
index 0000000..c7d4d8e
--- /dev/null
+++ "b/code/21-\345\244\232\350\276\223\345\205\245\350\276\223\345\207\272\351\200\232\351\201\223.ipynb"
@@ -0,0 +1,389 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 多输入多输出通道\n",
+ ":label:`sec_channels`\n",
+ "\n",
+ "虽然我们在 :numref:`subsec_why-conv-channels`中描述了构成每个图像的多个通道和多层卷积层。例如彩色图像具有标准的RGB通道来指示红、绿和蓝。\n",
+ "但是到目前为止，我们仅展示了单个输入和单个输出通道的简化例子。\n",
+ "这使得我们可以将输入、卷积核和输出看作二维张量。\n",
+ "\n",
+ "当我们添加通道时，我们的输入和隐藏的表示都变成了三维张量。例如，每个RGB输入图像具有$3\\times h\\times w$的形状。我们将这个大小为$3$的轴称为*通道*（channel）维度。在本节中，我们将更深入地研究具有多输入和多输出通道的卷积核。\n",
+ "\n",
+ "## 多输入通道\n",
+ "\n",
+ "当输入包含多个通道时，需要构造一个与输入数据具有相同输入通道数的卷积核，以便与输入数据进行互相关运算。假设输入的通道数为$c_i$，那么卷积核的输入通道数也需要为$c_i$。如果卷积核的窗口形状是$k_h\\times k_w$，那么当$c_i=1$时，我们可以把卷积核看作形状为$k_h\\times k_w$的二维张量。\n",
+ "\n",
+ "然而，当$c_i>1$时，我们卷积核的每个输入通道将包含形状为$k_h\\times k_w$的张量。将这些张量$c_i$连结在一起可以得到形状为$c_i\\times k_h\\times k_w$的卷积核。由于输入和卷积核都有$c_i$个通道，我们可以对每个通道输入的二维张量和卷积核的二维张量进行互相关运算，再对通道求和（将$c_i$的结果相加）得到二维张量。这是多通道输入和多输入通道卷积核之间进行二维互相关运算的结果。\n",
+ "\n",
+ "在 :numref:`fig_conv_multi_in`中，我们演示了一个具有两个输入通道的二维互相关运算的示例。阴影部分是第一个输出元素以及用于计算这个输出的输入和核张量元素：$(1\\times1+2\\times2+4\\times3+5\\times4)+(0\\times0+1\\times1+3\\times2+4\\times3)=56$。\n",
+ "\n",
+ "\n",
+ ":label:`fig_conv_multi_in`\n",
+ "\n",
+ "为了加深理解，我们(**实现一下多输入通道互相关运算**)。\n",
+ "简而言之，我们所做的就是对每个通道执行互相关操作，然后将结果相加。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from d2l import torch as d2l"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {
+ "origin_pos": 3,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "def corr2d_multi_in(X, K):\n",
+ " # 先遍历“X”和“K”的第0个维度（通道维度），再把它们加在一起\n",
+ " return sum(d2l.corr2d(x, k) for x, k in zip(X, K))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 5
+ },
+ "source": [
+ "我们可以构造与 :numref:`fig_conv_multi_in`中的值相对应的输入张量`X`和核张量`K`，以(**验证互相关运算的输出**)。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[ 56., 72.],\n",
+ " [104., 120.]])"
+ ]
+ },
+ "execution_count": 34,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X = torch.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],\n",
+ " [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])\n",
+ "K = torch.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])\n",
+ "\n",
+ "corr2d_multi_in(X, K)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 7
+ },
+ "source": [
+ "## 多输出通道\n",
+ "\n",
+ "到目前为止，不论有多少输入通道，我们还只有一个输出通道。然而，正如我们在 :numref:`subsec_why-conv-channels`中所讨论的，每一层有多个输出通道是至关重要的。在最流行的神经网络架构中，随着神经网络层数的加深，我们常会增加输出通道的维数，通过减少空间分辨率以获得更大的通道深度。直观地说，我们可以将每个通道看作是对不同特征的响应。而现实可能更为复杂一些，因为每个通道不是独立学习的，而是为了共同使用而优化的。因此，多输出通道并不仅是学习多个单通道的检测器。\n",
+ "\n",
+ "用$c_i$和$c_o$分别表示输入和输出通道的数目，并让$k_h$和$k_w$为卷积核的高度和宽度。为了获得多个通道的输出，我们可以为每个输出通道创建一个形状为$c_i\\times k_h\\times k_w$的卷积核张量，这样卷积核的形状是$c_o\\times c_i\\times k_h\\times k_w$。在互相关运算中，每个输出通道先获取所有输入通道，再以对应该输出通道的卷积核计算出结果。\n",
+ "\n",
+ "如下所示，我们实现一个[**计算多个通道的输出的互相关函数**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {
+ "origin_pos": 8,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "def corr2d_multi_in_out(X, K):\n",
+ " # 迭代“K”的第0个维度，每次都对输入“X”执行互相关运算。\n",
+ " # 最后将所有结果都叠加在一起\n",
+ " return torch.stack([corr2d_multi_in(X, k) for k in K], 0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 9
+ },
+ "source": [
+ "通过将核张量`K`与`K+1`（`K`中每个元素加$1$）和`K+2`连接起来，构造了一个具有$3$个输出通道的卷积核。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {
+ "origin_pos": 10,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "torch.Size([3, 3, 3, 2, 2, 2])"
+ ]
+ },
+ "execution_count": 40,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "K = torch.stack((K, K + 1, K + 2), 0) #构造一个多输出通道的卷积核\n",
+ "K.shape"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 11
+ },
+ "source": [
+ "下面，我们对输入张量`X`与卷积核张量`K`执行互相关运算。现在的输出包含$3$个通道，第一个通道的结果与先前输入张量`X`和多输入单输出通道的结果一致。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "metadata": {
+ "origin_pos": 12,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "ename": "ValueError",
+ "evalue": "too many values to unpack (expected 2)",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/var/folders/m2/mcjqb__94rgf7qkm22vhg9xh0000gn/T/ipykernel_17262/2068829854.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcorr2d_multi_in_out\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/var/folders/m2/mcjqb__94rgf7qkm22vhg9xh0000gn/T/ipykernel_17262/439246277.py\u001b[0m in \u001b[0;36mcorr2d_multi_in_out\u001b[0;34m(X, K)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# 迭代“K”的第0个维度，每次都对输入“X”执行互相关运算。\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;31m# 最后将所有结果都叠加在一起\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcorr2d_multi_in\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/var/folders/m2/mcjqb__94rgf7qkm22vhg9xh0000gn/T/ipykernel_17262/439246277.py\u001b[0m in \u001b[0;36m\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# 迭代“K”的第0个维度，每次都对输入“X”执行互相关运算。\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;31m# 最后将所有结果都叠加在一起\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcorr2d_multi_in\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/var/folders/m2/mcjqb__94rgf7qkm22vhg9xh0000gn/T/ipykernel_17262/1525835610.py\u001b[0m in \u001b[0;36mcorr2d_multi_in\u001b[0;34m(X, K)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcorr2d_multi_in\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# 先遍历“X”和“K”的第0个维度（通道维度），再把它们加在一起\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0md2l\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorr2d\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/var/folders/m2/mcjqb__94rgf7qkm22vhg9xh0000gn/T/ipykernel_17262/1525835610.py\u001b[0m in \u001b[0;36m\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcorr2d_multi_in\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# 先遍历“X”和“K”的第0个维度（通道维度），再把它们加在一起\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0md2l\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorr2d\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/d2l/torch.py\u001b[0m in \u001b[0;36mcorr2d\u001b[0;34m(X, K)\u001b[0m\n\u001b[1;32m 462\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 463\u001b[0m Defined in :numref:`sec_conv_layer`\"\"\"\n\u001b[0;32m--> 464\u001b[0;31m \u001b[0mh\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 465\u001b[0m \u001b[0mY\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0md2l\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mh\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mw\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 466\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mY\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mValueError\u001b[0m: too many values to unpack (expected 2)"
+ ]
+ }
+ ],
+ "source": [
+ "corr2d_multi_in_out(X, K)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 13
+ },
+ "source": [
+ "## $1\\times 1$ 卷积层\n",
+ "\n",
+ "[~~1x1卷积~~]\n",
+ "\n",
+ "$1 \\times 1$卷积，即$k_h = k_w = 1$，看起来似乎没有多大意义。\n",
+ "毕竟，卷积的本质是有效提取相邻像素间的相关特征，而$1 \\times 1$卷积显然没有此作用。\n",
+ "尽管如此，$1 \\times 1$仍然十分流行，时常包含在复杂深层网络的设计中。下面，让我们详细地解读一下它的实际作用。\n",
+ "\n",
+ "因为使用了最小窗口，$1\\times 1$卷积失去了卷积层的特有能力——在高度和宽度维度上，识别相邻元素间相互作用的能力。\n",
+ "其实$1\\times 1$卷积的唯一计算发生在通道上。\n",
+ "\n",
+ " :numref:`fig_conv_1x1`展示了使用$1\\times 1$卷积核与$3$个输入通道和$2$个输出通道的互相关计算。\n",
+ "这里输入和输出具有相同的高度和宽度，输出中的每个元素都是从输入图像中同一位置的元素的线性组合。\n",
+ "我们可以将$1\\times 1$卷积层看作是在每个像素位置应用的全连接层，以$c_i$个输入值转换为$c_o$个输出值。\n",
+ "因为这仍然是一个卷积层，所以跨像素的权重是一致的。\n",
+ "同时，$1\\times 1$卷积层需要的权重维度为$c_o\\times c_i$，再额外加上一个偏差。\n",
+ "\n",
+ "\n",
+ ":label:`fig_conv_1x1`\n",
+ "\n",
+ "下面，我们使用全连接层实现$1 \\times 1$卷积。\n",
+ "请注意，我们需要对输入和输出的数据形状进行微调。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 42,
+ "metadata": {
+ "origin_pos": 14,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "def corr2d_multi_in_out_1x1(X, K):\n",
+ " c_i, h, w = X.shape\n",
+ " c_o = K.shape[0]\n",
+ " X = X.reshape((c_i, h * w)) # 拉成一个向量\n",
+ " K = K.reshape((c_o, c_i)) # 后两个维度都是1\n",
+ " # 全连接层中的矩阵乘法\n",
+ " Y = torch.matmul(K, X)\n",
+ " return Y.reshape((c_o, h, w)) # 重新转换成矩阵"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 15
+ },
+ "source": [
+ "当执行$1\\times 1$卷积运算时，上述函数相当于先前实现的互相关函数`corr2d_multi_in_out`。让我们用一些样本数据来验证这一点。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 43,
+ "metadata": {
+ "origin_pos": 16,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "X = torch.normal(0, 1, (3, 3, 3))\n",
+ "K = torch.normal(0, 1, (2, 3, 1, 1))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "metadata": {
+ "origin_pos": 18,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(tensor([[[-0.0482, 2.0465, 0.5210],\n",
+ " [ 0.6970, -0.3922, 1.6800],\n",
+ " [-1.2534, -0.9197, -0.9118]],\n",
+ " \n",
+ " [[ 0.1979, -1.3069, 1.0266],\n",
+ " [ 0.5115, -1.7857, 1.3425],\n",
+ " [-0.9712, 0.5453, -0.3722]]]),\n",
+ " tensor([[[-0.0482, 2.0465, 0.5210],\n",
+ " [ 0.6970, -0.3922, 1.6800],\n",
+ " [-1.2534, -0.9197, -0.9118]],\n",
+ " \n",
+ " [[ 0.1979, -1.3069, 1.0266],\n",
+ " [ 0.5115, -1.7857, 1.3425],\n",
+ " [-0.9712, 0.5453, -0.3722]]]))"
+ ]
+ },
+ "execution_count": 45,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Y1 = corr2d_multi_in_out_1x1(X, K)\n",
+ "Y2 = corr2d_multi_in_out(X, K)\n",
+ "assert float(torch.abs(Y1 - Y2).sum()) < 1e-6 # 比较实用卷积方法和全连接层进行卷积的方法结果的差别\n",
+ "Y1,Y2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 19
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* 多输入多输出通道可以用来扩展卷积层的模型。\n",
+ "* 当以每像素为基础应用时，$1\\times 1$卷积层相当于全连接层。\n",
+ "* $1\\times 1$卷积层通常用于调整网络层的通道数量和控制模型复杂性。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 假设我们有两个卷积核，大小分别为$k_1$和$k_2$（中间没有非线性激活函数）。\n",
+ " 1. 证明运算可以用单次卷积来表示。\n",
+ " 1. 这个等效的单卷积的维数是多少呢？\n",
+ " 1. 反之亦然吗？\n",
+ "1. 假设输入为$c_i\\times h\\times w$，卷积核大小为$c_o\\times c_i\\times k_h\\times k_w$，填充为$(p_h, p_w)$，步幅为$(s_h, s_w)$。\n",
+ " 1. 前向传播的计算成本（乘法和加法）是多少？\n",
+ " 1. 内存占用是多少？\n",
+ " 1. 反向传播的内存占用是多少？\n",
+ " 1. 反向传播的计算成本是多少？\n",
+ "1. 如果我们将输入通道$c_i$和输出通道$c_o$的数量加倍，计算数量会增加多少？如果我们把填充数量翻一番会怎么样？\n",
+ "1. 如果卷积核的高度和宽度是$k_h=k_w=1$，前向传播的计算复杂度是多少？\n",
+ "1. 本节最后一个示例中的变量`Y1`和`Y2`是否完全相同？为什么？\n",
+ "1. 当卷积窗口不是$1\\times 1$时，如何使用矩阵乘法实现卷积？\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 21,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1854)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git "a/code/22-\346\261\240\345\214\226\345\261\202.ipynb" "b/code/22-\346\261\240\345\214\226\345\261\202.ipynb"
new file mode 100644
index 0000000..7470289
--- /dev/null
+++ "b/code/22-\346\261\240\345\214\226\345\261\202.ipynb"
@@ -0,0 +1,460 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 池化层\n",
+ "\n",
+ "本节将介绍*池化*（pooling）层，它具有目的：类似于数据增强，降低卷积层对位置的敏感性；一定程度减少计算。\n",
+ "\n",
+ "## 最大池化层和平均池化层\n",
+ "\n",
+ "与卷积层类似，池化层运算符由一个固定形状的窗口组成，该窗口根据其步幅大小在输入的所有区域上滑动，为固定形状窗口遍历的每个位置计算一个输出。\n",
+ "然而，不同于卷积层中的输入与卷积核之间的互相关计算，**池化层不包含参数**。\n",
+ "相反，池运算符是确定性的，我们通常计算池化窗口中所有元素的最大值或平均值。这些操作分别称为*最大池化层*（maximum pooling）和*平均池化层*（average pooling）。\n",
+ "\n",
+ "在这两种情况下，与互相关运算符一样，池化窗口从输入张量的左上角开始，从左往右、从上往下的在输入张量内滑动。在池化窗口到达的每个位置，它计算该窗口中输入子张量的最大值或平均值。计算最大值或平均值是取决于使用了最大池化层还是平均池化层。\n",
+ "\n",
+ "\n",
+ "上图中的输出张量的高度为$2$，宽度为$2$。这四个元素为每个池化窗口中的最大值：\n",
+ "\n",
+ "$$\n",
+ "\\max(0, 1, 3, 4)=4,\\\\\n",
+ "\\max(1, 2, 4, 5)=5,\\\\\n",
+ "\\max(3, 4, 6, 7)=7,\\\\\n",
+ "\\max(4, 5, 7, 8)=8.\\\\\n",
+ "$$\n",
+ "\n",
+ "池化窗口形状为$p \\times q$的池化层称为$p \\times q$池化层，池化操作称为$p \\times q$池化。\n",
+ "\n",
+ "回到本节开头提到的对象边缘检测示例，现在我们将使用卷积层的输出作为$2\\times 2$最大池化的输入。\n",
+ "设置卷积层输入为`X`，池化层输出为`Y`。\n",
+ "无论`X[i, j]`和`X[i, j + 1]`的值是否不同，或`X[i, j + 1]`和`X[i, j + 2]`的值是否不同，池化层始终输出`Y[i, j] = 1`。\n",
+ "也就是说，使用$2\\times 2$最大池化层，即使在高度或宽度上移动一个元素，卷积层仍然可以识别到模式。\n",
+ "\n",
+ "在下面的代码中的`pool2d`函数，我们(**实现池化层的前向传播**)。然而，这里我们没有卷积核，输出为输入中每个区域的最大值或平均值。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from d2l import torch as d2l"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 3,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "def pool2d(X, pool_size, mode='max'):\n",
+ " p_h, p_w = pool_size\n",
+ " Y = torch.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w + 1))\n",
+ " for i in range(Y.shape[0]):\n",
+ " for j in range(Y.shape[1]): # 枚举输出的每个位置，[i,j]对应输入的位置[i至i+p_h,j至j+p_w]\n",
+ " if mode == 'max': # 最大池化\n",
+ " Y[i, j] = X[i: i + p_h, j: j + p_w].max() # max函数返回最大值\n",
+ " elif mode == 'avg': # 平均池化\n",
+ " Y[i, j] = X[i: i + p_h, j: j + p_w].mean() # mean函数返回平均值\n",
+ " return Y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 5
+ },
+ "source": [
+ "我们可以构建上图中的输入张量`X`，[**验证二维最大池化层的输出**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[4., 5.],\n",
+ " [7., 8.]])"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])\n",
+ "pool2d(X, (2, 2))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 7
+ },
+ "source": [
+ "此外，我们还可以(**验证平均池化层**)。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "origin_pos": 8,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[2., 3.],\n",
+ " [5., 6.]])"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "pool2d(X, (2, 2), 'avg')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 9
+ },
+ "source": [
+ "## 填充和步幅\n",
+ "\n",
+ "与卷积层一样，池化层也可以改变输出形状。和以前一样，我们可以通过填充和步幅以获得所需的输出形状。\n",
+ "下面，我们用深度学习框架中内置的二维最大池化层，来演示池化层中填充和步幅的使用。\n",
+ "我们首先构造了一个输入张量`X`，它有四个维度，其中样本数和通道数都是1。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "origin_pos": 11,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[[[ 0., 1., 2., 3.],\n",
+ " [ 4., 5., 6., 7.],\n",
+ " [ 8., 9., 10., 11.],\n",
+ " [12., 13., 14., 15.]]]])"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X = torch.arange(16, dtype=torch.float32).reshape(\n",
+ " (1, 1, 4, 4)) # 维度[batch_size，通道数，H，W]\n",
+ "X"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 13
+ },
+ "source": [
+ "默认情况下，(**深度学习框架中的步幅与池化窗口的大小相同**)。\n",
+ "因此，如果我们使用形状为`(3, 3)`的池化窗口，那么默认情况下，我们得到的步幅形状为`(3, 3)`。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "origin_pos": 15,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[[[10.]]]])"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "pool2d = nn.MaxPool2d(3)\n",
+ "pool2d(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 17
+ },
+ "source": [
+ "[**填充和步幅可以手动设定**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "origin_pos": 19,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[[[ 5., 7.],\n",
+ " [13., 15.]]]])"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "pool2d = nn.MaxPool2d(3, padding=1, stride=2)\n",
+ "pool2d(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 22,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "当然，我们可以(**设定一个任意大小的矩形池化窗口，并分别设定填充和步幅的高度和宽度**)。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "origin_pos": 25,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[[[ 5., 7.],\n",
+ " [13., 15.]]]])"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "pool2d = nn.MaxPool2d((2, 3), stride=(2, 3), padding=(0, 1))\n",
+ "pool2d(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 27
+ },
+ "source": [
+ "## 多个通道\n",
+ "\n",
+ "在处理多通道输入数据时，[**池化层在每个输入通道上单独运算**]，而不是像卷积层一样在通道上对输入进行汇总。\n",
+ "这意味着池化层的输出通道数与输入通道数相同。\n",
+ "下面，我们将在通道维度上连结张量`X`和`X + 1`，以构建具有2个通道的输入。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "origin_pos": 29,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[[[ 0., 1., 2., 3.],\n",
+ " [ 4., 5., 6., 7.],\n",
+ " [ 8., 9., 10., 11.],\n",
+ " [12., 13., 14., 15.]],\n",
+ "\n",
+ " [[ 1., 2., 3., 4.],\n",
+ " [ 5., 6., 7., 8.],\n",
+ " [ 9., 10., 11., 12.],\n",
+ " [13., 14., 15., 16.]]]])"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X = torch.cat((X, X + 1), 1) # 在第一个维度也就是通道维度拼接\n",
+ "X"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 31
+ },
+ "source": [
+ "如下所示，池化后输出通道的数量仍然是2。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "origin_pos": 33,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([[[[ 5., 7.],\n",
+ " [13., 15.]],\n",
+ "\n",
+ " [[ 6., 8.],\n",
+ " [14., 16.]]]])"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "pool2d = nn.MaxPool2d(3, padding=1, stride=2)\n",
+ "pool2d(X)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 36
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* 对于给定输入元素，最大池化层会输出该窗口内的最大值，平均池化层会输出该窗口内的平均值。\n",
+ "* 池化层的主要优点之一是减轻卷积层对位置的过度敏感。\n",
+ "* 我们可以指定池化层的填充和步幅。\n",
+ "* 使用最大池化层以及大于1的步幅，可减少空间维度（如高度和宽度）。\n",
+ "* 池化层的输出通道数与输入通道数相同。\n",
+ "\n",
+ "## 问题和练习\n",
+ "\n",
+ "1. 你能将平均池化层作为卷积层的特殊情况实现吗？\n",
+ ">设卷积层大小是$m\\times n$，卷积层里面每个元素参数是$\\dfrac{1} {m\\times n}$，这样就是一个平均池化层作为卷积层的实现\n",
+ "1. 假设池化层的输入大小为$c\\times h\\times w$，则汇聚窗口的形状为$p_h\\times p_w$，填充为$(p_h, p_w)$，步幅为$(s_h, s_w)$。这个池化层的计算成本是多少？\n",
+ ">$ c\\times \\left \\lfloor \\dfrac {h-p_h+s_h}{s_h}\\right \\rfloor \\times \\left \\lfloor \\dfrac {w-p_w+s_w}{s_w}\\right \\rfloor $\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 38,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.12"
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/code/23-LeNet.ipynb b/code/23-LeNet.ipynb
new file mode 100644
index 0000000..5584551
--- /dev/null
+++ b/code/23-LeNet.ipynb
@@ -0,0 +1,1314 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 卷积神经网络（LeNet）\n",
+ ":label:`sec_lenet`\n",
+ "\n",
+ "通过之前几节，我们学习了构建一个完整卷积神经网络的所需组件。\n",
+ "回想一下，之前我们将softmax回归模型（ :numref:`sec_softmax_scratch`）和多层感知机模型（ :numref:`sec_mlp_scratch`）应用于Fashion-MNIST数据集中的服装图片。\n",
+ "为了能够应用softmax回归和多层感知机，我们首先将每个大小为$28\\times28$的图像展平为一个784维的固定长度的一维向量，然后用全连接层对其进行处理。\n",
+ "而现在，我们已经掌握了卷积层的处理方法，我们可以在图像中保留空间结构。\n",
+ "同时，用卷积层代替全连接层的另一个好处是：模型更简洁、所需的参数更少。\n",
+ "\n",
+ "在本节中，我们将介绍LeNet，它是最早发布的卷积神经网络之一，因其在计算机视觉任务中的高效性能而受到广泛关注。\n",
+ "这个模型是由AT&T贝尔实验室的研究员Yann LeCun在1989年提出的（并以其命名），目的是识别图像 :cite:`LeCun.Bottou.Bengio.ea.1998`中的手写数字。\n",
+ "当时，Yann LeCun发表了第一篇通过反向传播成功训练卷积神经网络的研究，这项工作代表了十多年来神经网络研究开发的成果。\n",
+ "\n",
+ "当时，LeNet取得了与支持向量机（support vector machines）性能相媲美的成果，成为监督学习的主流方法。\n",
+ "LeNet被广泛用于自动取款机（ATM）机中，帮助识别处理支票的数字。\n",
+ "时至今日，一些自动取款机仍在运行Yann LeCun和他的同事Leon Bottou在上世纪90年代写的代码呢！\n",
+ "\n",
+ "## LeNet\n",
+ "\n",
+ "总体来看，(**LeNet（LeNet-5）由两个部分组成：**)(~~卷积编码器和全连接层密集块~~)\n",
+ "\n",
+ "* 卷积编码器：由两个卷积层组成;\n",
+ "* 全连接层密集块：由三个全连接层组成。\n",
+ "\n",
+ "该架构如 :numref:`img_lenet`所示。\n",
+ "\n",
+ "\n",
+ ":label:`img_lenet`\n",
+ "\n",
+ "每个卷积块中的基本单元是一个卷积层、一个sigmoid激活函数和平均汇聚层。请注意，虽然ReLU和最大汇聚层更有效，但它们在20世纪90年代还没有出现。每个卷积层使用$5\\times 5$卷积核和一个sigmoid激活函数。这些层将输入映射到多个二维特征输出，通常同时增加通道的数量。第一卷积层有6个输出通道，而第二个卷积层有16个输出通道。每个$2\\times2$池操作（步骤2）通过空间下采样将维数减少4倍。卷积的输出形状由批量大小、通道数、高度、宽度决定。\n",
+ "\n",
+ "为了将卷积块的输出传递给稠密块，我们必须在小批量中展平每个样本。换言之，我们将这个四维输入转换成全连接层所期望的二维输入。这里的二维表示的第一个维度索引小批量中的样本，第二个维度给出每个样本的平面向量表示。LeNet的稠密块有三个全连接层，分别有120、84和10个输出。因为我们仍在执行分类，所以输出层的10维对应于最后输出结果的数量。\n",
+ "\n",
+ "通过下面的LeNet代码，你会相信用深度学习框架实现此类模型非常简单。我们只需要实例化一个`Sequential`块并将需要的层连接在一起。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "#导入所需的库\n",
+ "import torch\n",
+ "from torch import nn\n",
+ "from d2l import torch as d2l\n",
+ "#定义网络结构\n",
+ "net = nn.Sequential(\n",
+ " nn.Conv2d(1, 6, kernel_size=5, padding=2), nn.Sigmoid(),\n",
+ " nn.AvgPool2d(kernel_size=2, stride=2),\n",
+ " nn.Conv2d(6, 16, kernel_size=5), nn.Sigmoid(),\n",
+ " nn.AvgPool2d(kernel_size=2, stride=2),\n",
+ " nn.Flatten(),\n",
+ " nn.Linear(16 * 5 * 5, 120), nn.Sigmoid(),\n",
+ " nn.Linear(120, 84), nn.Sigmoid(),\n",
+ " nn.Linear(84, 10))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 4
+ },
+ "source": [
+ "我们对原始模型做了一点小改动，去掉了最后一层的高斯激活。除此之外，这个网络与最初的LeNet-5一致。\n",
+ "\n",
+ "下面，我们将一个大小为$28 \\times 28$的单通道（黑白）图像通过LeNet。通过在每一层打印输出的形状，我们可以[**检查模型**]，以确保其操作与我们期望的 :numref:`img_lenet_vert`一致。\n",
+ "\n",
+ "\n",
+ ":label:`img_lenet_vert`\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Conv2d output shape: \t torch.Size([1, 6, 28, 28])\n",
+ "Sigmoid output shape: \t torch.Size([1, 6, 28, 28])\n",
+ "AvgPool2d output shape: \t torch.Size([1, 6, 14, 14])\n",
+ "Conv2d output shape: \t torch.Size([1, 16, 10, 10])\n",
+ "Sigmoid output shape: \t torch.Size([1, 16, 10, 10])\n",
+ "AvgPool2d output shape: \t torch.Size([1, 16, 5, 5])\n",
+ "Flatten output shape: \t torch.Size([1, 400])\n",
+ "Linear output shape: \t torch.Size([1, 120])\n",
+ "Sigmoid output shape: \t torch.Size([1, 120])\n",
+ "Linear output shape: \t torch.Size([1, 84])\n",
+ "Sigmoid output shape: \t torch.Size([1, 84])\n",
+ "Linear output shape: \t torch.Size([1, 10])\n"
+ ]
+ }
+ ],
+ "source": [
+ "#把每一层数据的shape给打印出来\n",
+ "X = torch.rand(size=(1, 1, 28, 28), dtype=torch.float32)#创建符合要求的张量\n",
+ "for layer in net:\n",
+ " X = layer(X)#通过每一层\n",
+ " print(layer.__class__.__name__,'output shape: \\t',X.shape)#打印"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 8
+ },
+ "source": [
+ "请注意，在整个卷积块中，与上一层相比，每一层特征的高度和宽度都减小了。\n",
+ "第一个卷积层使用2个像素的填充，来补偿$5 \\times 5$卷积核导致的特征减少。\n",
+ "相反，第二个卷积层没有填充，因此高度和宽度都减少了4个像素。\n",
+ "随着层叠的上升，通道的数量从输入时的1个，增加到第一个卷积层之后的6个，再到第二个卷积层之后的16个。\n",
+ "同时，每个汇聚层的高度和宽度都减半。最后，每个全连接层减少维数，最终输出一个维数与结果分类数相匹配的输出。\n",
+ "\n",
+ "## 模型训练\n",
+ "\n",
+ "现在我们已经实现了LeNet，让我们看看[**LeNet在Fashion-MNIST数据集上的表现**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 9,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "batch_size = 256#批量大小\n",
+ "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)#下载或加载数据集，得到训练和测试集的迭代对象"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 10
+ },
+ "source": [
+ "虽然卷积神经网络的参数较少，但与深度的多层感知机相比，它们的计算成本仍然很高，因为每个参数都参与更多的乘法。\n",
+ "如果你有机会使用GPU，可以用它加快训练。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 11,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "为了进行评估，我们需要[**对**] :numref:`sec_softmax_scratch`中描述的(**`evaluate_accuracy`函数进行轻微的修改**)。\n",
+ "由于完整的数据集位于内存中，因此在模型使用GPU计算数据集之前，我们需要将其复制到显存中。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "origin_pos": 13,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "def evaluate_accuracy_gpu(net, data_iter, device=None): #@save\n",
+ " \"\"\"使用GPU计算模型在数据集上的精度\"\"\"\n",
+ " if isinstance(net, nn.Module):\n",
+ " net.eval() # 设置为评估模式\n",
+ " if not device:\n",
+ " device = next(iter(net.parameters())).device\n",
+ " # 正确预测的数量，总预测的数量\n",
+ " metric = d2l.Accumulator(2)#创建一个累加器，包含2个要累加的元素\n",
+ " with torch.no_grad():\n",
+ " for X, y in data_iter:\n",
+ " if isinstance(X, list):\n",
+ " # BERT微调所需的（之后将介绍）\n",
+ " X = [x.to(device) for x in X]\n",
+ " else:\n",
+ " X = X.to(device)\n",
+ " y = y.to(device)\n",
+ " metric.add(d2l.accuracy(net(X), y), y.numel())#把每一组数据预测结果正确的个数和长度累加\n",
+ " return metric[0] / metric[1]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 14
+ },
+ "source": [
+ "[**为了使用GPU，我们还需要一点小改动**]。\n",
+ "与 :numref:`sec_softmax_scratch`中定义的`train_epoch_ch3`不同，在进行正向和反向传播之前，我们需要将每一小批量数据移动到我们指定的设备（例如GPU）上。\n",
+ "\n",
+ "如下所示，训练函数`train_ch6`也类似于 :numref:`sec_softmax_scratch`中定义的`train_ch3`。\n",
+ "由于我们将实现多层神经网络，因此我们将主要使用高级API。\n",
+ "以下训练函数假定从高级API创建的模型作为输入，并进行相应的优化。\n",
+ "我们使用在 :numref:`subsec_xavier`中介绍的Xavier随机初始化模型参数。\n",
+ "与全连接层一样，我们使用交叉熵损失函数和小批量随机梯度下降。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "origin_pos": 16,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "#@save\n",
+ "def train_ch6(net, train_iter, test_iter, num_epochs, lr, device):\n",
+ " \"\"\"用GPU训练模型(在第六章定义)\"\"\"\n",
+ " def init_weights(m):\n",
+ " if type(m) == nn.Linear or type(m) == nn.Conv2d:\n",
+ " nn.init.xavier_uniform_(m.weight)#对linear类型的层用xavier初始化\n",
+ " net.apply(init_weights)\n",
+ " print('training on', device)\n",
+ " net.to(device)\n",
+ " optimizer = torch.optim.SGD(net.parameters(), lr=lr)\n",
+ " loss = nn.CrossEntropyLoss()\n",
+ " animator = d2l.Animator(xlabel='epoch', xlim=[1, num_epochs],\n",
+ " legend=['train loss', 'train acc', 'test acc'])#动画需要\n",
+ " timer, num_batches = d2l.Timer(), len(train_iter)\n",
+ " for epoch in range(num_epochs):\n",
+ " # 训练损失之和，训练准确率之和，范例数\n",
+ " metric = d2l.Accumulator(3)\n",
+ " net.train()\n",
+ " for i, (X, y) in enumerate(train_iter):\n",
+ " timer.start()\n",
+ " optimizer.zero_grad()#梯度清零\n",
+ " X, y = X.to(device), y.to(device)\n",
+ " y_hat = net(X)#正向传播\n",
+ " l = loss(y_hat, y)#计算损失\n",
+ " l.backward()#反向传播\n",
+ " optimizer.step()#梯度下降\n",
+ " with torch.no_grad():\n",
+ " metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0])#训练损失之和，训练准确率之和，范例数\n",
+ " timer.stop()\n",
+ " train_l = metric[0] / metric[2]\n",
+ " train_acc = metric[1] / metric[2]\n",
+ " if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:\n",
+ " animator.add(epoch + (i + 1) / num_batches,\n",
+ " (train_l, train_acc, None))\n",
+ " test_acc = evaluate_accuracy_gpu(net, test_iter)#评估测试集的精度\n",
+ " animator.add(epoch + 1, (None, None, test_acc))\n",
+ " print(f'loss {train_l:.3f}, train acc {train_acc:.3f}, '\n",
+ " f'test acc {test_acc:.3f}')\n",
+ " print(f'{metric[2] * num_epochs / timer.sum():.1f} examples/sec '\n",
+ " f'on {str(device)}')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 18
+ },
+ "source": [
+ "现在，我们[**训练和评估LeNet-5模型**]。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "origin_pos": 19,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "loss 0.470, train acc 0.824, test acc 0.797\n",
+ "26005.4 examples/sec on cuda:0\n"
+ ]
+ },
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "lr, num_epochs = 0.9, 10\n",
+ "train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 20
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* 卷积神经网络（CNN）是一类使用卷积层的网络。\n",
+ "* 在卷积神经网络中，我们组合使用卷积层、非线性激活函数和汇聚层。\n",
+ "* 为了构造高性能的卷积神经网络，我们通常对卷积层进行排列，逐渐降低其表示的空间分辨率，同时增加通道数。\n",
+ "* 在传统的卷积神经网络中，卷积块编码得到的表征在输出之前需由一个或多个全连接层进行处理。\n",
+ "* LeNet是最早发布的卷积神经网络之一。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 将平均汇聚层替换为最大汇聚层，会发生什么？\n",
+ "1. 尝试构建一个基于LeNet的更复杂的网络，以提高其准确性。\n",
+ " 1. 调整卷积窗口大小。\n",
+ " 1. 调整输出通道的数量。\n",
+ " 1. 调整激活函数（如ReLU）。\n",
+ " 1. 调整卷积层的数量。\n",
+ " 1. 调整全连接层的数量。\n",
+ " 1. 调整学习率和其他训练细节（例如，初始化和轮数）。\n",
+ "1. 在MNIST数据集上尝试以上改进的网络。\n",
+ "1. 显示不同输入（例如毛衣和外套）时，LeNet第一层和第二层的激活值。\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 22,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1860)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/code/24-AlexNet.ipynb b/code/24-AlexNet.ipynb
new file mode 100644
index 0000000..85b9b2e
--- /dev/null
+++ b/code/24-AlexNet.ipynb
@@ -0,0 +1,337 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 0
+ },
+ "source": [
+ "# 深度卷积神经网络（AlexNet）\n",
+ ":label:`sec_alexnet`\n",
+ "\n",
+ "在LeNet提出后，卷积神经网络在计算机视觉和机器学习领域中很有名气。但卷积神经网络并没有主导这些领域。这是因为虽然LeNet在小数据集上取得了很好的效果，但是在更大、更真实的数据集上训练卷积神经网络的性能和可行性还有待研究。事实上，在上世纪90年代初到2012年之间的大部分时间里，神经网络往往被其他机器学习方法超越，如支持向量机（support vector machines）。\n",
+ "\n",
+ "在计算机视觉中，直接将神经网络与其他机器学习方法进行比较也许不公平。这是因为，卷积神经网络的输入是由原始像素值或是经过简单预处理（例如居中、缩放）的像素值组成的。但在使用传统机器学习方法时，从业者永远不会将原始像素作为输入。在传统机器学习方法中，计算机视觉流水线是由经过人的手工精心设计的特征流水线组成的。对于这些传统方法，大部分的进展都来自于对特征有了更聪明的想法，并且学习到的算法往往归于事后的解释。\n",
+ "\n",
+ "虽然上世纪90年代就有了一些神经网络加速卡，但仅靠它们还不足以开发出有大量参数的深层多通道多层卷积神经网络。此外，当时的数据集仍然相对较小。除了这些障碍，训练神经网络的一些关键技巧仍然缺失，包括启发式参数初始化、随机梯度下降的变体、非挤压激活函数和有效的正则化技术。\n",
+ "\n",
+ "因此，与训练*端到端*（从像素到分类结果）系统不同，经典机器学习的流水线看起来更像下面这样：\n",
+ "\n",
+ "1. 获取一个有趣的数据集。在早期，收集这些数据集需要昂贵的传感器（在当时最先进的图像也就100万像素）。\n",
+ "2. 根据光学、几何学、其他知识以及偶然的发现，手工对特征数据集进行预处理。\n",
+ "3. 通过标准的特征提取算法，如SIFT（尺度不变特征变换） :cite:`Lowe.2004`和SURF（加速鲁棒特征） :cite:`Bay.Tuytelaars.Van-Gool.2006`或其他手动调整的流水线来输入数据。\n",
+ "4. 将提取的特征送入最喜欢的分类器中（例如线性模型或其它核方法），以训练分类器。\n",
+ "\n",
+ "如果你和机器学习研究人员交谈，你会发现他们相信机器学习既重要又美丽：优雅的理论去证明各种模型的性质。机器学习是一个正在蓬勃发展、严谨且非常有用的领域。然而，如果你和计算机视觉研究人员交谈，你会听到一个完全不同的故事。他们会告诉你图像识别的诡异事实————推动领域进步的是数据特征，而不是学习算法。计算机视觉研究人员相信，从对最终模型精度的影响来说，更大或更干净的数据集、或是稍微改进的特征提取，比任何学习算法带来的进步要大得多。\n",
+ "\n",
+ "## 学习表征\n",
+ "\n",
+ "另一种预测这个领域发展的方法————观察图像特征的提取方法。在2012年前，图像特征都是机械地计算出来的。事实上，设计一套新的特征函数、改进结果，并撰写论文是盛极一时的潮流。SIFT :cite:`Lowe.2004`、SURF :cite:`Bay.Tuytelaars.Van-Gool.2006`、HOG（定向梯度直方图） :cite:`Dalal.Triggs.2005`、[bags of visual words](https://en.wikipedia.org/wiki/Bag-of-words_model_in_computer_vision)和类似的特征提取方法占据了主导地位。\n",
+ "\n",
+ "另一组研究人员，包括Yann LeCun、Geoff Hinton、Yoshua Bengio、Andrew Ng、Shun ichi Amari和Juergen Schmidhuber，想法则与众不同：他们认为特征本身应该被学习。此外，他们还认为，在合理地复杂性前提下，特征应该由多个共同学习的神经网络层组成，每个层都有可学习的参数。在机器视觉中，最底层可能检测边缘、颜色和纹理。事实上，Alex Krizhevsky、Ilya Sutskever和Geoff Hinton提出了一种新的卷积神经网络变体*AlexNet*。在2012年ImageNet挑战赛中取得了轰动一时的成绩。AlexNet以Alex Krizhevsky的名字命名，他是论文 :cite:`Krizhevsky.Sutskever.Hinton.2012`的第一作者。\n",
+ "\n",
+ "有趣的是，在网络的最底层，模型学习到了一些类似于传统滤波器的特征抽取器。 :numref:`fig_filters`是从AlexNet论文 :cite:`Krizhevsky.Sutskever.Hinton.2012`复制的，描述了底层图像特征。\n",
+ "\n",
+ "\n",
+ ":width:`400px`\n",
+ ":label:`fig_filters`\n",
+ "\n",
+ "AlexNet的更高层建立在这些底层表示的基础上，以表示更大的特征，如眼睛、鼻子、草叶等等。而更高的层可以检测整个物体，如人、飞机、狗或飞盘。最终的隐藏神经元可以学习图像的综合表示，从而使属于不同类别的数据易于区分。尽管一直有一群执着的研究者不断钻研，试图学习视觉数据的逐级表征，然而很长一段时间里这些尝试都未有突破。深度卷积神经网络的突破出现在2012年。突破可归因于两个关键因素。\n",
+ "\n",
+ "### 缺少的成分：数据\n",
+ "\n",
+ "包含许多特征的深度模型需要大量的有标签数据，才能显著优于基于凸优化的传统方法（如线性方法和核方法）。\n",
+ "然而，限于早期计算机有限的存储和90年代有限的研究预算，大部分研究只基于小的公开数据集。例如，不少研究论文基于加州大学欧文分校（UCI）提供的若干个公开数据集，其中许多数据集只有几百至几千张在非自然环境下以低分辨率拍摄的图像。这一状况在2010年前后兴起的大数据浪潮中得到改善。2009年，ImageNet数据集发布，并发起ImageNet挑战赛：要求研究人员从100万个样本中训练模型，以区分1000个不同类别的对象。ImageNet数据集由斯坦福教授李飞飞小组的研究人员开发，利用谷歌图像搜索（Google Image Search）对每一类图像进行预筛选，并利用亚马逊众包（Amazon Mechanical Turk）来标注每张图片的相关类别。这种规模是前所未有的。这项被称为ImageNet的挑战赛推动了计算机视觉和机器学习研究的发展，挑战研究人员确定哪些模型能够在更大的数据规模下表现最好。\n",
+ "\n",
+ "### 缺少的成分：硬件\n",
+ "\n",
+ "深度学习对计算资源要求很高，训练可能需要数百个迭代轮数，每次迭代都需要通过代价高昂的许多线性代数层传递数据。这也是为什么在20世纪90年代至21世纪初，优化凸目标的简单算法是研究人员的首选。然而，用GPU训练神经网络改变了这一格局。*图形处理器*（Graphics Processing Unit，GPU）早年用来加速图形处理，使电脑游戏玩家受益。GPU可优化高吞吐量的$4 \\times 4$矩阵和向量乘法，从而服务于基本的图形任务。幸运的是，这些数学运算与卷积层的计算惊人地相似。由此，英伟达（NVIDIA）和ATI已经开始为通用计算操作优化gpu，甚至把它们作为*通用GPU*（general-purpose GPUs，GPGPU）来销售。\n",
+ "\n",
+ "那么GPU比CPU强在哪里呢？\n",
+ "\n",
+ "首先，我们深度理解一下中央处理器（Central Processing Unit，CPU）的*核心*。\n",
+ "CPU的每个核心都拥有高时钟频率的运行能力，和高达数MB的三级缓存（L3Cache）。\n",
+ "它们非常适合执行各种指令，具有分支预测器、深层流水线和其他使CPU能够运行各种程序的功能。\n",
+ "然而，这种明显的优势也是它的致命弱点：通用核心的制造成本非常高。\n",
+ "它们需要大量的芯片面积、复杂的支持结构（内存接口、内核之间的缓存逻辑、高速互连等等），而且它们在任何单个任务上的性能都相对较差。\n",
+ "现代笔记本电脑最多有4核，即使是高端服务器也很少超过64核，因为它们的性价比不高。\n",
+ "\n",
+ "相比于CPU，GPU由$100 \\sim 1000$个小的处理单元组成（NVIDIA、ATI、ARM和其他芯片供应商之间的细节稍有不同），通常被分成更大的组（NVIDIA称之为warps）。\n",
+ "虽然每个GPU核心都相对较弱，有时甚至以低于1GHz的时钟频率运行，但庞大的核心数量使GPU比CPU快几个数量级。\n",
+ "例如，NVIDIA最近一代的Ampere GPU架构为每个芯片提供了高达312 TFlops的浮点性能，而CPU的浮点性能到目前为止还没有超过1 TFlops。\n",
+ "之所以有如此大的差距，原因其实很简单：首先，功耗往往会随时钟频率呈二次方增长。\n",
+ "对于一个CPU核心，假设它的运行速度比GPU快4倍，你可以使用16个GPU内核取代，那么GPU的综合性能就是CPU的$16 \\times 1/4 = 4$倍。\n",
+ "其次，GPU内核要简单得多，这使得它们更节能。\n",
+ "此外，深度学习中的许多操作需要相对较高的内存带宽，而GPU拥有10倍于CPU的带宽。\n",
+ "\n",
+ "回到2012年的重大突破，当Alex Krizhevsky和Ilya Sutskever实现了可以在GPU硬件上运行的深度卷积神经网络时，一个重大突破出现了。他们意识到卷积神经网络中的计算瓶颈：卷积和矩阵乘法，都是可以在硬件上并行化的操作。\n",
+ "于是，他们使用两个显存为3GB的NVIDIA GTX580 GPU实现了快速卷积运算。他们的创新[cuda-convnet](https://code.google.com/archive/p/cuda-convnet/)几年来它一直是行业标准，并推动了深度学习热潮。\n",
+ "\n",
+ "## AlexNet\n",
+ "\n",
+ "2012年，AlexNet横空出世。它首次证明了学习到的特征可以超越手工设计的特征。它一举打破了计算机视觉研究的现状。\n",
+ "AlexNet使用了8层卷积神经网络，并以很大的优势赢得了2012年ImageNet图像识别挑战赛。\n",
+ "\n",
+ "AlexNet和LeNet的架构非常相似，如 :numref:`fig_alexnet`所示。\n",
+ "注意，这里我们提供了一个稍微精简版本的AlexNet，去除了当年需要两个小型GPU同时运算的设计特点。\n",
+ "\n",
+ "\n",
+ ":label:`fig_alexnet`\n",
+ "\n",
+ "AlexNet和LeNet的设计理念非常相似，但也存在显著差异。\n",
+ "首先，AlexNet比相对较小的LeNet5要深得多。\n",
+ "AlexNet由八层组成：五个卷积层、两个全连接隐藏层和一个全连接输出层。\n",
+ "其次，AlexNet使用ReLU而不是sigmoid作为其激活函数。\n",
+ "下面，让我们深入研究AlexNet的细节。\n",
+ "\n",
+ "### 模型设计\n",
+ "\n",
+ "在AlexNet的第一层，卷积窗口的形状是$11\\times11$。\n",
+ "由于ImageNet中大多数图像的宽和高比MNIST图像的多10倍以上，因此，需要一个更大的卷积窗口来捕获目标。\n",
+ "第二层中的卷积窗口形状被缩减为$5\\times5$，然后是$3\\times3$。\n",
+ "此外，在第一层、第二层和第五层卷积层之后，加入窗口形状为$3\\times3$、步幅为2的最大汇聚层。\n",
+ "而且，AlexNet的卷积通道数目是LeNet的10倍。\n",
+ "\n",
+ "在最后一个卷积层后有两个全连接层，分别有4096个输出。\n",
+ "这两个巨大的全连接层拥有将近1GB的模型参数。\n",
+ "由于早期GPU显存有限，原版的AlexNet采用了双数据流设计，使得每个GPU只负责存储和计算模型的一半参数。\n",
+ "幸运的是，现在GPU显存相对充裕，所以我们现在很少需要跨GPU分解模型（因此，我们的AlexNet模型在这方面与原始论文稍有不同）。\n",
+ "\n",
+ "### 激活函数\n",
+ "\n",
+ "此外，AlexNet将sigmoid激活函数改为更简单的ReLU激活函数。\n",
+ "一方面，ReLU激活函数的计算更简单，它不需要如sigmoid激活函数那般复杂的求幂运算。\n",
+ "另一方面，当使用不同的参数初始化方法时，ReLU激活函数使训练模型更加容易。\n",
+ "当sigmoid激活函数的输出非常接近于0或1时，这些区域的梯度几乎为0，因此反向传播无法继续更新一些模型参数。\n",
+ "相反，ReLU激活函数在正区间的梯度总是1。\n",
+ "因此，如果模型参数没有正确初始化，sigmoid函数可能在正区间内得到几乎为0的梯度，从而使模型无法得到有效的训练。\n",
+ "\n",
+ "### 容量控制和预处理\n",
+ "\n",
+ "AlexNet通过dropout（ :numref:`sec_dropout`）控制全连接层的模型复杂度，而LeNet只使用了权重衰减。\n",
+ "为了进一步扩充数据，AlexNet在训练时增加了大量的图像增强数据，如翻转、裁切和变色。\n",
+ "这使得模型更健壮，更大的样本量有效地减少了过拟合。\n",
+ "我们将在 :numref:`sec_image_augmentation`中更详细地讨论数据扩充。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from d2l import torch as d2l\n",
+ "\n",
+ "net = nn.Sequential(\n",
+ " # 这里，我们使用一个11*11的更大窗口来捕捉对象。\n",
+ " # 同时，步幅为4，以减少输出的高度和宽度。\n",
+ " # 另外，输出通道的数目远大于LeNet\n",
+ " nn.Conv2d(1, 96, kernel_size=11, stride=4, padding=1), nn.ReLU(),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2),\n",
+ " # 减小卷积窗口，使用填充为2来使得输入与输出的高和宽一致，且增大输出通道数\n",
+ " nn.Conv2d(96, 256, kernel_size=5, padding=2), nn.ReLU(),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2),\n",
+ " # 使用三个连续的卷积层和较小的卷积窗口。\n",
+ " # 除了最后的卷积层，输出通道的数量进一步增加。\n",
+ " # 在前两个卷积层之后，汇聚层不用于减少输入的高度和宽度\n",
+ " nn.Conv2d(256, 384, kernel_size=3, padding=1), nn.ReLU(),\n",
+ " nn.Conv2d(384, 384, kernel_size=3, padding=1), nn.ReLU(),\n",
+ " nn.Conv2d(384, 256, kernel_size=3, padding=1), nn.ReLU(),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2),\n",
+ " nn.Flatten(),\n",
+ " # 这里，全连接层的输出数量是LeNet中的好几倍。使用dropout层来减轻过拟合\n",
+ " nn.Linear(6400, 4096), nn.ReLU(),\n",
+ " nn.Dropout(p=0.5),\n",
+ " nn.Linear(4096, 4096), nn.ReLU(),\n",
+ " nn.Dropout(p=0.5),\n",
+ " # 最后是输出层。由于这里使用Fashion-MNIST，所以用类别数为10，而非论文中的1000\n",
+ " nn.Linear(4096, 10))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 4
+ },
+ "source": [
+ "[**我们构造一个**]高度和宽度都为224的(**单通道数据，来观察每一层输出的形状**)。\n",
+ "它与 :numref:`fig_alexnet`中的AlexNet架构相匹配。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "origin_pos": 6,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Conv2d output shape:\t torch.Size([1, 96, 54, 54])\n",
+ "ReLU output shape:\t torch.Size([1, 96, 54, 54])\n",
+ "MaxPool2d output shape:\t torch.Size([1, 96, 26, 26])\n",
+ "Conv2d output shape:\t torch.Size([1, 256, 26, 26])\n",
+ "ReLU output shape:\t torch.Size([1, 256, 26, 26])\n",
+ "MaxPool2d output shape:\t torch.Size([1, 256, 12, 12])\n",
+ "Conv2d output shape:\t torch.Size([1, 384, 12, 12])\n",
+ "ReLU output shape:\t torch.Size([1, 384, 12, 12])\n",
+ "Conv2d output shape:\t torch.Size([1, 384, 12, 12])\n",
+ "ReLU output shape:\t torch.Size([1, 384, 12, 12])\n",
+ "Conv2d output shape:\t torch.Size([1, 256, 12, 12])\n",
+ "ReLU output shape:\t torch.Size([1, 256, 12, 12])\n",
+ "MaxPool2d output shape:\t torch.Size([1, 256, 5, 5])\n",
+ "Flatten output shape:\t torch.Size([1, 6400])\n",
+ "Linear output shape:\t torch.Size([1, 4096])\n",
+ "ReLU output shape:\t torch.Size([1, 4096])\n",
+ "Dropout output shape:\t torch.Size([1, 4096])\n",
+ "Linear output shape:\t torch.Size([1, 4096])\n",
+ "ReLU output shape:\t torch.Size([1, 4096])\n",
+ "Dropout output shape:\t torch.Size([1, 4096])\n",
+ "Linear output shape:\t torch.Size([1, 10])\n"
+ ]
+ }
+ ],
+ "source": [
+ "X = torch.randn(1, 1, 224, 224)\n",
+ "for layer in net:\n",
+ " X=layer(X)\n",
+ " print(layer.__class__.__name__,'output shape:\\t',X.shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 8
+ },
+ "source": [
+ "## 读取数据集\n",
+ "\n",
+ "尽管本文中AlexNet是在ImageNet上进行训练的，但我们在这里使用的是Fashion-MNIST数据集。因为即使在现代GPU上，训练ImageNet模型，同时使其收敛可能需要数小时或数天的时间。\n",
+ "将AlexNet直接应用于Fashion-MNIST的一个问题是，[**Fashion-MNIST图像的分辨率**]（$28 \\times 28$像素）(**低于ImageNet图像。**)\n",
+ "为了解决这个问题，(**我们将它们增加到$224 \\times 224$**)（通常来讲这不是一个明智的做法，但我们在这里这样做是为了有效使用AlexNet架构）。\n",
+ "我们使用`d2l.load_data_fashion_mnist`函数中的`resize`参数执行此调整。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "origin_pos": 9,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "batch_size = 128\n",
+ "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=224)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 10
+ },
+ "source": [
+ "## [**训练AlexNet**]\n",
+ "\n",
+ "现在，我们可以开始训练AlexNet了。与 :numref:`sec_lenet`中的LeNet相比，这里的主要变化是使用更小的学习速率训练，这是因为网络更深更广、图像分辨率更高，训练卷积神经网络就更昂贵。\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "origin_pos": 11,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "training on cpu\n"
+ ]
+ }
+ ],
+ "source": [
+ "lr, num_epochs = 0.01, 5\n",
+ "d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 12
+ },
+ "source": [
+ "## 小结\n",
+ "\n",
+ "* AlexNet的架构与LeNet相似，但使用了更多的卷积层和更多的参数来拟合大规模的ImageNet数据集。\n",
+ "* 今天，AlexNet已经被更有效的架构所超越，但它是从浅层网络到深层网络的关键一步。\n",
+ "* 尽管AlexNet的代码只比LeNet多出几行，但学术界花了很多年才接受深度学习这一概念，并应用其出色的实验结果。这也是由于缺乏有效的计算工具。\n",
+ "* Dropout、ReLU和预处理是提升计算机视觉任务性能的其他关键步骤。\n",
+ "\n",
+ "## 练习\n",
+ "\n",
+ "1. 试着增加迭代轮数。对比LeNet的结果有什么不同？为什么？\n",
+ "1. AlexNet对于Fashion-MNIST数据集来说可能太复杂了。\n",
+ " 1. 尝试简化模型以加快训练速度，同时确保准确性不会显著下降。\n",
+ " 1. 设计一个更好的模型，可以直接在$28 \\times 28$图像上工作。\n",
+ "1. 修改批量大小，并观察模型精度和GPU显存变化。\n",
+ "1. 分析了AlexNet的计算性能。\n",
+ " 1. 在AlexNet中主要是哪部分占用显存？\n",
+ " 1. 在AlexNet中主要是哪部分需要更多的计算？\n",
+ " 1. 计算结果时显存带宽如何？\n",
+ "1. 将dropout和ReLU应用于LeNet-5，效果有提升吗？再试试预处理会怎么样？\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "origin_pos": 14,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "source": [
+ "[Discussions](https://discuss.d2l.ai/t/1863)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.2"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/code/25-VGG.ipynb b/code/25-VGG.ipynb
new file mode 100644
index 0000000..4ee9650
--- /dev/null
+++ b/code/25-VGG.ipynb
@@ -0,0 +1,1007 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "c2247b6e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from d2l import torch as d2l\n",
+ "\n",
+ "#VGG块\n",
+ "def vgg_block(num_convs, in_channels, out_channels):#块中卷积层数，输入输出通道数\n",
+ " layers = []\n",
+ " for _ in range(num_convs):#用for循环\n",
+ " layers.append(nn.Conv2d(\n",
+ " in_channels, out_channels, kernel_size=3, padding=1))\n",
+ " layers.append(nn.ReLU())\n",
+ " in_channels = out_channels#添加一层后取当前输出通道数为下一层输入通道数，\n",
+ " #这里说明VGG块改变通道数的方法是在第一层就将通道数改变好，后面层中通道数全不变\n",
+ " layers.append(nn.MaxPool2d(kernel_size=2, stride=2))\n",
+ " return nn.Sequential(*layers)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "0bf25540",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#每个VGG块的（卷积层数，输出通道数）\n",
+ "conv_arch = ((1, 64), (1, 128), (2, 256), (2, 512), (2, 512))\n",
+ "\n",
+ "#VGG网络\n",
+ "def vgg(conv_arch):\n",
+ " conv_blks = []\n",
+ " in_channels = 1#初始输入图像为单通道\n",
+ " for (num_convs, out_channels) in conv_arch:#依次读取VGG块尺寸并创建\n",
+ " conv_blks.append(vgg_block(\n",
+ " num_convs, in_channels, out_channels))\n",
+ " in_channels = out_channels#输入通道数随每层输出通道数更新\n",
+ " \n",
+ " return nn.Sequential(\n",
+ " *conv_blks, nn.Flatten(),#“*”将列表中所有元素解开成独立的参数\n",
+ " nn.Linear(out_channels * 7 * 7, 4096), nn.ReLU(),\n",
+ " nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(),\n",
+ " nn.Dropout(0.5), nn.Linear(4096, 10))\n",
+ "\n",
+ "net = vgg(conv_arch)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "b8c794b4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Sequential ouput shape:\t torch.Size([1, 64, 112, 112])\n",
+ "Sequential ouput shape:\t torch.Size([1, 128, 56, 56])\n",
+ "Sequential ouput shape:\t torch.Size([1, 256, 28, 28])\n",
+ "Sequential ouput shape:\t torch.Size([1, 512, 14, 14])\n",
+ "Sequential ouput shape:\t torch.Size([1, 512, 7, 7])\n",
+ "Flatten ouput shape:\t torch.Size([1, 25088])\n",
+ "Linear ouput shape:\t torch.Size([1, 4096])\n",
+ "ReLU ouput shape:\t torch.Size([1, 4096])\n",
+ "Dropout ouput shape:\t torch.Size([1, 4096])\n",
+ "Linear ouput shape:\t torch.Size([1, 4096])\n",
+ "ReLU ouput shape:\t torch.Size([1, 4096])\n",
+ "Dropout ouput shape:\t torch.Size([1, 4096])\n",
+ "Linear ouput shape:\t torch.Size([1, 10])\n"
+ ]
+ }
+ ],
+ "source": [
+ "X = torch.randn(size=(1, 1, 224, 224))\n",
+ "for blk in net:\n",
+ " X = blk(X)\n",
+ " print(blk.__class__.__name__, 'ouput shape:\\t', X.shape)\n",
+ " \n",
+ "#总体而言，网络分为五块，每一块将输入宽高减半，通道数翻倍"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "27e6185a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#VGG-11计算量太大，这里构建了一个四分之一尺寸的网络来训练,计算量为1/16,但依然很大\n",
+ "\n",
+ "ratio = 4\n",
+ "small_conv_arch = [(pair[0], pair[1] // ratio) for pair in conv_arch]\n",
+ "net = vgg(small_conv_arch)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "fc93a8c0",
+ "metadata": {},
+ "outputs": [
+ {
+ "ename": "KeyboardInterrupt",
+ "evalue": "",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/var/folders/k1/yryg_xkx7078z6t5kzg6bmth0000gn/T/ipykernel_34700/2154047581.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_epochs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0.05\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m128\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mtrain_iter\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_iter\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0md2l\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_data_fashion_mnist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresize\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m224\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0md2l\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_ch6\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnet\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_iter\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_iter\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_epochs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0md2l\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtry_gpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/d2l/torch.py\u001b[0m in \u001b[0;36mtrain_ch6\u001b[0;34m(net, train_iter, test_iter, num_epochs, lr, device)\u001b[0m\n\u001b[1;32m 514\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 515\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 516\u001b[0;31m \u001b[0my_hat\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 517\u001b[0m \u001b[0ml\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 518\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m 139\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 141\u001b[0;31m \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 142\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m 139\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 141\u001b[0;31m \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 142\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/modules/pooling.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m 160\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 161\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 162\u001b[0;31m return F.max_pool2d(input, self.kernel_size, self.stride,\n\u001b[0m\u001b[1;32m 163\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpadding\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdilation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mceil_mode\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 164\u001b[0m self.return_indices)\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/_jit_internal.py\u001b[0m in \u001b[0;36mfn\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 420\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mif_true\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 421\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 422\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mif_false\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 423\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 424\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mif_true\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mif_false\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.9/site-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36m_max_pool2d\u001b[0;34m(input, kernel_size, stride, padding, dilation, ceil_mode, return_indices)\u001b[0m\n\u001b[1;32m 717\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mstride\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 718\u001b[0m \u001b[0mstride\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mannotate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 719\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_pool2d\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstride\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpadding\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdilation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mceil_mode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 720\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 721\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
+ ]
+ },
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "lr, num_epochs, batch_size = 0.05, 10, 128\n",
+ "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize = 224)\n",
+ "d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "817054d7",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/code/26-NiN.ipynb b/code/26-NiN.ipynb
new file mode 100644
index 0000000..0ac1946
--- /dev/null
+++ b/code/26-NiN.ipynb
@@ -0,0 +1,1249 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "YnAtHwhmmGOh",
+ "outputId": "9c2fd82f-fe05-4f0f-cd17-1923e9307495",
+ "slideshow": {
+ "slide_type": "skip"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Collecting git+https://github.com/d2l-ai/d2l-zh@release\n",
+ " Cloning https://github.com/d2l-ai/d2l-zh (to revision release) to /tmp/pip-req-build-rwrn0sr4\n",
+ " Running command git clone -q https://github.com/d2l-ai/d2l-zh /tmp/pip-req-build-rwrn0sr4\n",
+ " Running command git checkout -b release --track origin/release\n",
+ " Switched to a new branch 'release'\n",
+ " Branch 'release' set up to track remote branch 'release' from 'origin'.\n",
+ " Running command git submodule update --init --recursive -q\n",
+ "Requirement already satisfied: jupyter in /usr/local/lib/python3.7/dist-packages (from d2l==2.0.0b0) (1.0.0)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from d2l==2.0.0b0) (1.19.5)\n",
+ "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from d2l==2.0.0b0) (3.2.2)\n",
+ "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from d2l==2.0.0b0) (2.23.0)\n",
+ "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from d2l==2.0.0b0) (1.1.5)\n",
+ "Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from jupyter->d2l==2.0.0b0) (5.6.1)\n",
+ "Requirement already satisfied: ipykernel in /usr/local/lib/python3.7/dist-packages (from jupyter->d2l==2.0.0b0) (4.10.1)\n",
+ "Requirement already satisfied: notebook in /usr/local/lib/python3.7/dist-packages (from jupyter->d2l==2.0.0b0) (5.3.1)\n",
+ "Requirement already satisfied: qtconsole in /usr/local/lib/python3.7/dist-packages (from jupyter->d2l==2.0.0b0) (5.2.2)\n",
+ "Requirement already satisfied: ipywidgets in /usr/local/lib/python3.7/dist-packages (from jupyter->d2l==2.0.0b0) (7.6.5)\n",
+ "Requirement already satisfied: jupyter-console in /usr/local/lib/python3.7/dist-packages (from jupyter->d2l==2.0.0b0) (5.2.0)\n",
+ "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.7/dist-packages (from ipykernel->jupyter->d2l==2.0.0b0) (5.3.5)\n",
+ "Requirement already satisfied: traitlets>=4.1.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel->jupyter->d2l==2.0.0b0) (5.1.1)\n",
+ "Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel->jupyter->d2l==2.0.0b0) (5.1.1)\n",
+ "Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel->jupyter->d2l==2.0.0b0) (5.5.0)\n",
+ "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (0.8.1)\n",
+ "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (4.4.2)\n",
+ "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (0.7.5)\n",
+ "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (1.0.18)\n",
+ "Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (4.8.0)\n",
+ "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (57.4.0)\n",
+ "Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (2.6.1)\n",
+ "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (1.15.0)\n",
+ "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel->jupyter->d2l==2.0.0b0) (0.2.5)\n",
+ "Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->jupyter->d2l==2.0.0b0) (3.5.2)\n",
+ "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->jupyter->d2l==2.0.0b0) (1.0.2)\n",
+ "Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->jupyter->d2l==2.0.0b0) (5.1.3)\n",
+ "Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->jupyter->d2l==2.0.0b0) (0.2.0)\n",
+ "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (4.3.3)\n",
+ "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (4.9.1)\n",
+ "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (0.18.0)\n",
+ "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (5.4.0)\n",
+ "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (3.10.0.2)\n",
+ "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (4.10.0)\n",
+ "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (21.4.0)\n",
+ "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from importlib-resources>=1.4.0->jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->d2l==2.0.0b0) (3.7.0)\n",
+ "Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from notebook->jupyter->d2l==2.0.0b0) (0.12.1)\n",
+ "Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from notebook->jupyter->d2l==2.0.0b0) (1.8.0)\n",
+ "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from notebook->jupyter->d2l==2.0.0b0) (2.11.3)\n",
+ "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel->jupyter->d2l==2.0.0b0) (2.8.2)\n",
+ "Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel->jupyter->d2l==2.0.0b0) (22.3.0)\n",
+ "Requirement already satisfied: ptyprocess in /usr/local/lib/python3.7/dist-packages (from terminado>=0.8.1->notebook->jupyter->d2l==2.0.0b0) (0.7.0)\n",
+ "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->notebook->jupyter->d2l==2.0.0b0) (2.0.1)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->d2l==2.0.0b0) (1.3.2)\n",
+ "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->d2l==2.0.0b0) (3.0.6)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->d2l==2.0.0b0) (0.11.0)\n",
+ "Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter->d2l==2.0.0b0) (0.7.1)\n",
+ "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter->d2l==2.0.0b0) (0.3)\n",
+ "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter->d2l==2.0.0b0) (1.5.0)\n",
+ "Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter->d2l==2.0.0b0) (0.5.0)\n",
+ "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter->d2l==2.0.0b0) (0.8.4)\n",
+ "Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter->d2l==2.0.0b0) (4.1.0)\n",
+ "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->jupyter->d2l==2.0.0b0) (21.3)\n",
+ "Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->jupyter->d2l==2.0.0b0) (0.5.1)\n",
+ "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->d2l==2.0.0b0) (2018.9)\n",
+ "Requirement already satisfied: qtpy in /usr/local/lib/python3.7/dist-packages (from qtconsole->jupyter->d2l==2.0.0b0) (2.0.0)\n",
+ "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->d2l==2.0.0b0) (3.0.4)\n",
+ "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->d2l==2.0.0b0) (1.24.3)\n",
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->d2l==2.0.0b0) (2021.10.8)\n",
+ "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->d2l==2.0.0b0) (2.10)\n",
+ "Building wheels for collected packages: d2l\n",
+ " Building wheel for d2l (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ " Created wheel for d2l: filename=d2l-2.0.0b0-py3-none-any.whl size=80071 sha256=52bdc5edc13a6233bf007431cd81aac7f01ddffc0b2e0a61eeae9f9a17c5c9e1\n",
+ " Stored in directory: /tmp/pip-ephem-wheel-cache-9zyqtm5h/wheels/73/f4/42/d2b85ca46d85a241d6aa57c1c24027de2d2258202bb67945f9\n",
+ "Successfully built d2l\n",
+ "Installing collected packages: d2l\n",
+ "Successfully installed d2l-2.0.0b0\n"
+ ]
+ }
+ ],
+ "source": [
+ "!pip install git+https://github.com/d2l-ai/d2l-zh@release # installing d2l\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "3J6COHAcmGOj",
+ "origin_pos": 0,
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "# 网络中的网络（NiN）\n",
+ "\n",
+ "## (**NiN块结构**)\n",
+ "\n",
+ "\n",
+ ":width:`600px`\n",
+ ":label:`fig_nin`\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "id": "icjYVLrqmGOk",
+ "origin_pos": 2,
+ "slideshow": {
+ "slide_type": "slide"
+ },
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from d2l import torch as d2l\n",
+ "\n",
+ "#定义NiN块\n",
+ "def nin_block(in_channels, out_channels, kernel_size, strides, padding):\n",
+ " return nn.Sequential(\n",
+ " nn.Conv2d(in_channels, out_channels, kernel_size, strides, padding),\n",
+ " nn.ReLU(),\n",
+ " nn.Conv2d(out_channels, out_channels, kernel_size=1), nn.ReLU(),\n",
+ " nn.Conv2d(out_channels, out_channels, kernel_size=1), nn.ReLU())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "m76Gv4E2mGOl",
+ "origin_pos": 4,
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "## [**NiN模型**]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "id": "ZM95zdrnmGOm",
+ "origin_pos": 6,
+ "slideshow": {
+ "slide_type": "-"
+ },
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "net = nn.Sequential(\n",
+ " nin_block(1, 96, kernel_size=11, strides=4, padding=0),\n",
+ " nn.MaxPool2d(3, stride=2),\n",
+ " nin_block(96, 256, kernel_size=5, strides=1, padding=2),\n",
+ " nn.MaxPool2d(3, stride=2),\n",
+ " nin_block(256, 384, kernel_size=3, strides=1, padding=1),\n",
+ " nn.MaxPool2d(3, stride=2),\n",
+ " nn.Dropout(0.5),\n",
+ " # 标签类别数是10\n",
+ " nin_block(384, 10, kernel_size=3, strides=1, padding=1),\n",
+ " nn.AdaptiveAvgPool2d((1, 1)), #全局平均池化，高宽都变成1\n",
+ " # 将四维的输出转成二维的输出，其形状为(批量大小,10)\n",
+ " nn.Flatten())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qpeUzjQ5mGOn",
+ "origin_pos": 8,
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "**demo测试，查看每个块的输出情况**\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "jOu7BZpsmGOo",
+ "origin_pos": 10,
+ "outputId": "80275b87-d88b-4d17-fbc1-90bf1dc46241",
+ "slideshow": {
+ "slide_type": "-"
+ },
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Sequential output shape:\t torch.Size([1, 96, 54, 54])\n",
+ "MaxPool2d output shape:\t torch.Size([1, 96, 26, 26])\n",
+ "Sequential output shape:\t torch.Size([1, 256, 26, 26])\n",
+ "MaxPool2d output shape:\t torch.Size([1, 256, 12, 12])\n",
+ "Sequential output shape:\t torch.Size([1, 384, 12, 12])\n",
+ "MaxPool2d output shape:\t torch.Size([1, 384, 5, 5])\n",
+ "Dropout output shape:\t torch.Size([1, 384, 5, 5])\n",
+ "Sequential output shape:\t torch.Size([1, 10, 5, 5])\n",
+ "AdaptiveAvgPool2d output shape:\t torch.Size([1, 10, 1, 1])\n",
+ "Flatten output shape:\t torch.Size([1, 10])\n"
+ ]
+ }
+ ],
+ "source": [
+ "X = torch.rand(size=(1, 1, 224, 224))\n",
+ "for layer in net:\n",
+ " X = layer(X)\n",
+ " print(layer.__class__.__name__,'output shape:\\t', X.shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "dsBcacGnmGOq",
+ "origin_pos": 12,
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "**训练模型**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 261
+ },
+ "id": "ERZoQ6v-mGOr",
+ "origin_pos": 13,
+ "outputId": "6b85b5ef-f4f3-4195-bb5c-0ad95998bfd0",
+ "slideshow": {
+ "slide_type": "-"
+ },
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "lr, num_epochs, batch_size = 0.1, 10, 128\n",
+ "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=224)\n",
+ "d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "celltoolbar": "幻灯片",
+ "colab": {
+ "name": "nin.ipynb",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/code/27-googlenet.ipynb b/code/27-googlenet.ipynb
new file mode 100644
index 0000000..218c185
--- /dev/null
+++ b/code/27-googlenet.ipynb
@@ -0,0 +1,1162 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "-"
+ }
+ },
+ "source": [
+ "# 含并行连结的网络（GoogLeNet）\n",
+ "\n",
+ "Inception块"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "origin_pos": 2,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "from torch import nn\n",
+ "from torch.nn import functional as F\n",
+ "from d2l import torch as d2l\n",
+ "\n",
+ "\n",
+ "class Inception(nn.Module):\n",
+ " def __init__(self, in_channels, c1, c2, c3, c4, **kwargs):\n",
+ " super(Inception, self).__init__(**kwargs)\n",
+ " self.p1_1 = nn.Conv2d(in_channels, c1, kernel_size=1)\n",
+ " #1X1卷积\n",
+ " self.p2_1 = nn.Conv2d(in_channels, c2[0], kernel_size=1)\n",
+ " self.p2_2 = nn.Conv2d(c2[0], c2[1], kernel_size=3, padding=1)\n",
+ " #1X1卷积，3X3卷积\n",
+ " self.p3_1 = nn.Conv2d(in_channels, c3[0], kernel_size=1)\n",
+ " self.p3_2 = nn.Conv2d(c3[0], c3[1], kernel_size=5, padding=2)\n",
+ " #1X1卷积，5X5卷积\n",
+ " self.p4_1 = nn.MaxPool2d(kernel_size=3, stride=1, padding=1)\n",
+ " self.p4_2 = nn.Conv2d(in_channels, c4, kernel_size=1)\n",
+ " #maxpool,1x1卷积\n",
+ " def forward(self, x):\n",
+ " p1 = F.relu(self.p1_1(x))\n",
+ " p2 = F.relu(self.p2_2(F.relu(self.p2_1(x))))\n",
+ " p3 = F.relu(self.p3_2(F.relu(self.p3_1(x))))\n",
+ " p4 = F.relu(self.p4_2(self.p4_1(x)))\n",
+ " #定义四个模块，然后将输出在通道维度上拼接在一起\n",
+ " return torch.cat((p1, p2, p3, p4), dim=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "GoogLeNet模型"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "origin_pos": 22,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "b1 = nn.Sequential(nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3),\n",
+ " nn.ReLU(),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2, padding=1))\n",
+ "\n",
+ "b2 = nn.Sequential(nn.Conv2d(64, 64, kernel_size=1),\n",
+ " nn.ReLU(),\n",
+ " nn.Conv2d(64, 192, kernel_size=3, padding=1),\n",
+ " nn.ReLU(),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2, padding=1))\n",
+ "\n",
+ "b3 = nn.Sequential(Inception(192, 64, (96, 128), (16, 32), 32),\n",
+ " Inception(256, 128, (128, 192), (32, 96), 64),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2, padding=1))\n",
+ "\n",
+ "b4 = nn.Sequential(Inception(480, 192, (96, 208), (16, 48), 64),\n",
+ " Inception(512, 160, (112, 224), (24, 64), 64),\n",
+ " Inception(512, 128, (128, 256), (24, 64), 64),\n",
+ " Inception(512, 112, (144, 288), (32, 64), 64),\n",
+ " Inception(528, 256, (160, 320), (32, 128), 128),\n",
+ " nn.MaxPool2d(kernel_size=3, stride=2, padding=1))\n",
+ "\n",
+ "b5 = nn.Sequential(Inception(832, 256, (160, 320), (32, 128), 128),\n",
+ " Inception(832, 384, (192, 384), (48, 128), 128),\n",
+ " nn.AdaptiveAvgPool2d((1,1)),\n",
+ " nn.Flatten())\n",
+ "\n",
+ "net = nn.Sequential(b1, b2, b3, b4, b5, nn.Linear(1024, 10))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "为了使Fashion-MNIST上的训练短小精悍，我们将输入的高和宽从224降到96"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "origin_pos": 26,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Sequential output shape:\t torch.Size([1, 64, 24, 24])\n",
+ "Sequential output shape:\t torch.Size([1, 192, 12, 12])\n",
+ "Sequential output shape:\t torch.Size([1, 480, 6, 6])\n",
+ "Sequential output shape:\t torch.Size([1, 832, 3, 3])\n",
+ "Sequential output shape:\t torch.Size([1, 1024])\n",
+ "Linear output shape:\t torch.Size([1, 10])\n"
+ ]
+ }
+ ],
+ "source": [
+ "X = torch.rand(size=(1, 1, 96, 96))\n",
+ "for layer in net:\n",
+ " X = layer(X)\n",
+ " print(layer.__class__.__name__,'output shape:\\t', X.shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "训练模型"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "origin_pos": 29,
+ "tab": [
+ "pytorch"
+ ]
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "loss 0.246, train acc 0.906, test acc 0.884\n",
+ "3448.9 examples/sec on cuda:0\n"
+ ]
+ },
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "lr, num_epochs, batch_size = 0.1, 10, 128\n",
+ "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=96)\n",
+ "d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())"
+ ]
+ }
+ ],
+ "metadata": {
+ "celltoolbar": "Slideshow",
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.12"
+ },
+ "rise": {
+ "autolaunch": true,
+ "enable_chalkboard": true,
+ "overlay": "