-
Notifications
You must be signed in to change notification settings - Fork 58
/
README.md
457 lines (383 loc) · 106 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
# 上篇
<div class="output_wrapper" id="output_wrapper_id" style="font-size: 16px; color: rgb(62, 62, 62); line-height: 1.6; word-spacing: 0px; letter-spacing: 0px; font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">反向传播算法（Backpropagation Algorithm，简称BP算法）是深度学习的重要思想基础，对于初学者来说也是必须要掌握的基础知识！本文希望以一个清晰的脉络和详细的说明，来让读者彻底明白BP算法的原理和计算过程。</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">全文分为上下两篇，上篇主要介绍BP算法的原理（即公式的推导），介绍完原理之后，我们会将一些具体的数据带入一个简单的三层神经网络中，去完整的体验一遍BP算法的计算过程；下篇是一个项目实战，我们将带着读者一起亲手实现一个BP神经网络（不使用任何第三方的深度学习框架）来解决一个具体的问题。</p>
<h1 id="h1bp" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.6em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1. BP算法的推导</span></h1>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-8fa1e0378e60af5b?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="图1 一个简单的三层神经网络" title="图1 一个简单的三层神经网络" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;">图1 一个简单的三层神经网络</figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">图 1 所示是一个简单的三层（两个隐藏层，一个输出层）神经网络结构，假设我们使用这个神经网络来解决二分类问题，我们给这个网络一个输入样本<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FsetpUHcGTLKUvxXzVdZV5wxuKXX" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，通过前向运算得到输出<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FhwEK6wSnzkiU3wFH_ax7WKP3h6o" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。输出值<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FhwEK6wSnzkiU3wFH_ax7WKP3h6o" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>的值域为<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FufSaWB_0zRV0gb725WoXFXACs63" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，例如<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FhwEK6wSnzkiU3wFH_ax7WKP3h6o" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>的值越接近0，代表该样本是"0"类的可能性越大，反之是"1"类的可能性大。</p>
<h2 id="h11" style="line-height: inherit; margin: 1.5em 0px; font-weight: bold; font-size: 1.4em; margin-bottom: 2em; margin-right: 5px; padding: 8px 15px; letter-spacing: 2px; background-image: linear-gradient(to right bottom, rgb(0, 188, 212), rgb(63, 81, 181)); background-color: rgb(63, 81, 181); color: rgb(255, 255, 255); border-left: 10px solid rgb(51, 51, 51); border-radius: 5px; text-shadow: rgb(102, 102, 102) 1px 1px 1px; box-shadow: rgb(102, 102, 102) 1px 1px 2px;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.1 前向传播的计算</span></h2>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">为了便于理解后续的内容，我们需要先搞清楚前向传播的计算过程，以图1所示的内容为例：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">输入的样本为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fg_MKVdqcjv21Bwse4NxTOG9uIMu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第一层网络的参数为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FiGENpVoTsbf_5B2-nCCXR9TvfcE" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第二层网络的参数为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FutDepG1pRHH2BhXzta-dUt_pbRc" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第三层网络的参数为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fl6FpC8CFlcqrFP5aIE-QpIJDHOd" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<h3 id="h111" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.3em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.1.1 第一层隐藏层的计算</span></h3>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-e076570b8b21eb63?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="图2 计算第一层隐藏层" title="图2 计算第一层隐藏层" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;">图2 计算第一层隐藏层</figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第一层隐藏层有三个神经元：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fg0dczGIZ82S_Q5NjHz-TFHDfQ3J" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>、<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fo1NGpv9yVCxuNC3hDh7dPOlUUvX" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FlMJqiC7cX7KsmFdsnmsTivZ6h87" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。该层的输入为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fo8pwDl-PuO9TsqeJdyZhIULXVOn" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">以<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fg0dczGIZ82S_Q5NjHz-TFHDfQ3J" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>神经元为例，则其输入为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Frg-g5h2nmOvfT7NkmL_ZiNhURXQ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">同理有：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fq3JNN-4ThHWFZ9NaP43qSW-r2sd" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">假设我们选择函数<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FrDos9anAPoqvPoO6q8fBJv5GOpr" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>作为该层的激活函数（图1中的激活函数都标了一个下标，一般情况下，同一层的激活函数都是一样的，不同层可以选择不同的激活函数），那么该层的输出为：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FvgQSnWlx7dQnLRGTJgBchiOL_8e" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>、<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FggWOCBSpCYRR_tOxbZr32MSgSn-" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fl6U_KibqpBNJXLfjFVlRFbhZpjX" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。</p>
<h3 id="h112" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.3em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.1.2 第二层隐藏层的计算</span></h3>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-684285514d2a07da?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="图3 计算第二层隐藏层" title="图3 计算第二层隐藏层" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;">图3 计算第二层隐藏层</figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第二层隐藏层有两个神经元：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FkyboGq0mr2dQKu1k1HdT3Cf3RxS" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FoeF8HgU86WQpaahpMnVrb2dCaZl" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。该层的输入为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fj7PMyiadecg7uV5uMZQmLqnYuD2" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">即第二层的输入是第一层的输出乘以第二层的权重，再加上第二层的偏置。因此得到和的输入分别为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fq0942vpsPn4xma8KBeN9wuPCekK" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">该层的输出分别为：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Furt8hV6Bdm1rKTA7xZhO5uGvKM_" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FmL8uh0fWknEsLnRA34dMLF1F9zy" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。</p>
<h3 id="h113" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.3em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.1.3 输出层的计算</span></h3>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-ebb295e1f681e589?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="图4 计算输出层" title="图4 计算输出层" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;">图4 计算输出层</figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">输出层只有一个神经元<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FvSCSWvZGS9DIkUS1WXgNdR-ynWR" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>：。该层的输入为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Ftw19VtaHQfoj4_qKh4TPPVORlpQ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">即：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FtZU5ZvIeQDX3aUYih5gulaXRIKG" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">因为该网络要解决的是一个二分类问题，所以输出层的激活函数也可以使用一个Sigmoid型函数，神经网络最后的输出为：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FglTu-uAV2Vmh_kWMClX3ePvQmim" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。</p>
<h2 id="h12" style="line-height: inherit; margin: 1.5em 0px; font-weight: bold; font-size: 1.4em; margin-bottom: 2em; margin-right: 5px; padding: 8px 15px; letter-spacing: 2px; background-image: linear-gradient(to right bottom, rgb(0, 188, 212), rgb(63, 81, 181)); background-color: rgb(63, 81, 181); color: rgb(255, 255, 255); border-left: 10px solid rgb(51, 51, 51); border-radius: 5px; text-shadow: rgb(102, 102, 102) 1px 1px 1px; box-shadow: rgb(102, 102, 102) 1px 1px 2px;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.2 反向传播的计算</span></h2>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">在1.1节里，我们已经了解了数据沿着神经网络前向传播的过程，这一节我们来介绍更重要的反向传播的计算过程。假设我们使用随机梯度下降的方式来学习神经网络的参数，损失函数定义为<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FhQBpef_i94sgyz-PeheBeQJBu4o" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，其中 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FplnQuLsAVax4eN9U5X6P_PDw6zs" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span> 是该样本的真实类标。使用梯度下降进行参数的学习，我们必须计算出损失函数关于神经网络中各层参数（权重<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fj0h1Gz9OtsAEsZHETYqON1qkNnO" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和偏置<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fqw689D_cBdhkVnC9Fb1_w2HQvrR" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>）的偏导数。</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-07f069731997296c?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">假设我们要对第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层隐藏层的参数<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Flvvg3VQN_ChaR2uYJmZZ2Ek1mw2" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FrPHA4KRV9EUTgg8HcQmW_xaxMC4" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>求偏导数，即求 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FkQk3z3xdXOFfKxvzoNGHEBuCZJc" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FvAx7UKZci46WEJNI6AsDkz5O4kd" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。假设<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fm5AeGg9ErNDA7wAGe-p2RuyJLFX" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>代表第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层神经元的输入，即<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FvrvTZJUMJCzPZIPIbvHILM923IV" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，其中<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FgG6bteG5AnlCjFt7gIwr9l_HnZC" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span> 为前一层神经元的输出，则根据链式法则有：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fp4BFEFJbm-ohtL9bRr-mS4Pex65" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FraZTxmSKX2f8tEa9XXrGccxBZHk" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">因此，我们只需要计算偏导数<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FoRz10KpbC3LrIMlkJKBEc9iktgp" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>、<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FjlKKYzeK5llQx4rJOxouw6LycnW" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span> 和 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FufqZm1jsu-DVT40GkdiLBECHxkL" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。</p>
<h3 id="h121dfracpartialzkpartialwkd" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.3em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.2.1 计算偏导数$\frac{\partial z^{(k)}}{\partial W^{(k)}}$</span></h3>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">前面说过，第k层神经元的输入为：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FmVHsNxaY5ObHDyLUMx3q3sJ5ilH" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，因此可以得到：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-d82ff15d292c8671?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">上式中，<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FvN7fnip8kpzQt0vvnMRGDwAnuqq" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>代表第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层神经元的权重矩阵<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Flvvg3VQN_ChaR2uYJmZZ2Ek1mw2" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>的第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Foo9HKyH5iTzKIV-z4N2fX7gR4Yd" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>行，<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FkwhqhrjXsP7XDx-hMtIlJrz1Erw" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>代表第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层神经元的权重矩阵<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Flvvg3VQN_ChaR2uYJmZZ2Ek1mw2" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>的第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Foo9HKyH5iTzKIV-z4N2fX7gR4Yd" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>行中的第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FpD_iJnmyIkOiZuy2b7Ju6u-wOdM" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>列。</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">我们以1.1节中的简单神经网络为例，假设我们要计算第一层隐藏层的神经元关于权重矩阵的导数，则有：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fpr1Krw4jrxjmHVWCcrY21ZEog8e" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<h3 id="h122dfracpartialzkpartialbkd" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.3em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.2.2 计算偏导数$\frac{\partial z^{(k)}}{\partial b^{(k)}}$</span></h3>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">因为偏置b是一个常数项，因此偏导数的计算也很简单：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FhF4Ljwt8w2hV7MWYeR5n5U5HQt1" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">依然以第一层隐藏层的神经元为例，则有：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FowrTLRldcqd9taCI0gMJbgXZ1Yk" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<h3 id="h123dfracpartialmathrmlmathrmywidehatmathrmypartialzkd" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.3em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1.2.3 计算偏导数$\frac{\partial \mathrm{L}(\mathrm{y}, \widehat{\mathrm{y}})}{\partial z^{(k)}}$</span></h3>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">偏导数<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FoRz10KpbC3LrIMlkJKBEc9iktgp" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>又称为<strong style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold;">误差项（error term，也称为“灵敏度”）</strong>，一般用<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fnetlb6OsIb29vfzYw_mDnJxDfYA" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>表示，例如<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fh8gEBk0n1oIEiRHAuboVgNv31Gf" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>是第一层神经元的误差项，其值的大小代表了第一层神经元对于最终总误差的影响大小。</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">根据第一节的前向计算，我们知道第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FnjOPxASA9mmRAn9XkpkbU9Nu303" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层的输入与第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层的输出之间的关系为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FrwsCHd92SicMAPPSCIztyu4o4Mt" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">又因为<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FhBIKJjRqxOZsUqjzH0r13wbaTGD" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，根据链式法则，我们可以得到<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FrETnZIW5Llf2v6lComLc67chQ6I" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FmCrMSTFzhdRgs1ybh608HZFBYcZ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">由上式我们可以看到，第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层神经元的误差项<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fo03VYghzsxB9ZGn04yIEcqrOyt-" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>是由第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FnjOPxASA9mmRAn9XkpkbU9Nu303" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层的误差项乘以第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FnjOPxASA9mmRAn9XkpkbU9Nu303" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层的权重，再乘以第<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fte9RRX682XDTM734zfxdAdmSBwu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>层激活函数的导数（梯度）得到的。这就是误差的反向传播。<br> 现在我们已经计算出了偏导数 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fj2ED3RQsWNt1vdK0sO1vb5_3Xof" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>、<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FjlKKYzeK5llQx4rJOxouw6LycnW" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FufqZm1jsu-DVT40GkdiLBECHxkL" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，则<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FpXPg3JQueWbeiwtGolgyg-_zquF" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和 <span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FmttPfPRJSGRsUhmCjo70huzbV2f" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>可分别表示为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FnU4SVZJLVRI7j9ltddXuj3q0irI" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FiKMfbtt9S-NO-RpKHiZ1iZXyXWs" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">下面是基于随机梯度下降更新参数的反向传播算法：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-19c09dbf953a646d?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><strong style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold;">单纯的公式推导看起来有些枯燥，下面我们将实际的数据带入图1所示的神经网络中，完整的计算一遍。</strong></p>
<h1 id="h2bp" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.6em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">2. 图解BP算法</span></h1>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-553dca76ddfce469?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="图5 图解BP算法" title="图5 图解BP算法" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;">图5 图解BP算法</figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">我们依然使用如图5所示的简单的神经网络，其中所有参数的初始值如下：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">输入的样本为（假设其真实类标为"1"）：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FkVMpIW3QEBpwt54Uu7lfKJZic5i" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第一层网络的参数为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FluNjsl2oiIdDcUk2LzEfrbjD09J" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第二层网络的参数为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fm3COkkcPTZ8uBa1qlnizEXiBPXF" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">第三层网络的参数为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fnf3V_nTjCRtVtUa62Pd3RGipxsD" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">假设所有的激活函数均为Logistic函数：<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fi0ezdCgkeduKh3C45GinvJp3nqo" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>。使用均方误差函数作为损失函数：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fvc6B0oAHv1MbfscgmvfT1u4RrhS" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">为了方便求导，我们将损失函数简化为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fvq1MSyOkg_6EyaEACu8SidFWNZn" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<h2 id="h21" style="line-height: inherit; margin: 1.5em 0px; font-weight: bold; font-size: 1.4em; margin-bottom: 2em; margin-right: 5px; padding: 8px 15px; letter-spacing: 2px; background-image: linear-gradient(to right bottom, rgb(0, 188, 212), rgb(63, 81, 181)); background-color: rgb(63, 81, 181); color: rgb(255, 255, 255); border-left: 10px solid rgb(51, 51, 51); border-radius: 5px; text-shadow: rgb(102, 102, 102) 1px 1px 1px; box-shadow: rgb(102, 102, 102) 1px 1px 2px;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">2.1 前向传播</span></h2>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">我们首先初始化神经网络的参数，计算第一层神经元：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-f8fc7ad345ad6e5c?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FhICIZ2gI0hC-L8MnKSZdFLVPJhe" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FpbtgawXij1Iwvn8HodHGd4hBqtY" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">上图中我们计算出了第一层隐藏层的第一个神经元的输入<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fig58WjVQCCSf8pZ5KqJixrkn-3l" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和输出<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FvgQSnWlx7dQnLRGTJgBchiOL_8e" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，同理可以计算第二个和第三个神经元的输入和输出：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-b29719e65a7e36fe?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FuMJBBldJfH9LA-S1aHVl5TVJF5w" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fq_HI8d6dnWfuUKpAnNhXI3rSBNA" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fhui90PMCBLVTWFt_kAfcwnhqyxK" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fvq6ES1jdDvYYkbM_qQ12RVPYypA" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">接下来是第二层隐藏层的计算，首先我们计算第二层的第一个神经元的输入z₄和输出f₄(z₄)：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-2768830f95d1809d?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FgXh-QlCLeMCedqZpR3ACXFdalj5" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FnkKOqj1jyIbLAv2BwYnEZDud92w" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">同样方法可以计算该层的第二个神经元的输入<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FiyAw1j-sKCneRxWj0n6ZH7nFRhK" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和输出<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FmL8uh0fWknEsLnRA34dMLF1F9zy" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-2f0f27425a866d00?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FvKhpq6XFDMKfE49eg3MQvtFYBmu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FidshioQE47dHJIp_sZ9racvuSXX" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">最后计算输出层的输入<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FsNlPOe3EKiv_CfTnBE5CuT7lBdv" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>和输出<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/Fmyj1_xKKxLwrqBri-Ssviz2m0jn" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-8b0e94d6b5a02ef6?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FvxB_fGJvQDGrrHgQCGtJxu6MMqR" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FqMfnGSXs3OIcnnrbtxq1hf_jL37" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<h2 id="h22" style="line-height: inherit; margin: 1.5em 0px; font-weight: bold; font-size: 1.4em; margin-bottom: 2em; margin-right: 5px; padding: 8px 15px; letter-spacing: 2px; background-image: linear-gradient(to right bottom, rgb(0, 188, 212), rgb(63, 81, 181)); background-color: rgb(63, 81, 181); color: rgb(255, 255, 255); border-left: 10px solid rgb(51, 51, 51); border-radius: 5px; text-shadow: rgb(102, 102, 102) 1px 1px 1px; box-shadow: rgb(102, 102, 102) 1px 1px 2px;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">2.2 误差反向传播</span></h2>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-106caa21dad2c0c1?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">首先计算输出层的误差项<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FoRIL0or9lryUUQCMKYQQLUi-lGV" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，我们的误差函数为<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FpZSz36KnWzuBcBt6Lfp4v2CShJL" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，由于该样本的类标为“1”，而预测值为<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FrjR7X0XvFcFd9g3_YYUTubEfIVQ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，因此误差为<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FkYVhgYsfIwevFGK7-Wg8aZkqLFT" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>，输出层的误差项为：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Frx4UK_Zk8XUECdWB_SsYm8iU4vQ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">接着计算第二层隐藏层的误差项，根据误差项的计算公式有：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FuExXfpNETqOABoGq9tN9DoTUnFJ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span><br><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FoDuakZ3tMbB-Sg5zUt2x4WV28Re" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span><br><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fl32wyKDPIg5XKLpg600ZVyiqrTO" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span><br><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FkRdp7tuRUqjpkPUEHOuzM8lLHeu" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">最后是计算第一层隐藏层的误差项：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fji7qK5y6DIRWBIoUG__JhhmCuUj" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span><br><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FplSPUstX85a-sGSa-L4QyP-NIq5" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span><br><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fkb70fUeJ6lrZxkUWLn9wPyW6Cu6" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span><br><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/FrIJMuv66b-ffyIAStDb_XlkZ7qO" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<h2 id="h23" style="line-height: inherit; margin: 1.5em 0px; font-weight: bold; font-size: 1.4em; margin-bottom: 2em; margin-right: 5px; padding: 8px 15px; letter-spacing: 2px; background-image: linear-gradient(to right bottom, rgb(0, 188, 212), rgb(63, 81, 181)); background-color: rgb(63, 81, 181); color: rgb(255, 255, 255); border-left: 10px solid rgb(51, 51, 51); border-radius: 5px; text-shadow: rgb(102, 102, 102) 1px 1px 1px; box-shadow: rgb(102, 102, 102) 1px 1px 2px;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">2.3 更新参数</span></h2>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">上一小节中我们已经计算出了每一层的误差项，现在我们要利用每一层的误差项和梯度来更新每一层的参数，权重W和偏置b的更新公式如下：</p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;"><span class="katex-display" style="display: block; text-align: center; color: inherit; line-height: inherit; margin: 0px; padding: 0px; font-size: 1.22em;"><span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; white-space: nowrap; display: inline-block; text-align: center; padding: 3px;"><img src="http://qiniu.aihubs.net/Fs5enMZwYZUYpNRixQUkRMpHhTsC" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span></span></p>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">通常权重<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FmguTcePO7AgKoD9kqfCpmeEb7mQ" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>的更新会加上一个正则化项来避免过拟合，这里为了简化计算，我们省去了正则化项。上式中<span class="katex" style="font: 1.21em/1.2 KaTeX_Main, 'Times New Roman', serif; text-indent: 0px; text-rendering: auto; font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 8px 3px;"><img src="http://qiniu.aihubs.net/FjPgmKTIvtw1KBnbavv1PR3qee4k" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 0px auto; max-width: 100%; display: inline-block; vertical-align: middle;"></span>的是学习率，我们设其值为0.1。参数更新的计算相对简单，每一层的计算方式都相同，因此本文仅演示第一层隐藏层的参数更新：</p>
<figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://upload-images.jianshu.io/upload_images/13056713-1845ea8fb509679c?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure>
<h1 id="h3" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; font-size: 1.6em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3. 小结</span></h1>
<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px;">至此，我们已经完整介绍了BP算法的原理，并使用具体的数值做了计算。在下篇中，我们将带着读者一起亲手实现一个BP神经网络（不使用任何第三方的深度学习框架）。</p></div>
# 下篇
在上篇中我们详细介绍了BP算法的原理和推导过程，并且用实际的数据进行了计算演练。在下篇中，我们将自己实现BP算法（不使用第三方的算法框架），并用来解决鸢尾花分类问题。
鸢尾花数据集如图2所示，总共有三个品种的鸢尾花（setosa、versicolor和virginica），每个类别50条样本数据，每个样本有四个特征（花萼长度、花萼宽度、花瓣长度以及花瓣宽度）。
首先我们导入需要的包：
``` python
from csv import reader
import numpy as np
from sklearn.preprocessing import MinMaxScaler
import random
import matplotlib.pyplot as plt
import math
```
接下来我们实现一个数据集的加载和预处理的函数`load_dataset`：
``` python
def load_dataset(dataset_path, n_train_data):
"""加载数据集，对数据进行预处理，并划分训练集和验证集
:param dataset_path: 数据集文件路径
:param n_train_data: 训练集的数据量
:return: 划分好的训练集和验证集
"""
dataset = []
label_dict = {'Iris-setosa': 0, 'Iris-versicolor': 1, 'Iris-virginica': 2}
with open(dataset_path, 'r') as file:
# 读取CSV文件，以逗号为分隔符
csv_reader = reader(file, delimiter=',')
for row in csv_reader:
# 将字符串类型的特征值转换为浮点型
row[0:4] = list(map(float, row[0:4]))
# 将标签替换为整型
row[4] = label_dict[row[4]]
# 将处理好的数据加入数据集中
dataset.append(row)
# 对数据进行归一化处理
dataset = np.array(dataset)
mms = MinMaxScaler()
for i in range(dataset.shape[1] - 1):
dataset[:, i] = mms.fit_transform(dataset[:, i].reshape(-1, 1)).flatten()
# 将类标转为整型
dataset = dataset.tolist()
for row in dataset:
row[4] = int(row[4])
# 打乱数据集
random.shuffle(dataset)
# 划分训练集和验证集
train_data = dataset[0:n_train_data]
val_data = dataset[n_train_data:]
return train_data, val_data
```
在`load_dataset`函数中，我们实现了数据集的读取、数据的归一化处理以及对数据集进行了`shuffle`操作等，最后函数返回了划分好的训练集和验证集。
实现数据预处理之后，接下来我们开始实现BP算法的关键部分（**如果读者对算法原理有不清楚的地方，可以查看"一文彻底搞懂BP算法：原理推导+数据演示+项目实战（上篇）"**）。首先我们实现神经元的计算部分、激活函数以及激活函数的求导部分。
``` python
def fun_z(weights, inputs):
"""计算神经元的输入：z = weight * inputs + b
:param weights: 网络参数（权重矩阵和偏置项）
:param inputs: 上一层神经元的输出
:return: 当前层神经元的输入
"""
bias_term = weights[-1]
z = 0
for i in range(len(weights)-1):
z += weights[i] * inputs[i]
z += bias_term
return z
def sigmoid(z):
"""激活函数(Sigmoid)：f(z) = Sigmoid(z)
:param z: 神经元的输入
:return: 神经元的输出
"""
return 1.0 / (1.0 + math.exp(-z))
def sigmoid_derivative(output):
"""Sigmoid激活函数求导
:param output: 激活函数的输出值
:return: 求导计算结果
"""
return output * (1.0 - output)
```
函数`fun_z`实现了公式"z = weight * inputs + b"，其中inputs是上一层网络的输出，weight是当前层的权重矩阵，b是当前层的偏置项，计算得到的z是当前层的输入。
函数`sigmoid`是Sigmoid激活函数的实现，将z作为激活函数的输入，计算得到当前层的输出，并传递到下一层。
函数`sigmoid_derivative`是Sigmoid函数求导的实现，在误差反向传播的时候需要用到。
接下来我们实现BP网络的前向传播：
``` python
def forward_propagate(network, inputs):
"""前向传播计算
:param network: 神经网络
:param inputs: 一个样本数据
:return: 前向传播计算的结果
"""
for layer in network: # 循环计算每一层
new_inputs = []
for neuron in layer: # 循环计算每一层的每一个神经元
z = fun_z(neuron['weights'], inputs)
neuron['output'] = sigmoid(z)
new_inputs.append(neuron['output'])
inputs = new_inputs
return inputs
```
前向计算的过程比较简单，和我们在上篇中介绍的计算过程一致。稍微麻烦一点的是误差反向传播的计算：
``` python
def backward_propagate_error(network, actual_label):
"""误差进行反向传播
:param network: 神经网络
:param actual_label: 真实的标签值
:return:
"""
for i in reversed(range(len(network))): # 从最后一层开始计算误差
layer = network[i]
errors = list()
if i != len(network)-1: # 不是输出层
for j in range(len(layer)): # 计算每一个神经元的误差
error = 0.0
for neuron in network[i + 1]:
error += (neuron['weights'][j] * neuron['delta'])
errors.append(error)
else: # 输出层
for j in range(len(layer)): # 计算每一个神经元的误差
neuron = layer[j]
errors.append(actual_label[j] - neuron['output'])
# 计算误差项 delta
for j in range(len(layer)):
neuron = layer[j]
neuron['delta'] = errors[j] * sigmoid_derivative(neuron['output'])
```
误差反向传播过程中，我们首先需要根据模型的输出来计算得到误差，然后计算输出层的误差项。得到输出层的误差项之后，我们就可以根据上篇中介绍的"第k层神经元的误差项是由第k+1层的误差项乘以第k+1层的权重，再乘以第k层激活函数的导数得到"来计算其它层的误差项。
在计算得到每一层的误差项之后，我们根据上篇中介绍的权重矩阵和偏置项的更新公式来更新参数：
``` python
def update_parameters(network, row, l_rate):
"""利用误差更新神经网络的参数（权重矩阵和偏置项）
:param network: 神经网络
:param row: 一个样本数据
:param l_rate: 学习率
:return:
"""
for i in range(len(network)):
inputs = row[:-1]
if i != 0: # 获取上一层网络的输出
inputs = [neuron['output'] for neuron in network[i - 1]]
for neuron in network[i]:
# 更新权重矩阵
for j in range(len(inputs)):
neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]
# 更新偏置项
neuron['weights'][-1] += l_rate * neuron['delta']
```
到这里所有的关键部分我们都已经实现了，接下来我们实现网络的初始化以及网络的训练部分，首先实现网络的初始化：
``` python
def initialize_network(n_inputs, n_hidden, n_outputs):
"""初始化BP网络（初始化隐藏层和输出层的参数：权重矩阵和偏置项）
:param n_inputs: 特征列数
:param n_hidden: 隐藏层神经元个数
:param n_outputs: 输出层神经元个数，即分类的总类别数
:return: 初始化后的神经网络
"""
network = list()
# 隐藏层
hidden_layer = [{'weights': [random.random() for i in range(n_inputs + 1)]} for i in range(n_hidden)]
network.append(hidden_layer)
# 输出层
output_layer = [{'weights': [random.random() for i in range(n_hidden + 1)]} for i in range(n_outputs)]
network.append(output_layer)
return network
```
这里我们初始化了一个两层神经网络（一个隐藏层和一个输出层）。在初始化参数的时候，我们将权重矩阵和偏置项放在了一个数组中（"weights"），数组的最后一个元素是偏置项，前面的元素是权重矩阵。
接下来我们实现模型的训练部分：
``` python
def train(train_data, l_rate, epochs, n_hidden, val_data):
"""训练神经网络（迭代n_epoch个回合）
:param train_data: 训练集
:param l_rate: 学习率
:param epochs: 迭代的回合数
:param n_hidden: 隐藏层神经元个数
:param val_data: 验证集
:return: 训练好的网络
"""
# 获取特征列数
n_inputs = len(train_data[0]) - 1
# 获取分类的总类别数
n_outputs = len(set([row[-1] for row in train_data]))
# 初始化网络
network = initialize_network(n_inputs, n_hidden, n_outputs)
acc = []
for epoch in range(epochs): # 训练epochs个回合
for row in train_data:
# 前馈计算
_ = forward_propagate(network, row)
# 处理一下类标，用于计算误差
actual_label = [0 for i in range(n_outputs)]
actual_label[row[-1]] = 1
# 误差反向传播计算
backward_propagate_error(network, actual_label)
# 更新参数
update_parameters(network, row, l_rate)
# 保存当前epoch模型在验证集上的准确率
acc.append(validation(network, val_data))
# 绘制出训练过程中模型在验证集上的准确率变化
plt.xlabel('epochs')
plt.ylabel('accuracy')
plt.plot(acc)
plt.show()
return network
```
我们总共训练了`epochs`个回合，这里我们使用随机梯度下降来优化模型，因此每次都用一个样本来更新参数。接下来我们实现一个函数用来验证模型的效果：
``` python
def validation(network, val_data):
"""测试模型在验证集上的效果
:param network: 神经网络
:param val_data: 验证集
:return: 模型在验证集上的准确率
"""
# 获取预测类标
predicted_label = []
for row in val_data:
prediction = predict(network, row)
predicted_label.append(prediction)
# 获取真实类标
actual_label = [row[-1] for row in val_data]
# 计算准确率
accuracy = accuracy_calculation(actual_label, predicted_label)
# print("测试集实际类标：", actual_label)
# print("测试集上的预测类标：", predicted_label)
return accuracy
```
训练过程中的每一个回合，我们都用模型对验证集进行一次预测，并将预测的结果保存，用来绘制训练过程中模型在验证集上的准确率的变化过程。准确率的计算以及使用模型进行预测的实现如下：
``` python
def accuracy_calculation(actual_label, predicted_label):
"""计算准确率
:param actual_label: 真实类标
:param predicted_label: 模型预测的类标
:return: 准确率（百分制）
"""
correct_count = 0
for i in range(len(actual_label)):
if actual_label[i] == predicted_label[i]:
correct_count += 1
return correct_count / float(len(actual_label)) * 100.0
def predict(network, row):
"""使用模型对当前输入的数据进行预测
:param network: 神经网络
:param row: 一个数据样本
:return: 预测结果
"""
outputs = forward_propagate(network, row)
return outputs.index(max(outputs))
```
最后我们运行代码：
``` python
if __name__ == "__main__":
file_path = './iris.csv'
# 参数设置
l_rate = 0.2 # 学习率
epochs = 300 # 迭代训练的次数
n_hidden = 5 # 隐藏层神经元个数
n_train_data = 130 # 训练集的大小（总共150条数据，训练集130条，验证集20条）
# 加载数据并划分训练集和验证集
train_data, val_data = load_dataset(file_path, n_train_data)
# 训练模型
network = train(train_data, l_rate, epochs, n_hidden, val_data)
```
训练过程如图3所示：
