-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path10regularization.html
145 lines (122 loc) · 8.98 KB
/
10regularization.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="Deep Learning Tutorial using Keras">
<meta name="author" content="Lindsey M Kitchell">
<title>Intro to Deep Learning</title>
<!-- Bootstrap core CSS -->
<link href="vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="css/simple-sidebar.css" rel="stylesheet">
<!-- fonts -->
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,400,600,700&display=swap" rel="stylesheet">
</head>
<body>
<div class="d-flex" id="wrapper">
<!-- Sidebar -->
<div class="bg-light border-right" id="sidebar-wrapper">
<div class="sidebar-heading">Deep Learning With Keras</div>
<div class="list-group list-group-flush">
<a href="1introtodeeplearning.html" class="list-group-item list-group-item-action bg-light">1. Intro to Deep Learning</a>
<a href="2introtokeras.html" class="list-group-item list-group-item-action bg-light">2. Intro to Keras</a>
<a href="3mlpsinkeras.html" class="list-group-item list-group-item-action bg-light">3. MLPs in Keras</a>
<a href="4cnnsinkeras.html" class="list-group-item list-group-item-action bg-light">4. CNNs in Keras</a>
<a href="5activationfunctions.html" class="list-group-item list-group-item-action bg-light">5. Activation Functions</a>
<a href="6otherkerasfunctions.html" class="list-group-item list-group-item-action bg-light">6. Other Useful Keras Functions</a>
<a href="7lossfunctionsoptimizers.html" class="list-group-item list-group-item-action bg-light">7. Loss Functions and Optimizers</a>
<a href="8evaluatingnns.html" class="list-group-item list-group-item-action bg-light">8. Evaluating Neural Networks</a>
<a href="9datapreprocessing.html" class="list-group-item list-group-item-action bg-light">9. Data Preprocessing</a>
<a href="10regularization.html" class="list-group-item list-group-item-action bg-light">10. Regularization</a>
<a href="11hyperparametertuning.html" class="list-group-item list-group-item-action bg-light">11. Hyperparameter Tuning</a>
</div>
</div>
<!-- /#sidebar-wrapper -->
<!-- Page Content -->
<div id="page-content-wrapper">
<nav class="navbar navbar-expand-lg navbar-light bg-light border-bottom">
<button class="btn btn-primary" id="menu-toggle">Toggle Menu</button>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<ul class="navbar-nav ml-auto mt-2 mt-lg-0">
<li class="nav-item active">
<a class="nav-link" href="index.html">Home <span class="sr-only">(current)</span></a>
</li>
<li class="nav-item">
<a class="nav-link" target="_blank" href="https://lindseykitchell.weebly.com/">About the Author</a>
</li>
<!--
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Dropdown
</a>
<div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdown">
<a class="dropdown-item" href="#">Action</a>
<a class="dropdown-item" href="#">Another action</a>
<div class="dropdown-divider"></div>
<a class="dropdown-item" href="#">Something else here</a>
</div>
</li>
-->
</ul>
</div>
</nav>
<div class="container-fluid">
<h1 id="regularization">Regularization</h1>
<hr>
<p>We have already learning that overfitting is when the model becomes too specific and too well trained to the training data. One way this is avoided is by validating and testing the network on new data the model has not see before. There are a few additional ways we can prevent overfitting. One way is to modulate the amount of information the model model is allowed to store. If the model can only learned a small number of patterns, the optimization process will force it to focus on the most important patterns. This process is called <strong>regularization</strong>. </p>
<p>The most common methods of regularization include:</p>
<h3 id="reducing-the-network-size">Reducing the network size</h3>
<p>Reducing the number of layers and number of units/nodes per layer is a simple way to regularize your network. </p>
<h3 id="weight-regularization">Weight regularization</h3>
<p>When training a network it is possible that multiple sets of weight values could explain the data. <em>Weight regularization</em> is based off of the idea that the 'simplest' model is likely the best one or the one least likely to overfit. A 'simple model' is a model where the distribution of parameter values has less entropy. A way to prevent overfitting is to put constraints on the complexity of a network and forcing its weights to only take small values, making the distribution more <em>regular</em>. The constraints are added by adding a cost for large weights to the loss function of the network. There are two cost options:</p>
<ul>
<li>L1 - the cost is proportional to the absolute value of the weight coefficients (also called the L1 norm of the weights</li>
<li>L2 - the cost is proportional to the square of the value of the weight coefficients (also called the L2 norem of the weights and weight decay)</li>
</ul>
<p>In Keras, weight regularization is added by passing weight regularizers instances to the <code>kernal_regularizer</code> argument when adding a layer to the model. <a href="https://keras.io/regularizers/">Keras documentation</a>.</p>
<p><strong>L1</strong></p>
<pre><code class="lang-python">keras<span class="hljs-selector-class">.regularizers</span><span class="hljs-selector-class">.l1</span>(<span class="hljs-number">0</span>.)
</code></pre>
<p><strong>L2</strong></p>
<pre><code class="lang-python">keras<span class="hljs-selector-class">.regularizers</span><span class="hljs-selector-class">.l2</span>(<span class="hljs-number">0</span>.)
</code></pre>
<p><strong>L1 and L2</strong></p>
<pre><code class="lang-python">keras<span class="hljs-selector-class">.regularizers</span><span class="hljs-selector-class">.l1_l2</span>(<span class="hljs-number">0</span>.)
</code></pre>
<p>The number in the parenthesis is what is multiplied to the weight coefficient value and added to the total loss of the network.</p>
<pre><code class="lang-python">from keras import regularizers
<span class="hljs-keyword">model</span> = <span class="hljs-keyword">models</span>.Sequential()
<span class="hljs-keyword">model</span>.add(layers.Dense(<span class="hljs-number">16</span>, kernel_regularizer=regularizers.l2(<span class="hljs-number">0.001</span>),
activation=<span class="hljs-string">'relu'</span>, input_shape=(<span class="hljs-number">10000</span>,)))
<span class="hljs-keyword">model</span>.add(layers.Dense(<span class="hljs-number">16</span>, kernel_regularizer=regularizers.l2(<span class="hljs-number">0.001</span>),
activation=<span class="hljs-string">'relu'</span>))
<span class="hljs-keyword">model</span>.add(layers.Dense(<span class="hljs-number">1</span>, activation=<span class="hljs-string">'sigmoid'</span>))
</code></pre>
<h3 id="dropout">Dropout</h3>
<p>Dropout has already been discussed but I will mention it again here. Dropout consists of 'dropping' a set percentage of training samples each training round. This is done by adding a dropout layer to the model</p>
<pre><code class="lang-python"><span class="hljs-selector-tag">model</span><span class="hljs-selector-class">.add</span>(<span class="hljs-selector-tag">layers</span><span class="hljs-selector-class">.Dropout</span>(0<span class="hljs-selector-class">.5</span>))
</code></pre>
<h3 id="useful-resources">Useful resources</h3>
<p><a href="http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/4.4-overfitting-and-underfitting.ipynb">http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/4.4-overfitting-and-underfitting.ipynb</a></p>
<p><strong>Please move on to <a href="11hyperparametertuning.html">Hyperparameter Tuning</a>.</strong></p>
</div>
</div>
<!-- /#page-content-wrapper -->
</div>
<!-- /#wrapper -->
<!-- Bootstrap core JavaScript -->
<script src="vendor/jquery/jquery.min.js"></script>
<script src="vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<!-- Menu Toggle Script -->
<script>
$("#menu-toggle").click(function(e) {
e.preventDefault();
$("#wrapper").toggleClass("toggled");
});
</script>
</body>
</html>