-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.md
239 lines (150 loc) · 12.5 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
# **Behavioral Cloning**
## Udacity Self Driving Car Engineer Nanodegree - Project 4
The goals / steps of this project are the following:
* Use the simulator to collect data of good driving behavior
* Build, a convolution neural network in Keras that predicts steering angles from images
* Train and validate the model with a training and validation set
* Test that the model successfully drives around track one without leaving the road
* Summarize the results with a written report
[//]: # (Image References)
[model_architecture]: ./examples/model-architecture.png "Model Architecture"
[nvidia_model]: ./examples/nVidia_model.png "NVIDIA Model Architecture"
[distribution_1]: ./examples/steering_distribution_before.png "Steering Distribution Before"
[distribution_2]: ./examples/steering_distribution_after.png "Steering Distribution After"
[sample_image]: ./examples/sample_image1.png "Sample Image"
[cropped_image]: ./examples/cropped_image.png "Sample Image"
[random_brightness]: ./examples/random_brightness.png "Random Brightness"
[random_shift]: ./examples/random_shift.png "Random Shift"
[horizontal_flip]: ./examples/horizontal_flip.png "Horizontal Flip"
[train_image_batch]: ./examples/train_image_batch.png "Train Image Batch"
[train_history]: ./examples/training-history.png "Training History"
[video_track_1]: video.mp4 "Video"
## Rubric Points
### Here I will consider the [rubric points](https://review.udacity.com/#!/rubrics/432/view) individually and describe how I addressed each point in my implementation.
---
### Files Submitted & Code Quality
#### 1. Submission includes all required files and can be used to run the simulator in autonomous mode
My project includes the following files:
* [model.py](model.py) containing the script to create and train the model
* [data.py](data.py) containing the script to load and distribute the data
* [config.py](config.py) defines global parameters for the project
* [drive.py](drive.py) for driving the car in autonomous mode
* `model.h5` the trained convolution neural network
* [video.mp4](video.mp4) video of the recordings by drive.py. while the car is driving track-1 in the simulator.
* `README.md` summarizing the results
#### 2. Submission includes functional code
Using the Udacity provided simulator and my drive.py file, the car can be driven autonomously around the track by executing
```sh
python drive.py model.h5
```
#### 3. Submission code is usable and readable
The [model.py](model.py) file contains the code for training and saving the convolution neural network. The file shows the pipeline I used for training and validating the model, and it contains comments to explain how the code works.
The [data.py](data.py) file contains the code to load the simulator data from the `driving_log.csv` file and to compensate the distribution of the data.
The [config.py](config.py) is used to define some project wide parameters, such as batch size, number of epochs, ect.
### Model Architecture and Training Strategy
#### Model Architectur
My model architecture is bases on the NVIDIA model ([End to End Learning for Self-Driving Cars Paper](https://arxiv.org/pdf/1604.07316v1.pdf)),
because it has a fairly simple architecture and because it demonstrably provides good results for use with self-driving cars.
It is a feed-foward layered architecture in which the output of a layer is fed into the overlying layer.
The network consists of a normalization layer, 5 convolutional layers, a dropdown layer, followed by 3 fully connected layers and a final output layer.
Since this is a regression problem, the output layer is a single continuous value that provides the predicted steering angle.
**NVIDIA Model Architecture**
![alt text][nvidia_model]
**Final Model Architecture**
For this project I have made some modifications to the original NVIDIA model. The final model architecture is described here:
First I used two layers for preprocessing. The lambda layer normalizes and mean-centers the image data between +/- 0.5.
The following cropping layer removes the sky and the car front from the image, which have no relevance to driving behavior.
Each of the 5 convolutional layers has a 1x1 stride, and a 2x2 max pooling operation to reduce spatial resolution.
The first 3 convolutional layers use a 5x5 filter while the final 2 use a 3x3 filter as the input dimensionality is reduced.
After the convolutional layers, a 50% dropout layer was added to prevent overfitting.
For non-linearity, [ELU activations](https://arxiv.org/pdf/1511.07289v1.pdf) are used for each convolutional and each fully connected layer.
The original NVIDIA model uses an input size of 66x200x3. I tried that, but with that I did not manage to drive the car a full round in the simulator (currently I do not know exactly why!).
So I use the original image size of 160x320x3 as input size for my network. The cropping layer will reduce the size to 80x320x3 before the image is fed to the convolutional layers.
Here the final model architecture:
![alt text][model_architecture]
The implementation can be found in [model.py](model.py) lines 140-176.
#### Attempts to reduce overfitting in the model
The model was trained and validated on different data sets to ensure that the model was not overfitting ([model.py](model.py) line 39).
And the model was tested by running it through the simulator and ensuring that the vehicle could stay on the track.
I used image normalization, image cropping and a dropout layer for regularization ([model.py](model.py) lines 140-176).
Also I used L2-regularisation on each convolutional and each fully connected layer to combat overfitting ([model.py](model.py) lines 157-173).
Further I used an EarlyStopping-callback to stop the training process automatically if the value-loss has not improved over some epochs.
And a ModelCheckpoint-callback to save only the best trained model ([model.py](model.py) lines 179-188).
#### Model parameter tuning
Here are the parameters for tuning the model and the training process.
The tuning parameters can be adjusted in the [config.py](config.py) file.
* `NUM_EPOCHS` = 20 (Max number of epochs, since early stopping is used)
* `BATCH_SIZE` = 128
* `LERNING_RATE` = 1.0e-4
* `L2_WEIGHT`= 0.001
* `STEERING_CORRECTION` = 0.25
* `STEERING_THRESHOLD` = 0.15
* `NUM_DATA_BINS`= 21
I used an adam optimizer with an learning rate of 1.0e-4 ([model.py](model.py) line 211).
Since this is a regression problem I used the MSE (Mean Squared Error) loss function ([model.py](model.py) line 211).
The `STEERING_CORRECTION` parameter is used to adjust the steering angles for the left (+) or right (-) camera images.
The `STEERING_THRESHOLD` parameter is used to split the driving samples where the car is driving straight ahead or cornering. This is needed for the alignment of the data distribution, explained later.
#### Appropriate training data
I used the [dataset](https://d17h27t6h515a5.cloudfront.net/topher/2016/December/584f6edd_data/data.zip) provided by Udacity to train the model, but this data set contains some pitfalls that make it difficult to train a model that lets the car drive a complete lap in the simulator without a breakdown.
A big problem is the extremely uneven number of records in which the car drives straight ahead (steering angle between -`STEERING_THRESHOLD` and +`STEERING_THRESHOLD`) versus the data sets where the car is cornering.
The following diagram shows the distribution of the data by the steering angle:
![alt text][distribution_1]
Another problem is that for some difficult situations, e.g. the bridge with another road surface, few sharp curves or the curve without roadside, only very few records are available compared to the normal cases. Unfortunately, it is hardly possible to identify these records.
To fix the extremely unequal number of data sets with straight-ahead driving and cornering, one could simply delete a randomly selected part of the data records with straight-ahead driving.
However, there is the risk that the rare special cases will be deleted, which would mean that the car leaves the road in such a special case. So that's not the solution!
To avoid this problem, I group the data sets based on the steering angle in `NUM_DATA_BINS` areas between -1 and +1 using a histogram.
After that, I'll find the group with the most records (that's where the car is driving straight ahead) and fill up each of the other groups with records randomly selected from the same group, up to the number of items of largest group * 0,75. Thus, the distribution of steering angles in the data set is compensated without losing the rare records. However, the disadvantage is that the number of data to be trained has increased massively.
The following diagram shows the distribution of the data by the steering angle, after the compensation has been made:
![alt text][distribution_2]
The data compensation is done in [data.py](data.py) lines 52-72.
#### Loading Data
In the first step, I load the data from the `driving_log.csv` file.
Hereby I decide on the basis of the steering angle which of the 3 camera images (center, left, right) I use.
For this I use the following rules:
* If steering angle is between -`STEERING_THRESHOLD` and +`STEERING_THRESHOLD` (car drives straight ahead), I use only the center image.
* If steering angle is greater than +`STEERING_THRESHOLD` (card drifts left), then I use the center and the left image.
The steering angle for the left image is corrected by adding `STEERING_CORRECTION` to it.
* If steering angle is less than -`STEERING_THRESHOLD` (car drifts right), then I use the center image and the right image.
The steering angle for the right image is corrected by subtracting `STEERING_CORRECTION` from it.
The `STEERING THRESHOLD` was determined by evaluating the steering distribution histogram.
The `STEERING_CORRECTION` value was determined by a lot of trial and error.
This step is done in [data.py](data.py) lines 9-49.
After loading the data it is shuffled and split into a train set and a validation set.
I use only 20% of the records for validation to avoid losing too many special training records.
After the train-test spilt, the train dataset contains 58914 items and the validation dataset contains 14729 items.
This step is done in [model.py](model.py) line 39.
#### Image Preprocessing
In the preprocessing step the images are normalized and mean-centered between -/+0.5.
In addition, image parts that do not contribute to driving behavior, such as the sky and the car front, are cut off.
This reduces the image size from 160x320x3 to 80x320x3.
Both preprocessing steps are done inside the network architecture using a lambda- and a cropping-layer [data.py](data.py) lines 146-149.
|Original Image|Cropped Image (ROI)|
|-------------|-------------|
|![][sample_image]|![][cropped_image]|
#### Image Augmentation
In order to obtain a large variance of training data, I apply various augmentation techniques to the training data:
**Random Brightness**
I apply a random brightness to the images, to simulate various lighting conditions such as sunlight, darkness, shadows, ect.
![alt text][random_brightness]
**Random horizontal and vertical shift**
I randomly move the pictures a little bit horizontally and vertically to simulate camera wobbles.
![alt text][random_shift]
**Horizontal Flip**
I flip every 2nd image horizontally to simulate the mirrored situation.
![alt text][horizontal_flip]
**Train Batch Generator**
I use a generator to randomly create new images from the train dataset for each batch.
The generator combines all the image augmentations described above and shuffles the data again after each epoch.
Here are a few examples of what the train generator delivers:
![alt text][train_image_batch]
This implementation is done in [model.py](model.py) lines 83-107.
#### Output Videos
Here are the result videos that I recorded while the car is driving the predetermined tracks in the simulator autonomously.
The videos were created on my computer with an Nvidia GeForce GTX 1050 GPU. Unfortunately, the videos have periodic short hangs, but this is because my computer does not have enough power.
**Track 1**
|Simulator View|Camera View|
|-------------|-------------|
|[](https://youtu.be/EpAXuGkuFr8)|[](video.mp4)|
**Challenge Track 2**
There are the one or the other dangerous situation here, but if one considers that the model has been trained exclusively with data of the simple track 1, the result is surprisingly good!
[](https://youtu.be/yizCGQiDgPs)