Deep Learning Nanodegree Foundation Notes & Exercises

Notes, coding exercises and projects from Udacity's Deep Learning Nanodegree Foundation program.

Projects

PROJECT 1: Bike Sharing Neural Network (Repo)

PROJECT 2: Image Classification Convolutional Neural Network (Repo)

PROJECT 3: Simpsons Script Generation with Recurrent Neural Networks (Repo)

PROJECT 4: English to French Translation with Sequence to Sequence RNN (Repo)

PROJECT 5: Face Generation DCGAN (Repo)

Lessons

1 - Intro to Neural Networks

• PROJECT 1: Bike Sharing Neural Network (Repo)

2 - Sentiment Analysis

Sentiment Analysis with Andrew Trask

• Framing Problems for Neural Networks (Notebook)
• Creating the Input/Output Data (Notebook)
• Building the Neural Network (Notebook)
• Making Learning Faster by Reducing Noise (Notebook)
• Making our Network Train and Run Faster (Notebook)
• Reducing Noise by Strategically Reducing Noise (Notebook)

Intro to TFLearn

• Sentiment Analysis with TFLearn (Notebook)
• Handwritten Digit Recognition with TFLearn and MNIST (Notebook)

3 - MiniFlow

• Forward Propagation (Notebook)
• Learning and Loss (Notebook)
• Linear Transform (Notebook)
• Sigmoid Function (Notebook)
• Cost (Notebook)
• Gradient Descent (Notebook)

4 - Intro to TensorFlow

• Intro to TensorFlow Notes (Markdown) (Notebook)
• TensorFlow Neural Network Lab (Notebook)

5 - Deep Neural Networks

• Deep Neural Networks Notes (Markdown) (Notebook)

6 - Convolutional Neural Networks

• Convolutional Neural Network Notes (Markdown) (Notebook)

• PROJECT 2: Image Classification Convolutional Neural Network (Repo)

7 - Recurrent Neural Networks

• Recurrent Neural Network Exercise (Anna Karenina Text Generation) (Notebook)

• Sentiment Analysis Recurrent Neural Network (Notebook)

• PROJECT 3: Simpsons Script Generation with Recurrent Neural Networks (Repo)

Resources

• Lecture on RNNs and LSTM by Andrej Karpathy (CS231n - Stanford)
• Understanding LTSM Networks by Christopher Olah (Blog)
• The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy (Blog)
• A Beginner's Guide to Recurrent Networks and LSTMs (Blog)
• Recurrent Neural Networks in TensorFlow (Tutorial)
• RNNs in TensorFlow (Blog)

8 - Embeddings and Word2Vec

• Embeddings and Word2Vec (Skip-Gram) Exercise (Notebook)

Resources

• Word2Vec Overview: The Skip-Gram Model by Chris McCormick (Blog)
• Efficient Estimation of Word Respresentations in Vector Space by Mikolov et al. (Paper) [PDF] - (original Word2Vec Paper)
• Distributed Representations of Words and Phrases and their Compositionality by Mikolov et al. (Paper) [PDF] - (improvements to orignial paper)
• Word2Vec: NLP with Deep Learning with TensorFlow by Thushan Ganegedara (Blog)
• TensorFlow Word2Vec Tutorial (Tutorial)

9 - TensorBoard

• Viewing Graphs (Notebook)
• Name Scopes (Notebook)
• Inspecting Variables - TF Summaries (Notebook)
• Choosing Hyperparameters (Notebook)

Resources

• Hands-on TensorBoard Tutorial (TensorFlow Dev Summit 2017)
• TensorBoard: Visualizing Learning (Tutorial)

10 - Weight Initialization

• Comparing Different Weight Initializations on MNIST (Notebook)

Resources

• Understanding the Difficulty of Training Deep Feedforward Neural Networks by Bengio et al. (Paper) [PDF]
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by Microsoft Research (Paper) [PDF]
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift by Google Research (Paper) [PDF]

11 - Transfer Learning

• Transfer Learning with VGGNet (Notebook)

12 - Sequence to Sequence RNN

• Character Sequence to Sequence (Notebook)

• PROJECT 4: English to French Translation with Sequence to Sequence RNN (Repo)

13 - Reinforcement Learning

• Playing Cart-Pole using Q-Learning (Notebook)

14 - Autoencoders

• Simple Autoencoder on MNIST (Notebook)
• Conventional Autoencoder on MNIST (Notebook)

15 - Generative Adversarial Networks

Intro to GANs

• Generating Handwritten Digits with GANs (Notebook)

Deep Convolutional GANs

• Batch Normalization Lesson (Notebook)
• Batch Normalization Exercise (Notebook)
• Deep Convolutional GAN on Street View House Numbers (SVHN) Dataset (Notebook)
• PROJECT 5: Face Generation DCGAN (Repo)