Schedule and Syllabus

Unless otherwise specified the course lectures and meeting times are:

Tuesday, Thursday 3:00-4:20
Location: Gates B1
EventDateDescriptionCourse Materials
Lecture Mar 29 Intro to NLP and Deep Learning Suggested Readings:
  1. [Linear Algebra Review]
  2. [Probability Review]
  3. [Convex Optimization Review]
  4. [More Optimization (SGD) Review]
  5. [From Frequency to Meaning: Vector Space Models of Semantics]

[Lecture Notes 1]
[python tutorial] [slides]
Lecture Mar 31 Simple Word Vector representations: word2vec, GloVe Suggested Readings:
  1. [Distributed Representations of Words and Phrases and their Compositionality]
  2. [Efficient Estimation of Word Representations in Vector Space]
A1 released Apr 4 Pset #1 released [Pset 1] [Pset 1 Solutions] [Pset 1 Solutions Code]
Lecture Apr 5 Advanced word vector representations: language models, softmax, single layer networks Suggested Readings:
  1. [GloVe: Global Vectors for Word Representation]
  2. [Improving Word Representations via Global Context and Multiple Word Prototypes]
Lecture Apr 7 Neural Networks and backpropagation -- for named entity recognition Suggested Readings:
  1. [UFLDL tutorial]
  2. [Learning Representations by Backpropogating Errors]
[Lecture Notes 3]
Lecture Apr 12 Project Advice, Neural Networks and Back-Prop (in full gory detail) Suggested Readings:
  1. [Natural Language Processing (almost) from Scratch]
  2. [A Neural Network for Factoid Question Answering over Paragraphs]
  3. [Grounded Compositional Semantics for Finding and Describing Images with Sentences]
  4. [Deep Visual-Semantic Alignments for Generating Image Descriptions]
  5. [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank]

Lecture Apr 14 Practical tips: gradient checks, overfitting, regularization, activation functions, details Suggested Readings:
  1. [Practical recommendations for gradient-based training of deep architectures]
  2. [UFLDL page on gradient checking]
A1 Due Apr 19 Pset #1 due
Lecture Apr 19 Introduction to Tensorflow Suggested Readings:
  1. [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems]
[slides] [AWS Tutorial] [AWS Tutorial Supplementary] [AWS Tutorial Video]
A2 released Apr 20 Pset #2 released [Pset 2][Pset 2 Solutions] [Pset 2 Solutions Code]
Lecture Apr 21 Recurrent neural networks -- for language modeling and other tasks Suggested Readings:
  1. [Recurrent neural network based language model]
  2. [Extensions of recurrent neural network language model]
  3. [Opinion Mining with Deep Recurrent Neural Networks]
Lecture Apr 26 GRUs and LSTMs -- for machine translation Suggested Readings:
  1. [Long Short-Term Memory]
  2. [Gated Feedback Recurrent Neural Networks]
  3. [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling]
Proposal due Apr 28 Course Project Proposal due [proposal description]
Lecture Apr 28 Recursive neural networks -- for parsing Suggested Readings:
  1. [Parsing with Compositional Vector Grammars]
  2. [Subgradient Methods for Structured Prediction]
  3. [Parsing Natural Scenes and Natural Language with Recursive Neural Networks]

[Lecture Notes 5]
Lecture May 3 Recursive neural networks -- for different tasks (e.g. sentiment analysis) Suggested Readings:
  1. [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank]
  2. [Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection]
  3. [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks]
A2 Due May 5 Pset #2 Due date
Lecture May 5 Review Session for Midterm

Suggested Readings: N/A

Midterm May 10 In-class midterm [midterm solutions]
A3 released May 12 Pset #3 released [Pset 3] [Pset 3 Solutions] [Pset 3 Solutions Code]
Lecture May 12 Convolutional neural networks -- for sentence classification Suggested Readings:
  1. [A Convolutional Neural Network for Modelling Sentences]
Milestone May 15 Course Project Milestone [milestone description]
Lecture May 17 Guest Lecture with Andrew Maas: Speech recognition Suggested Readings:
  1. [ Deep Neural Networks for Acoustic Modeling in Speech Recognition]
Lecture May 19 Guest Lecture with Thang Luong: Machine Translation Suggested Readings:
  1. [ Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models]
  2. [ Addressing the Rare Word Problem in Neural Machine Translation]
  3. [ Advances in natural language processing]
  4. [ Neural machine translation by jointly learning to align and translate]
A3 Due May 21 Pset #3 Due date
Lecture May 24 Guest Lecture with Quoc Le: Seq2Seq and Large Scale DL Suggested Readings:
  1. [ Sequence to Sequence with Neural Networks]
  2. [ Neural Machine Translation by Jointly Learning to Align and Translate]
  3. [ A Neural Conversation Model]
  4. [ Neural Programmer: Include Latent Programs with Gradient Descent]
Lecture May 26 The future of Deep Learning for NLP: Dynamic Memory Networks Suggested Readings:
  1. [Ask me anthing: Dynamic Memory Networks for NLP]
Poster Presentation June 1 Final project poster presentations: 2-5 pm, Gates patio
Final Project Due Jun 3 Final course project due date [project description]