wooodhead/lectures

Oxford Deep NLP 2017 course

Preamble

This repository contains the lecture slides and course description for the Deep Natural Language Processing course offered in Hilary Term 2017 at the University of Oxford.

This is an advanced course on natural language processing. Automatically processing natural language inputs and producing language outputs is a key component of Artificial General Intelligence. The ambiguities and noise inherent in human communication render traditional symbolic AI techniques ineffective for representing and analysing language data. Recently statistical techniques based on neural networks have achieved a number of remarkable successes in natural language processing leading to a great deal of commercial and academic interest in the field

This is an applied course focussing on recent advances in analysing and generating speech and text using recurrent neural networks. We introduce the mathematical definitions of the relevant machine learning models and derive their associated optimisation algorithms. The course covers a range of applications of neural networks in NLP including analysing latent dimensions in text, transcribing speech to text, translating between languages, and answering questions. These topics are organised into three high level themes forming a progression from understanding the use of neural networks for sequential language modelling, to understanding their use as conditional language models for transduction tasks, and finally to approaches employing these techniques in combination with other mechanisms for advanced applications. Throughout the course the practical implementation of such models on CPU and GPU hardware is also discussed.

This course is organised by Phil Blunsom and delivered in partnership with the DeepMind Natural Language Research Group.

Lecturers

• Phil Blunsom (Oxford University and DeepMind)
• Chris Dyer (Carnegie Mellon University and DeepMind)
• Edward Grefenstette (DeepMind)
• Karl Moritz Hermann (DeepMind)
• Andrew Senior (DeepMind)
• Wang Ling (DeepMind)
• Jeremy Appleyard (NVIDIA)

Timetable

Practicals

• Group 1 - Monday, 9:00-11:00 (Weeks 2-8), 60.05 Thom Building
• Group 2 - Friday, 16:00-18:00 (Weeks 2-8), Room 379

Lectures

Public Lectures are held in Lecture Theatre 1 of the Maths Institute, on Tuesdays and Thursdays, 16:00-18:00 (Hilary Term Weeks 1,3,4,5,6,7,8).

Lecture Materials

1. Lecture 1a - Introduction [Phil Blunsom]

This lecture introduces the course and motivates why it is interesting to study language processing using Deep Learning techniques.

[slides] [video]

2. Lecture 1b - Deep Neural Networks Are Our Friends [Wang Ling]

This lecture revises basic machine learning concepts that students should know before embarking on this course.

[[slides]](Lecture 1b - Deep Neural Networks Are Our Friends.pdf) [video]

3. Lecture 2a- Word Level Semantics [Ed Grefenstette]

Words are the core meaning bearing units in language. Representing and learning the meanings of words is a fundamental task in NLP and in this lecture the concept of a word embedding is introduced as a practical and scalable solution.

[slides] [video]

4. Lecture 2b - Overview of the Practicals [Chris Dyer]

This lecture motivates the practical segment of the course.

[slides] [video]

5. Lecture 3 - Language Modelling and RNNs Part 1 [Phil Blunsom]

Language modelling is important task of great practical use in many NLP applications. This lecture introduces language modelling, including traditional n-gram based approaches and more contemporary neural approaches. In particular the popular Recurrent Neural Network (RNN) language model is introduced and its basic training and evaluation algorithms described.

[slides] [video]

Reading

Textbook

• Deep Learning, Chapter 10.

Blogs

• The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej Karpathy.
• The unreasonable effectiveness of Character-level Language Models, Yoav Goldberg.
• Explaining and illustrating orthogonal initialization for recurrent neural networks, Stephen Merity.

6. Lecture 4 - Language Modelling and RNNs Part 2 [Phil Blunsom]

This lecture continues on from the previous one and considers some of the issues involved in producing an effective implementation of an RNN language model. The vanishing and exploding gradient problem is described and architectural solutions, such as Long Short Term Memory (LSTM), are introduced.

[slides] [video]

Reading

Textbook

• Deep Learning, Chapter 10.

Vanishing gradients, LSTMs etc.

• On the difficulty of training recurrent neural networks. Pascanu et al., ICML 2013.
• Long Short-Term Memory. Hochreiter and Schmidhuber, Neural Computation 1997.
• Learning Phrase Representations using RNN EncoderDecoder for Statistical Machine Translation. Cho et al, EMNLP 2014.
• Blog: Understanding LSTM Networks, Christopher Olah.

Dealing with large vocabularies

• A scalable hierarchical distributed language model. Mnih and Hinton, NIPS 2009.
• A fast and simple algorithm for training neural probabilistic language models. Mnih and Teh, ICML 2012.
• On Using Very Large Target Vocabulary for Neural Machine Translation. Jean et al., ACL 2015.
• Exploring the Limits of Language Modeling. Jozefowicz et al., arXiv 2016.
• Efficient softmax approximation for GPUs. Grave et al., arXiv 2016.
• Notes on Noise Contrastive Estimation and Negative Sampling. Dyer, arXiv 2014.
• Pragmatic Neural Language Modelling in Machine Translation. Baltescu and Blunsom, NAACL 2015

Regularisation and dropout

• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. Gal and Ghahramani, NIPS 2016.
• Blog: Uncertainty in Deep Learning, Yarin Gal.

Other stuff

• Recurrent Highway Networks. Zilly et al., arXiv 2016.
• Capacity and Trainability in Recurrent Neural Networks. Collins et al., arXiv 2016.
• Optimizing Performance of Recurrent Neural Networks on GPUs. Appleyard et al., arXiv 2016.

7. Lecture 5 - (Coming Soon) Text Classification [Karl Moritz Hermann]

[slides] [video]

8. Lecture 6 - (Coming Soon) Deep NLP on Nvidia GPUs [Jeremy Appleyard]

[slides] [video]

Piazza

We will be using Piazza to facilitate class discussion during the course. Rather than emailing questions directly, I encourage you to post your questions on Piazza to be answered by your fellow students, instructors, and lecturers. However do please do note that all the lecturers for this course are volunteering their time and may not always be available to give a reponse.

Find our class page at: https://piazza.com/ox.ac.uk/winter2017/dnlpht2017/home

Assessment

The primary assessment for this course will be a take-home assignment issued at the end of the term. This assignment will ask questions drawing on the concepts and models discussed in the course, as well as from selected research publications. The nature of the questions will include analysing mathematical descriptions of models and proposing extensions, improvements, or evaluations to such models. The assignment may also ask students to read specific research publications and discuss their proposed algorithms in the context of the course. In answering questions students will be expected to both present coherent written arguments and use appropriate mathematical formulae, and possibly pseudo-code, to illustrate answers.

The practical component of the course will be assessed in the usual way.