Deep learning with theano

Deep Learning with Theano Table of Contents Deep Learning with Theano Credits About the Author Acknowledgments About the Reviewers www.PacktPub.com eBooks, discount offers, and more Why subscribe? Customer Feedback Preface What this book covers Why Theano? What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Errata Piracy Questions Theano Basics The need for tensors Installing and loading Theano Conda package and environment manager Installing and running Theano on CPU GPU drivers and libraries Installing and running Theano on GPU Tensors Graphs and symbolic computing Operations on tensors Dimension manipulation operators Elementwise operators Reduction operators Linear algebra operators Memory and variables Functions and automatic differentiation Loops in symbolic computing Configuration, profiling and debugging Summary Classifying Handwritten Digits with a Feedforward Network The MNIST dataset Structure of a training program Classification loss function Single-layer linear model Cost function and errors Backpropagation and stochastic gradient descent Multiple layer model Convolutions and max layers Training Dropout Inference Optimization and other update rules Related articles Summary Encoding Word into Vector Encoding and embedding Dataset Continuous Bag of Words model Training the model Visualizing the learned embeddings Evaluating embeddings – analogical reasoning Evaluating embeddings – quantitative analysis Application of word embeddings Weight tying Further reading Summary Generating Text with a Recurrent Neural Net Need for RNN A dataset for natural language Simple recurrent network LSTM network Gated recurrent network Metrics for natural language performance Training loss comparison Example of predictions Applications of RNN Related articles Summary Analyzing Sentiment with a Bidirectional LSTM Installing and configuring Keras Programming with Keras SemEval 2013 dataset Preprocessing text data Designing the architecture for the model Vector representations of words Sentence representation using bi-LSTM Outputting probabilities with the softmax classifier Compiling and training the model Evaluating the model Saving and loading the model Running the example Further reading Summary Locating with Spatial Transformer Networks MNIST CNN model with Lasagne A localization network Recurrent neural net applied to images Unsupervised learning with co-localization Region-based localization networks Further reading Summary Classifying Images with Residual Networks Natural image datasets Batch normalization Global average pooling Residual connections Stochastic depth Dense connections Multi-GPU Data augmentation Further reading Summary Translating and Explaining with Encoding – decoding Networks Sequence-to-sequence networks for natural language processing Seq2seq for translation Seq2seq for chatbots Improving efficiency of sequence-to-sequence network Deconvolutions for images Multimodal deep learning Further reading Summary Selecting Relevant Inputs or Memories with the Mechanism of Attention Differentiable mechanism of attention Better translations with attention mechanism Better annotate images with attention mechanism Store and retrieve information in Neural Turing Machines Memory networks Episodic memory with dynamic memory networks Further reading Summary 10 Predicting Times Sequences with Advanced RNN Dropout for RNN Deep approaches for RNN Stacked recurrent networks Deep transition recurrent network Highway networks design principle Recurrent Highway Networks Further reading Summary 11 Learning from the Environment with Reinforcement Reinforcement learning tasks Simulation environments Q-learning Deep Q-network Training stability Policy gradients with REINFORCE algorithms Related articles Summary 12 Learning Features with Unsupervised Generative Networks Generative models Restricted Boltzmann Machines Deep belief bets Generative adversarial networks Improve GANs Semi-supervised learning Further reading Summary 13 Extending Deep Learning with Theano Theano Op in Python for CPU Theano Op in Python for the GPU Theano Op in C for CPU Theano Op in C for GPU Coalesced transpose via shared memory, NVIDIA parallel for all Model conversions The future of artificial intelligence Further reading Summary Index Deep Learning with Theano Deep Learning with Theano Copyright © 2017 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews Every effort has been made in the preparation of this book to ensure the accuracy of the information presented However, the information contained in this book is sold without warranty, either express or implied Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals However, Packt Publishing cannot guarantee the accuracy of this information First published: July 2017 Production reference: 1280717 Published by Packt Publishing Ltd Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78646-582-5 www.packtpub.com Credits Author Christopher Bourez Reviewers Matthieu de Beaucorps Frederic Bastien Arnaud Bergeron Pascal Lamblin Commissioning Editor Amey Varangaonkar Acquisition Editor Veena Pagare Content Development Editor Amrita Noronha Technical Editor Akash Patel Copy Editor Safis Editing Project Coordinator Shweta H Birwatkar Proofreader Safis Editing Indexer Pratik Shirodkar Graphics Tania Dutta Production Coordinator Shantanu Zagade Cover Work Shantanu N Zagade I identity connection / Residual connections identity connections / Highway networks design principle images deconvolutions / Deconvolutions for images Inceptionism / Deconvolutions for images Independent Component Analysis (ICA) / Visualizing the learned embeddings inference / Inference internal covariate shift / Batch normalization Intersection over Union (IOU) / Region-based localization networks K Keras installing / Installing and configuring Keras configuring / Installing and configuring Keras programming / Programming with Keras SemEval 2013 dataset / SemEval 2013 dataset model, training / Compiling and training the model model, compiling / Compiling and training the model kernel / Theano Op in C for GPU L Lasagne MNIST CNN model / MNIST CNN model with Lasagne Latent Sementic Analysis / Indexing (LSA / LSI) / Visualizing the learned embeddings layer input normalization / Batch normalization learned embeddings visualizing / Visualizing the learned embeddings linear algebra operators / Linear algebra operators Linear Discriminant Analysis (LDA) / Visualizing the learned embeddings localization network about / A localization network recurrent neural net, applied to images / Recurrent neural net applied to images Locally Linear Embedding (LLE) / Visualizing the learned embeddings Long Short-Term Memory (LSTM) / Sentence representation using bi-LSTM loops in symbolic computing / Loops in symbolic computing loss comparison training / Training loss comparison loss function classification / Classification loss function LSTM network / LSTM network M max layers / Convolutions and max layers memory / Memory and variables memory networks about / Memory networks episodic memory, with dynamic memory networks / Episodic memory with dynamic memory networks MNIST CNN model with Lasagne / MNIST CNN model with Lasagne MNIST dataset / The MNIST dataset model training / Training the model compiling, in Keras / Compiling and training the model training, in Keras / Compiling and training the model evaluating / Evaluating the model loading / Saving and loading the model saving / Saving and loading the model example, executing / Running the example model collapse / Improve GANs Modified National Institute of Standards and Technology (MNIST) / The MNIST dataset momentum / Optimization and other update rules Monte Carlo Tree Search (MCTS) / Q-learning multi-GPU / Multi-GPU multi-layer perceptron (MLP) / Multiple layer model Multi Dimensional Scaling (MDS) / Visualizing the learned embeddings multimodal deep learning / Multimodal deep learning multiple layer model / Multiple layer model N natural image datasets about / Natural image datasets batch normalization / Batch normalization global average pooling / Global average pooling natural language performance metrics for / Metrics for natural language performance Natural Language Processing (NLP) / Sequence-to-sequence networks for natural language processing negative particles / Restricted Boltzmann Machines Nesterov Accelerated Gradient / Optimization and other update rules network input normalization / Batch normalization Neural Machine Translation (NMT) / Weight tying Neural Network Language Models (NNLM) / Weight tying Neural Turing Machines (NTM) retrieve information in / Store and retrieve information in Neural Turing Machines store information in / Store and retrieve information in Neural Turing Machines about / Store and retrieve information in Neural Turing Machines O off-policy training / Training stability Online training / Training stability Open-AI Gym about / Simulation environments URL / Simulation environments optimal state value function v(s) / Q-learning optimization / Optimization and other update rules out-of-vocabulary (OOV) / Preprocessing text data P Part of Speech (POS) / Applications of RNN Platoon reference link / Multi-GPU policy gradients (PG) about / Policy gradients with REINFORCE algorithms with REINFORCE algorithms / Policy gradients with REINFORCE algorithms policy network / Policy gradients with REINFORCE algorithms positive and negative phases / Restricted Boltzmann Machines predictions example / Example of predictions Principal Component Analysis (PCA) / Visualizing the learned embeddings Q Q-learning / Q-learning quantitative analysis / Evaluating embeddings – quantitative analysis R recurrent highway networks (RHN) / Recurrent Highway Networks Recurrent Neural Network (RNN) / Encoding and embedding recurrent neural networks (RNN) need for / Need for RNN about / Need for RNN applications / Applications of RNN reduction operators / Reduction operators region-based localisation networks / Region-based localization networks Region Proposal Network (RPN) / Region-based localization networks reinforcement learning tasks / Reinforcement learning tasks replay memory / Training stability residual block / Residual connections residual connections / Residual connections residuals / Residual connections Restricted Boltzmann Machine / Restricted Boltzmann Machines RMSProp / Optimization and other update rules RNN dropout / Dropout for RNN deep approaches / Deep approaches for RNN S SegNet network / Deconvolutions for images semi-supervised learning / Semi-supervised learning sequence-to-sequence (Seq2seq) network for natural language processing / Sequence-to-sequence networks for natural language processing about / Sequence-to-sequence networks for natural language processing, Seq2seq for translation for translation / Seq2seq for translation for chatbots / Seq2seq for chatbots efficiency, improving / Improving efficiency of sequence-to-sequence network SharpMask / Deconvolutions for images simple recurrent network about / Simple recurrent network LSTM network / LSTM network gated recurrent network / Gated recurrent network simulation environments / Simulation environments single-layer linear model / Single-layer linear model Single Instruction Multiple Data (SIMD) / Theano Op in C for GPU spatial transformer networks (STN) / A localization network stability training / Training stability stacked recurrent networks / Stacked recurrent networks state-action value network / Deep Q-network state value network / Policy gradients with REINFORCE algorithms state values / Q-learning stochastic depth / Stochastic depth stochastic gradient descent (SGD) / Backpropagation and stochastic gradient descent, Optimization and other update rules Streaming Multiprocessors (SM) / Theano Op in C for GPU symbolic computing / Graphs and symbolic computing loops in / Loops in symbolic computing T t-distributed Stochastic Neighbor Embedding (t-SNE) / Visualizing the learned embeddings Tensor Processing Units (TPU) / Model conversions tensors need for / The need for tensors about / Tensors operations on / Operations on tensors dimension manipulation operators / Dimension manipulation operators elementwise operators / Elementwise operators reduction operators / Reduction operators linear algebra operators / Linear algebra operators text data preprocessing / Preprocessing text data Theano installing / Installing and loading Theano loading / Installing and loading Theano conda package / Conda package and environment manager environment manager / Conda package and environment manager installing, on CPU / Installing and running Theano on CPU executing, on CPU / Installing and running Theano on CPU GPU drivers / GPU drivers and libraries GPU libraries / GPU drivers and libraries installing, on GPU / Installing and running Theano on GPU executing, on GPU / Installing and running Theano on GPU debugging / Configuration, profiling and debugging profiling / Configuration, profiling and debugging configuration / Configuration, profiling and debugging Theano Op in Python, for CPU / Theano Op in Python for CPU in Python, for GPU / Theano Op in Python for the GPU in C, for CPU / Theano Op in C for CPU in C for GPU / Theano Op in C for GPU Torcs URL / Simulation environments training program structure / Structure of a training program script environment, setting up / Structure of a training program data, loading / Structure of a training program data, preprocessing / Structure of a training program model, building / Structure of a training program training / Structure of a training program U unsupervised learning with co-localisation / Unsupervised learning with co-localization update rules / Optimization and other update rules V validation dataset training / Training variables / Memory and variables variational RNN / Dropout for RNN W weight tying (WT) / Weight tying word embeddings application / Application of word embeddings Word Error Rate (WER) / Metrics for natural language performance Y You Only See Once (YOLO) architecture / Region-based localization networks ... In this last case, extra amounts of unlabeled data train better features as a basis for supervised learning Chapter 13, Extending Deep Learning with Theano, extends the set of possibilities in Deep Learning with Theano It addresses the way to create new operators for the computation graph, either in Python for... Semi-supervised learning Further reading Summary 13 Extending Deep Learning with Theano Theano Op in Python for CPU Theano Op in Python for the GPU Theano Op in C for CPU Theano Op in C for GPU Coalesced transpose via shared memory, NVIDIA parallel for all... Model conversions The future of artificial intelligence Further reading Summary Index Deep Learning with Theano Deep Learning with Theano Copyright © 2017 Packt Publishing All rights reserved No part of this book may be reproduced, stored in