Building high-level features using large-scale unsupervised learning

Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California, Santa Barbara Slide adapted from Andrew Ng (Stanford), Nando de Freitas (UBC) Agenda Motivation Approach Sparse Deep Auto-encoder Local Receptive Field L2 Pooling Local contrast normalization Overall Model Parallelism Evaluation Discussion Motivation Motivation • Feature learning • Supervised learning • Need large number of labeled data • Unsupervised learning • Example: Build face detector without having labeled face images • Building high-level features using unlabeled data Motivation • Previous works • Auto encoder • Sparse coding • Result: Only learns low level features • Reason: Computational constraints • Approach • Dataset • Model • Computational resources Approach Sparse Deep Auto-encoder • Auto-encoder • Neural network • Unsupervised learning • Back-propagation Sparse Deep Auto-encoder (cnt’d) • Sparse Coding • Input: Images x(1), x(2) x(m) • Learn: Bases (features) f1, f2, , fk, so that each input x can be approximately decomposed as: x=∑ajfj s.t aj’s are mostly zero (“sparse”) Sparse Deep Auto-encoder (cnt’d) Sparse Deep Auto-encoder (cnt’d) • Sparse Coding • Regularizer 10 27 28 Model Parallelism • Weights divided according to locality of image and store on different machine 29 evaluation 30 Evaluation • 10M Youtube unlabeled frames of size 200x200 • 1B parameters • 1000 machines • 16,000 cores 31 Experiment on Faces • Test set • 37,000 images • 13,026 face images • Best neuron 32 Experiment on Faces (cnt’d) • Visualization • Top stimulus (images) for face neuron • Optimal stimulus for face neuron 33 Experiment on Faces (cnt’d) • Invariances Properties 34 Experiment on Faces (cnt’d) • Invariances Properties 35 Experiment on Cat/Human body • Test set • Cat: 10,000 positive, 18,409 negative • Human body: 13,026 positive, 23,974 negative • Accuracy 36 ImageNet classification • Recognizing images • Dataset • 20,000 categories • 14M images • Accuracy • 15.8% • State of art: 9.3% 37 DISCUSSION 38 Discussion • Deep learning • Unsupervised feature learning • Learning multiple layers of representation • Increase accuracy: Invariance, contrast normalization • Scalability 39 REFERENCES 40 References Quoc Le et al., “Building High-level Features using Large Scale Unsupervised Learning” Nando de Freitas, “Deep Learning”, URL: https://www.youtube.com/watch? v=g4ZmJJWR34Q Andrew Ng, “Sparse autoencoder”, URL: http://www.stanford.edu/class/archive/cs/cs294a/cs294a.1104/sparseAutoencode r.pdf Andrew Ng, “Machine Learning and AI via Brain Simulations”, URL: https://forum.stanford.edu/events/2011slides/plenary/2011plenaryNg.pdf Andrew Ng, “Deep Learning”, URL: http://www.ipam.ucla.edu/publications/gss2012/gss2012_10595.pdf 41 ... Computational resources Approach Sparse Deep Auto-encoder • Auto-encoder • Neural network • Unsupervised learning • Back-propagation Sparse Deep Auto-encoder (cnt’d) • Sparse Coding • Input: Images... zero (“sparse”) Sparse Deep Auto-encoder (cnt’d) Sparse Deep Auto-encoder (cnt’d) • Sparse Coding • Regularizer 10 Sparse Deep Auto-encoder (cnt’d) • Sparse Deep Auto-encoder • Multiple hidden layers...Agenda Motivation Approach Sparse Deep Auto-encoder Local Receptive Field L2 Pooling Local contrast normalization Overall Model Parallelism