Artificial intelligence for big data

Artificial Intelligence for Big Data Complete guide to automating Big Data solutions using Artificial Intelligence techniques Anand Deshpande Manish Kumar BIRMINGHAM - MUMBAI Artificial Intelligence for Big Data Copyright © 2018 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews Every effort has been made in the preparation of this book to ensure the accuracy of the information presented However, the information contained in this book is sold without warranty, either express or implied Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals However, Packt Publishing cannot guarantee the accuracy of this information Commissioning Editor: Sunith Shetty Acquisition Editor: Tushar Gupta Content Development Editor: Tejas Limkar Technical Editor: Dinesh Chaudhary Copy Editor: Safis Editing Project Coordinator: Manthan Patel Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Tania Dutta Production Coordinator: Aparna Bhagat First published: May 2018 Production reference: 1170518 Published by Packt Publishing Ltd Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78847-217-3 www.packtpub.com mapt.io Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career For more information, please visit our website Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy Get in touch with us at service@packtpub.com for more details At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks Contributors About the authors Anand Deshpande is the Director of big data delivery at Datametica Solutions He is responsible for partnering with clients on their data strategies and helps them become datadriven He has extensive experience with big data ecosystem technologies He has developed a special interest in data science, cognitive intelligence, and an algorithmic approach to data management and analytics He is a regular speaker on data science and big data at various events This book and anything worthwhile in my life is possible only with the blessings of my spiritual Guru, parents, and in-laws; and with unconditional support and love from my wife, Mugdha, and daughters, Devyani and Sharvari Thank you to my co-author, Manish Kumar, for his cooperation Many thanks to Mr Rajiv Gupta and Mr Sunil Kakade for their support and mentoring Manish Kumar is a Senior Technical Architect at Datametica Solutions He has more than 11 years of industry experience in data management as a data, solutions, and product architect He has extensive experience in building effective ETL pipelines, implementing security over Hadoop, implementing real-time data analytics solutions, and providing innovative and best possible solutions to data science problems He is a regular speaker on big data and data science I would like to thank my parents, Dr N.K Singh and Dr Rambha Singh, for their blessings The time spent on this book has taken some precious time from my wife, Mrs Swati Singh, and my adorable son, Lakshya Singh I not have enough words to thank my co-author and friend, Mr Anand Deshpande Niraj Kumar and Rajiv Gupta have my gratitude too About the reviewers Albenzo Coletta is a senior software and system engineer in robotics, defense, avionics, and telecoms He has a master's in computational robotics He was an industrial researcher in AI, a designer for a robotic communications system for COMAU, and a business analyst He designed a neuro-fuzzy system for financial problems (with Sannio University) and also designed a recommender system for a few key Italian editorial groups He was also a consultant at UCID (Ministry of Economics and Finance) He developed a mobile human robotic interaction system Giancarlo Zaccone has more than 10 years, experience in managing research projects in scientific and industrial areas He has worked as a researcher at the CNR, the National Research Council, in projects on parallel numerical computing, and in scientific visualization He is a senior software engineer at a consulting company, developing and testing software systems for space and defense applications He holds a master's in physics from University of Naples Federico II and a 2nd-level PG master's in scientific computing from La Sapienza of Rome Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea Table of Contents Preface Chapter 1: Big Data and Artificial Intelligence Systems Results pyramid What the human brain does best Sensory input Storage Processing power Low energy consumption What the electronic brain does best Speed information storage Processing by brute force Best of both worlds Big Data Evolution from dumb to intelligent machines Intelligence Types of intelligence Intelligence tasks classification Big data frameworks Batch processing Real-time processing Intelligent applications with Big Data Areas of AI Frequently asked questions Summary Chapter 2: Ontology for Big Data Human brain and Ontology Ontology of information science Ontology properties Advantages of Ontologies Components of Ontologies The role Ontology plays in Big Data Ontology alignment Goals of Ontology in big data Challenges with Ontology in Big Data RDF—the universal data format RDF containers RDF classes RDF properties RDF attributes 10 10 10 11 11 11 11 12 12 13 15 16 16 17 17 18 19 20 20 20 22 23 24 26 27 28 29 30 32 32 33 33 36 37 37 38 Table of Contents Using OWL, the Web Ontology Language SPARQL query language Generic structure of an SPARQL query Additional SPARQL features Building intelligent machines with Ontologies Ontology learning Ontology learning process Frequently asked questions Summary Chapter 3: Learning from Big Data Supervised and unsupervised machine learning The Spark programming model The Spark MLlib library The transformer function The estimator algorithm Pipeline Regression analysis Linear regression Least square method Generalized linear model Logistic regression classification technique Logistic regression with Spark Polynomial regression Stepwise regression Forward selection Backward elimination Ridge regression LASSO regression Data clustering The K-means algorithm K-means implementation with Spark ML Data dimensionality reduction Singular value decomposition Matrix theory and linear algebra overview The important properties of singular value decomposition SVD with Spark ML The principal component analysis method The PCA algorithm using SVD Implementing SVD with Spark ML Content-based recommendation systems Frequently asked questions Summary Chapter 4: Neural Network for Big Data [ ii ] 38 40 42 43 44 47 48 50 51 52 53 58 61 61 62 62 63 64 64 68 68 70 70 72 72 72 73 73 73 75 77 78 80 80 84 84 86 87 87 88 93 94 95 Table of Contents Fundamentals of neural networks and artificial neural networks Perceptron and linear models Component notations of the neural network Mathematical representation of the simple perceptron model Activation functions Sigmoid function Tanh function ReLu Nonlinearities model Feed-forward neural networks Gradient descent and backpropagation Gradient descent pseudocode Backpropagation model Overfitting Recurrent neural networks The need for RNNs Structure of an RNN Training an RNN Frequently asked questions Summary Chapter 5: Deep Big Data Analytics Deep learning basics and the building blocks Gradient-based learning Backpropagation Non-linearities Dropout Building data preparation pipelines Practical approach to implementing neural net architectures Hyperparameter tuning Learning rate Number of training iterations Number of hidden units Number of epochs Experimenting with hyperparameters with Deeplearning4j Distributed computing Distributed deep learning DL4J and Spark API overview TensorFlow Keras Frequently asked questions Summary Chapter 6: Natural Language Processing [ iii ] 96 98 99 100 102 103 104 104 106 106 108 112 113 115 117 117 118 118 120 122 123 124 126 128 130 132 133 140 143 144 145 146 146 147 152 154 155 155 157 158 159 161 162 ...Artificial Intelligence for Big Data Complete guide to automating Big Data solutions using Artificial Intelligence techniques Anand Deshpande Manish Kumar BIRMINGHAM - MUMBAI Artificial Intelligence for. .. progressively learn about Artificial Intelligence for Big Data starting from the fundamentals and eventually move towards cognitive intelligence Chapter 1, Big Data and Artificial Intelligence Systems,... information storage Processing by brute force Best of both worlds Big Data Evolution from dumb to intelligent machines Intelligence Types of intelligence Intelligence tasks classification Big data