Artificial Intelligence for Big Data Complete guide to automating Big Data solutions using Artificial Intelligence techniques Artificial Intelligence for Big Data Copyright © 2018 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews Every effort has been made in the.
Artificial Intelligence for Big Data Complete guide to automating Big Data solutions using Artificial Intelligence techniques Artificial Intelligence for Big Data Copyright © 2018 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews Every effort has been made in the preparation of this book to ensure the accuracy of the information presented However, the information contained in this book is sold without warranty, either express or implied Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals However, Packt Publishing cannot guarantee the accuracy of this information Commissioning Editor: Sunith Shetty Acquisition Editor: Tushar Gupta Content Development Editor: Tejas Limkar Technical Editor: Dinesh Chaudhary Copy Editor: Safis Editing Project Coordinator: Manthan Patel Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Tania Dutta Production Coordinator: Aparna Bhagat First published: May 2018 Production reference: 1170518 Published by Packt Publishing Ltd Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78847-217- Table of Contents Preface Chapter 1: Big Data and Artificial Intelligence Systems Results pyramid What the human brain does best Sensory input Storage Processing power Low energy consumption What the electronic brain does best Speed information storage Processing by brute force Best of both worlds Big Data Evolution from dumb to intelligent machines Intelligence Types of intelligence Intelligence tasks classification Big data frameworks Batch processing Real-time processing Intelligent applications with Big Data Areas of AI Frequently asked questions Summary Chapter 2: Ontology for Big Data Human brain and Ontology Ontology of information science Ontology properties Advantages of Ontologies Components of Ontologies The role Ontology plays in Big Data Ontology alignment Goals of Ontology in big data Challenges with Ontology in Big Data RDF—the universal data format RDF containers RDF classes RDF properties RDF attributes 10 10 10 11 11 11 11 12 12 13 15 16 16 17 17 18 19 20 20 20 22 23 24 26 27 28 29 30 32 32 33 33 36 37 37 38 Table of Contents Using OWL, the Web Ontology Language SPARQL query language Generic structure of an SPARQL query Additional SPARQL features Building intelligent machines with Ontologies Ontology learning Ontology learning process Frequently asked questions Summary Chapter 3: Learning from Big Data Supervised and unsupervised machine learning The Spark programming model The Spark MLlib library The transformer function The estimator algorithm Pipeline Regression analysis Linear regression Least square method Generalized linear model Logistic regression classification technique Logistic regression with Spark Polynomial regression Stepwise regression Forward selection Backward elimination Ridge regression LASSO regression Data clustering The K-means algorithm K-means implementation with Spark ML Data dimensionality reduction Singular value decomposition Matrix theory and linear algebra overview The important properties of singular value decomposition SVD with Spark ML The principal component analysis method The PCA algorithm using SVD Implementing SVD with Spark ML Content-based recommendation systems Frequently asked questions Summary Chapter 4: Neural Network for Big Data [ ii ] 38 40 42 43 44 47 48 50 51 52 53 58 61 61 62 62 63 64 64 68 68 70 70 72 72 72 73 73 73 75 77 78 80 80 84 84 86 87 87 88 93 94 95 Table of Contents Fundamentals of neural networks and artificial neural networks Perceptron and linear models Component notations of the neural network Mathematical representation of the simple perceptron model Activation functions Sigmoid function Tanh function ReLu Nonlinearities model Feed-forward neural networks Gradient descent and backpropagation Gradient descent pseudocode Backpropagation model Overfitting Recurrent neural networks The need for RNNs Structure of an RNN Training an RNN Frequently asked questions Summary Chapter 5: Deep Big Data Analytics Deep learning basics and the building blocks Gradient-based learning Backpropagation Non-linearities Dropout Building data preparation pipelines Practical approach to implementing neural net architectures Hyperparameter tuning Learning rate Number of training iterations Number of hidden units Number of epochs Experimenting with hyperparameters with Deeplearning4j Distributed computing Distributed deep learning DL4J and Spark API overview TensorFlow Keras Frequently asked questions Summary Chapter 6: Natural Language Processing [ iii ] 96 98 99 100 102 103 104 104 106 106 108 112 113 115 117 117 118 118 120 122 123 124 126 128 130 132 133 140 143 144 145 146 146 147 152 154 155 155 157 158 159 161 162 Table of Contents Natural language processing basics Text preprocessing Removing stop words Stemming Porter stemming Snowball stemming Lancaster stemming Lovins stemming Dawson stemming Lemmatization N-grams Feature extraction One hot encoding TF-IDF CountVectorizer Word2Vec CBOW Skip-Gram model Applying NLP techniques Text classification Introduction to Naive Bayes' algorithm Random Forest Naive Bayes' text classification code example Implementing sentiment analysis Frequently asked questions Summary Chapter 7: Fuzzy Systems Fuzzy logic fundamentals Fuzzy sets and membership functions Attributes and notations of crisp sets Operations on crisp sets Properties of crisp sets Fuzzification Defuzzification Defuzzification methods Fuzzy inference ANFIS network Adaptive network ANFIS architecture and hybrid learning algorithm Fuzzy C-means clustering NEFCLASS Frequently asked questions Summary Chapter 8: Genetic Programming 163 165 165 167 167 168 168 169 169 170 170 171 171 172 175 176 176 178 179 180 181 182 183 185 187 188 189 190 191 192 193 194 194 197 197 197 198 198 199 202 206 208 209 210 [ iv ] .. .Artificial Intelligence for Big Data Complete guide to automating Big Data solutions using Artificial Intelligence techniques Artificial Intelligence for Big Data Copyright... progressively learn about Artificial Intelligence for Big Data starting from the fundamentals and eventually move towards cognitive intelligence Chapter 1, Big Data and Artificial Intelligence Systems,... WinRAR/7-Zip for Windows Zipeg/iZip/UnRarX for Mac 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing /Artificial- Intelligence- for- Big- Data