Chapter 1. A Gentle Introduction to Spark What is Apache Spark? Apache Spark is a processing system that makes working with big data simple. It is a group of much more than a programming paradigm but an ecosystem of a variety of packages, libraries, and systems built on top of the Core of Spark. Spark Core consists of two APIs. The Unstructured and Structured APIs. The Unstructured API is Spark’s lower level set of APIs including Resilient Distributed Datasets (RDDs), Accumulators, and Broadcast variables. The Structured API consists of DataFrames, Datasets, Spark SQL and is the interface that most users should use. The difference between the two is that one is optimized to work with structured data in a spreadsheet-like interface while the other is meant for manipulation of raw java objects. Outside of Spark Core sit a variety of tools, libraries, and languages like MLlib for performing machine learning, the GraphX module for performing graph processing, and SparkR for working with Sp
Chapter 1. A Gentle Introduction to Spark
Chapter 2. Structured API Overview Spark’s Structured APIs For our purposes there is a spectrum of types of data. The two extremes of the spectrum are structured and unstructured. Structured and semi-structured data refer to to data that have structure that a computer can understand relatively easily. Unstructured data, like a poem or prose, is much harder to a computer to understand. Spark’s Structured APIs allow for transformations and actions on structured and semi-structured data. The Structured APIs specifically refer to operations on DataFrames, Datasets, and in Spark SQL and were created as a high level interface for users to manipulate big data. This section will cover all the principles of the Structured APIs. Although distinct in the book, the vast majority of these user-facing operations apply to both batch as well as streaming computation. The Structured API is the fundamental abstraction that you will leverage to write your data flows. Thus far in this book we have taken a
Chapter 2. Structured API Overview
Chapter 3. Basic Structured Operations Chapter Overview In the previous chapter we introduced the core abstractions of the Structured API. This chapter will move away from the architectural concepts and towards the tactical tools you will use to manipulate DataFrames and the data within them. This chapter will focus exclusively on single DataFrame operations and avoid aggregations, window functions, and joins which will all be discussed in depth later in this section. Definitionally, a DataFrame consists of a series of records (like rows in a table), that are of type Row, and a number of columns (like columns in a spreadsheet) that represent an computation expression that can performed on each individual record in the dataset. The schema defines the name as well as the type of data in each column. The partitioning of the DataFrame defines the layout of the DataFrame or Dataset’s physical distribution across the cluster. The partitioning scheme defines how that is broken up, this can be
Chapter 3. Basic Structured Operations
Chapter Overview
Chapter 4. Working with Different Types of Data Chapter Overview In the previous chapter, we covered basic DataFrame concepts and abstractions. this chapter will cover building expressions, which are the bread and butter of Spark’s structured operations. This chapter will cover working with a variety of different kinds of data including: Booleans Numbers Strings Dates and Timestamps Handling Null Complex Types User Defined Functions Where to Look for APIs Before we get started, it’s worth explaining where you as a user should start looking for transformations. Spark is a growing project and any book (including this one) is a snapshot in time. Therefore it is our priority to educate you as a user as to where you should look for functions in order to transform your data. The key places to look for transformations are: DataFrame (Dataset) Methods. This is actually a bit of a trick because a DataFrame is just a Dataset of Row types so you’ll actually end up looking at the Dataset methods.
Chapter 4. Working with Different Types of Data
Chapter Overview
Chapter 5. Aggregations What are aggregations? Aggregating is the act of collecting something together and is a cornerstone of big data analytics. In an aggregation you will specify a key or grouping and an aggregation function that specifies how you should transform one or more columns. This function must produce one result for each group given multiple input values. Spark’s aggregation capabilities sophisticated and mature, with a variety of different use cases and possibilities. In general, we use aggregations to summarize numerical data usually by means of some grouping. This might be a summation, a product, or simple counting. Spark also allows us aggregate any kind of value into an array, list or map as we will see in the complex types part of this chapter. In addition to working with any types of values, Spark also allows us to create a variety of different groupings types. The simplest grouping is to just summarize a complete DataFrame by performing an aggregation in a select s
Chapter 5. Aggregations
Chapter 6. Joins Joins will be an essential part of your Spark workloads. Spark’s ability to talk to a variety of data sources means your ability to tap into a variety of data sources across your company. What is a join? Join Expressions A join brings together two sets of data, the left and the right, by comparing the value of one or more keys of the left and right and evaluating the result of a join expression that determines whether or not Spark should join the left set of data with the right set of data on that given row. The most common join expression is that of an equi-join, where we compare whether or not the keys are equal, however there are other join expressions that we can specify like whether or not a value is greater than or equal to another value. Join expressions can be a variety of different things, we can even leverage complex types and perform something like checking whether or not key exists inside of an array. Join Types While the join expressions determines whether
Chapter 6. Joins
Chapter 7. Data Sources The Data Source APIs One of the reasons for Spark’s immense popularity is its ability to read and write to a variety of data sources. Thus far in this book we read data in CSV and JSON file format. This chapter will formally introduce the variety of other data sources that you can use with Spark. Spark has six “core” data sources and hundreds of external data sources written by the community. Spark’s core Data Sources are: CSV, JSON, Parquet, ORC, JDBC/ODBC Connections, and plain text files. As mentioned Spark has numerous community-created data sources including: Cassandra HBase MongoDB AWS Redshift and many others. This chapter will not cover writing your own data sources but rather the core concepts that you will need to work with any of the above data sources. After introducing the the core concepts, we will move onto demonstrations of each of Spark’s core data sources. Basics of Reading Data The foundation for reading data in Spark is the DataFrameReader. W
Chapter 7. Data Sources
Chapter 8. Spark SQL Spark SQL Concepts Spark SQL is arguably one of the most important and powerful concepts in Spark. This chapter will introduce the core concepts in Spark SQL that you need to understand. This chapter will not rewrite the ANSI-SQL specification or enumerate every single kind of SQL expression. If you read any other parts of this book, you will notice that we try to include SQL code wherever we include DataFrame code to make it easy to cross reference with code examples. Other examples are available in the appendix and reference sections. In a nutshell, Spark SQL allows the user to execute SQL queries against views or tables organized into databases. Users can also use system functions or define user functions and analyze query plans in order to optimize their workloads. What is SQL? SQL or Structured Query Language is a domain specific language for expressing relational operations over data. It is used in all relational databases and many “NoSQL” databases create th
Chapter 8. Spark SQL
Chapter 9. Datasets What are Datasets? Datasets are the foundational type of the Structured APIs. Earlier in this section we worked with DataFrames, which are Datasets of Type Row, and are available across Spark’s different languages. Datasets are a strictly JVM language feature that only work with Scala and Java. Datasets allow you to define the object that each row in your Dataset will consist of. In Scala this will be a case class object that essentially defines a schema that you can leverage and in Java you will define a Java Bean. Experienced users often refer to Datasets as the “typed set of APIs” in Spark. See the Structured API Overview Chapter for more information. In the introduction to the Structured APIs we discussed that Spark has types like StringType, BigIntType, StructType and so on. Those Spark specific types map to types available in each of Spark’s languages like String, Integer, Double. When you use the DataFrame API, you do not create Strings or Integers but Spark
Chapter 9. Datasets
Chapter 10. Low Level API Overview The Low Level APIs In the previous section we presented Spark’s Structured APIs which are what most users should be using regularly to manipulate their data. There are times where this high level manipulation will not fit the business or engineering problem you are trying to solve. In those cases you may need to use Spark’s lower level APIs specifically the Resilient Distributed Dataset (RDD), the SparkContext, and shared variables like accumulators and broadcast variables. These lower level APIs should be used for two core reasons: If you need some functionality that you cannot find in the higher level APIs. For the most part this case should be the exception. You need to maintain some legacy codebase that runs on RDDs. While those are the reasons you should use these lower level tools, it is well worth understanding these tools because all Spark workloads compile now to this fundamental primitives. When you’re calling a DataFrame transformation - it
Chapter 10. Low Level API Overview
Chapter 11. Basic RDD Operations RDD Overview Resilient Distributed Datasets (RDDs) are Spark’s oldest and lowest level abstraction made available to users. They were the primary API in the 1.X Series and are still available in 2.X, but are not commonly used by end users. An important fact to note, however, is that virtually all Spark code you run, where DataFrames or Datasets, compiles down to an RDD. The Spark UI, mentioned in later chapters, also describes things in terms of RDDs and therefore it behooves users to have at least a basic understanding of what an RDD is and how to use it. While many users forego RDDs because virtually all functionality they provide is available in Datasets and RDDs, users can still use RDDs if they are handling legacy code. A Resilient Distributed Dataset (RDD), the basic abstraction in Spark, represents an immutable, partitioned collection of elements that can be operated on in parallel. RDDs give the user complete control because every row in an RDD
Chapter 11. Basic RDD Operations
Chapter 12. Advanced RDDs Operations The previous chapter explored RDDs, which are Spark’s most stable API. This chapter will include relevant examples and point to the documentation for others. There is a wealth of information available about RDDs across the web and because the APIs have not changed for years, we will focus on the core concepts as opposed to just API examples. Advanced RDD operations revolve around three main concepts: Advanced single RDD Partition Level Operations Aggregations and Key-Value RDDs Custom Partitioning RDD Joins Advanced “Single RDD” Operations Pipe RDDs to System Commands The pipe method is probably one of the more interesting methods that Spark has. It allows you to return an RDD created by piping elements to a forked external process. The resulting RDD is computed by executing the given process once per partition. All elements of each input partition are written to a process’s stdin as lines of input separated by a newline. The resulting partition con
Chapter 12. Advanced RDDs Operations
Chapter 13. Distributed Variables Chapter Overview Spark, in addition to the RDD interface, maintains two level level variable types that you can leverage to make your processing more efficient. These are broadcast variables and accumulator variables. These variables serve two opposite purposes. Broadcast Variables Broadcast variables intend to share an immutable value efficiently around the cluster. This might be to share some immutable value and use it around the cluster without having to serialize it in a function to every node. We demonstrate this tool in the following figure. Now you might Broadcast variables are shared, immutable variables that is cached on every machine in the cluster instead of serialized with every single task. A use case might be a look up table accessed by an RDD. Serializing this lookup table with every task is wasteful because the driver must perform all of this work. You can achieve the same result with a broadcast variable. For example, let’s imagine tha
Chapter 13. Distributed Variables
Chapter Overview
Chapter 14. Advanced Analytics and Machine Learning Spark is an incredible tool for a variety of different use cases. Beyond large scale SQL analysis and Streaming, Spark also provides mature support for large scale machine learning and graph analysis. This sort of computation is what is commonly referred to as “advanced analytics”. This part of the book will focus on how you can use Spark to perform advanced analytics, from linear regression, to connected components graph analysis, and deep learning. Before covering those topics, we should define advanced analytics more formally. Gartner defines advanced analytics as follows: Advanced Analytics is the autonomous or semi-autonomous examination of data or content using sophisticated techniques and tools, typically beyond those of traditional business intelligence (BI), to discover deeper insights, make predictions, or generate recommendations. Advanced analytic techniques include those such as data/text mining, machine learning, pattern
Chapter 14. Advanced Analytics and Machine Learning
Chapter 15. Preprocessing and Feature Engineering Any data scientist worth her salt knows that one of the biggest challenges in advanced analytics is preprocessing. Not because it’s particularly complicated work, it just requires deep knowledge of the data you are working with and an understanding of what your model needs in order to successfully leverage this data. This chapter will cover the details of how you can use Spark to perform preprocessing and feature engineering. We will walk through the core requirements that you’re going to need to meet in order to train an MLlib model in terms of how your data is structured. We will then walk through the different tools Spark has to perform this kind of work. Formatting your models according to your use case To preprocess data for Spark’s different advanced analytics tools, you must consider your end objective. In the case of classification and regression, you want to get your data into a column of type Double to represent the label and
Chapter 15. Preprocessing and Feature Engineering
Chapter 16. Preprocessing Any data scientist worth her salt knows that one of the biggest challenges in advanced analytics is preprocessing. Not because it’s particularly complicated work, it just requires deep knowledge of the data you are working with and an understanding of what your model needs in order to successfully leverage this data. Formatting your models according to your use case To preprocess data for Spark’s different advanced analytics tools, you must consider your end objective. In the case of classification and regression, you want to get your data into a column of type Double to represent the label and a column of type Vector (either dense or sparse) to represent the features. In the case of recommendation, you want to get your data into a column of users, a column of targets (say movies or books), and a column of ratings. In the case of unsupervised learning, a column of type Vector (either dense or sparse) to represent the features. In the case of graph analytics, y
Chapter 16. Preprocessing
Chapter 17. Classification Classification is the task of predicting a label, category, class or qualitative variable given some input features. The simplest case is binary classification, where there are only two labels that you hope to predict. A typical example is fraud analytics, a given transaction can be fraudalent or not; or email spam, a given email can be spam or not spam. Beyond binary classification lies multiclass classification where one label is chosen from more than two distinct labels that can be produced. A typical example would be Facebook predicting the people in a given photo or a meterologist predicting the weather (rainy, sunny, cloudy, etc.). Finally, there is multilabel classification where a given input can produce multiple labels. For example you might want to predict weight and height from some lifestyle observations like athletic activities. Like our other advanced analytics chapters, this one cannot teach you the mathematical underpinnings of every model. Se
Chapter 17. Classification
Chapter 18. Regression Regression is the task of predicting quantitative values from a given set of features. This obviously differs from classification where the outputs are qualitative. A typical example might be predicting the value of a stock after a set amount of time or the temperature on a given day. This is a more difficult task that classification because there are infinite possible outputs. Like our other advanced analytics chapters, this one cannot teach you the mathematical underpinnings of every model. See chapter three in ISL and ESL for a review of regression. Now that we reviewed regression, it’s time to review the model scalability of each model. For the most part this should seem similar to the classification chapter, as there is significant overlap between the available models. This is as of Spark 2.2. Model Number Features Training Examples Linear Regression 1 to 10 million no limit Generalized Linear Regression 4096 no limit Isotonic Regression NA millions Decision
Chapter 18. Regression
Chapter 19. Recommendation Recommendation is, thus far, one of the best use cases for big data. At their core, recommendation algorithms are powerful tools to connect users with content. Amazon uses recommendation algorithms to recommend items to purchase, Google websites to visit, and Netflix movies to watch. There are many use cases for recommendation algorithms and in the big data space, Spark is the tool of choice used across a variety of companies in production. In fact, Netflix uses Spark as one of the core engines for making recommendations. To learn more about this use case you can see the talk by DB Tsai, a Spark Committer from Netflix at Spark Summit - https://spark-summit.org/east-2017/events/netflixs-recommendation-ml-pipeline-using-apache-spark/ Currently in Spark, there is one recommendation workhorse algorithm, Alternating Least Squares (ALS). This algorithm leverages a technique called collaborative filtering where large amounts of data are collected on user activity or
Chapter 19. Recommendation
Chapter 20. Clustering In addition to supervised learning, Spark includes a number of tools for performing unsupervised learning and in particular, clustering. The clustering methods in MLlib are not cutting edge but they are fundamental approaches found in industry. As things like deep learning in Spark mature, we are sure that more unsupervised models will pop up in Spark’s MLlib. Cluster is a bit different form supervised learning because it is not as straightforward to recommend scaling parameters. For instance, when clustering in high dimensional spaces, you are quite likely to overfit. Therefore in the following table we include both computational limits as well as a set of statistical recommendations. These are purely rules of thumb and should be helpful guides, not necessary strict requirements. Model Statistical Recommendation Computation Limits Training Examples K-means 50 to 100 maximum Features x clusters < 10 million no limit Bisecting K-means 50 to 100 maximum Features x
Chapter 20. Clustering
Chapter 21. Graph Analysis Graphs are an intuitive and natural way of describing relationships between objects. In the context of graphs, nodes or vertices are the units while edges define the relationships between those nodes. The process of graph analysis is the process of analyzing these relationships. An example analysis might be your friend group, in the context of graph analysis each vertex or node would represent a person and each edge would represent a relationship. You can see the above image is a representation of a directed graph where the edges are directional. There are also undirected graphs in which there is no start and beginning for given edges. Using our example, the length of the edge might represent the intimacy between different friends; acquaintances would have long edges between them while married individuals would have extremely short edges. We could infer this by looking at communication frequency between nodes and weighting the edges accordingly. Graph are a n
Chapter 21. Graph Analysis
Chapter 22. Deep Learning In order to define deep learning, we must first define neural networks. Neural networks allow computers to understand concepts by layering simple representations on top of one another. For the most part, each one of these representations, or layers, consist of a variety of inputs connected together that are activated when combined together, similar in concept to a neuron in the brain. Our goal is to train the network to associate certain inputs with certain outputs. Deep learning, or deep neural networks, just combine many of these layers together in various different architectures. Deep learning has gone through several periods of fading and resurgence and has only recently become popular in the past decade because of its ability to solve an incredibly diverse set of complex problems. Spark being a robust tool for performing operations in parallel has a number of good opportunities for end users to leverage both Spark and deep learning together. warning if yo
Chapter 22. Deep Learning