www.it-ebooks.info www.it-ebooks.info Hadoop MapReduce v2 Cookbook Second Edition www.it-ebooks.info Table of Contents Hadoop MapReduce v2 Cookbook Second Edition Credits About the Author Acknowledgments About the Author About the Reviewers www.PacktPub.com Support files, eBooks, discount offers, and more Why Subscribe? Free Access for Packt account holders Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Errata Piracy Questions Getting Started with Hadoop v2 Introduction Hadoop Distributed File System – HDFS Hadoop YARN Hadoop MapReduce Hadoop installation modes Setting up Hadoop v2 on your local machine Getting ready www.it-ebooks.info How to do it… How it works… Writing a WordCount MapReduce application, bundling it, and running it using the Hadoop local mode Getting ready How to do it… How it works… There’s more… See also Adding a combiner step to the WordCount MapReduce program How to do it… How it works… There’s more… Setting up HDFS Getting ready How to do it… See also Setting up Hadoop YARN in a distributed cluster environment using Hadoop v2 Getting ready How to do it… How it works… See also Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution Getting ready How to do it… There’s more… HDFS command-line file operations Getting ready How to do it… How it works… There’s more… www.it-ebooks.info Running the WordCount program in a distributed cluster environment Getting ready How to do it… How it works… There’s more… Benchmarking HDFS using DFSIO Getting ready How to do it… How it works… There’s more… Benchmarking Hadoop MapReduce using TeraSort Getting ready How to do it… How it works… Cloud Deployments – Using Hadoop YARN on Cloud Environments Introduction Running Hadoop MapReduce v2 computations using Amazon Elastic MapReduce Getting ready How to do it… See also Saving money using Amazon EC2 Spot Instances to execute EMR job flows How to do it… There’s more… See also Executing a Pig script using EMR How to do it… There’s more… Starting a Pig interactive session Executing a Hive script using EMR How to do it… There’s more… www.it-ebooks.info Starting a Hive interactive session See also Creating an Amazon EMR job flow using the AWS Command Line Interface Getting ready How to do it… There’s more… See also Deploying an Apache HBase cluster on Amazon EC2 using EMR Getting ready How to do it… See also Using EMR bootstrap actions to configure VMs for the Amazon EMR jobs How to do it… There’s more… Using Apache Whirr to deploy an Apache Hadoop cluster in a cloud environment How to do it… How it works… See also Hadoop Essentials – Configurations, Unit Tests, and Other APIs Introduction Optimizing Hadoop YARN and MapReduce configurations for cluster deployments Getting ready How to do it… How it works… There’s more… Shared user Hadoop clusters – using Fair and Capacity schedulers How to do it… How it works… There’s more… Setting classpath precedence to user-provided JARs How to do it… www.it-ebooks.info How it works… Speculative execution of straggling tasks How to do it… There’s more… Unit testing Hadoop MapReduce applications using MRUnit Getting ready How to do it… See also Integration testing Hadoop MapReduce applications using MiniYarnCluster Getting ready How to do it… See also Adding a new DataNode Getting ready How to do it… There’s more… Rebalancing HDFS See also Decommissioning DataNodes How to do it… How it works… See also Using multiple disks/volumes and limiting HDFS disk usage How to do it… Setting the HDFS block size How to do it… There’s more… See also Setting the file replication factor How to do it… How it works… www.it-ebooks.info There’s more… See also Using the HDFS Java API How to do it… How it works… There’s more… Configuring the FileSystem object Retrieving the list of data blocks of a file Developing Complex Hadoop MapReduce Applications Introduction Choosing appropriate Hadoop data types How to do it… There’s more… See also Implementing a custom Hadoop Writable data type How to do it… How it works… There’s more… See also Implementing a custom Hadoop key type How to do it… How it works… See also Emitting data of different value types from a Mapper How to do it… How it works… There’s more… See also Choosing a suitable Hadoop InputFormat for your input data format How to do it… How it works… www.it-ebooks.info There’s more… See also Adding support for new input data formats – implementing a custom InputFormat How to do it… How it works… There’s more… See also Formatting the results of MapReduce computations – using Hadoop OutputFormats How to do it… How it works… There’s more… Writing multiple outputs from a MapReduce computation How to do it… How it works… Using multiple input data types and multiple Mapper implementations in a single MapReduce application See also Hadoop intermediate data partitioning How to do it… How it works… There’s more… TotalOrderPartitioner KeyFieldBasedPartitioner Secondary sorting – sorting Reduce input values How to do it… How it works… See also Broadcasting and distributing shared resources to tasks in a MapReduce job – Hadoop DistributedCache How to do it… How it works… There’s more… www.it-ebooks.info ...www.it-ebooks.info Hadoop MapReduce v2 Cookbook Second Edition www.it-ebooks.info Table of Contents Hadoop MapReduce v2 Cookbook Second Edition Credits About the Author Acknowledgments... Index www.it-ebooks.info www.it-ebooks.info Hadoop MapReduce v2 Cookbook Second Edition www.it-ebooks.info www.it-ebooks.info Hadoop MapReduce v2 Cookbook Second Edition Copyright © 2015 Packt Publishing... Questions Getting Started with Hadoop v2 Introduction Hadoop Distributed File System – HDFS Hadoop YARN Hadoop MapReduce Hadoop installation modes Setting up Hadoop v2 on your local machine Getting ready