istory Mining the Social Web opics THIRD EDITION utorials Matthew A. Russell and Mikhail Klassen ffers & Deals ighlights ettings Support Sign Out Playlists Mining the Social Web History by Matthew A. Russell and Mikhail Klassen Topics Copyright © 2019 Matthew Russell, Mikhail Klassen. All rights reserved Tutorials Printed in Canada Offers & Deals Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 Highlights O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, Settings contact our corporate/institutional sales department: 8009989938 or corporate@oreilly.com Support Sign Out Acquistions Editor: Mary Treseler Development Editor: Alicia Young Production Editor: Nan Barber Copyeditor: Rachel Head Proofreader: Kim Cofer Indexer: WordCo Indexing Services, Inc Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest December 2018: Third Edition Revision History for the Third Edition 20181129: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491985045 for release details The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Mining the Social Web, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights 9781491985045 [MBP] Preface Safari Home Recommended The Web is more a social creation than a technical one I designed it for a social effect—to help people work together—and not as a technical toy. The Playlists ultimate goal of the Web is to support and improve our weblike existence in the world. We clump into families, associations, and companies. We develop trust across the miles and History distrust around the corner Topics —Tim BernersLee, Weaving the Web (Harper) A Note from Matthew Russell Tutorials It’s been more than five years since I put the final finishing touches on the manuscript for Offers & Deals Mining the Social Web, 2nd Edition, and a lot has changed since then. I have lived and learned a lot of new things, technology has continued to evolve at a blistering pace, and the social web Highlights itself has matured to such an extent that governments are now formulating legal policy around how data can be collected, shared, and used Settings Knowing that my own schedule could not possibly allow for the immense commitment needed Support to produce a new edition to freshen up and expand on the content, but believing wholeheartedly Signthat there has never been a better moment for the message this book delivers, I knew that it was Out time to find a coauthor to help deliver it to the next wave of entrepreneurs, technologists, and hackers who are curious about mining the social web. It took well over a year for me to find a coauthor who shared the same passion for the subject and possessed the skill and determination that’s required to write a book I can’t even begin to tell you how grateful I am for Mikhail Klassen and his incredible contributions in keeping this labor of love alive for many more years to come. In the pages ahead, you’ll see that he’s done a tremendous job of modernizing the code, improving the accessibility of its runtime environment, and expanding the content with a substantial new chapter—all in addition to editing and freshening up the overall manuscript itself and enthusiastically carrying the mantle forward for the next wave of entrepreneurs, technologists, and hackers who are curious about mining the social web README.1st This book has been carefully designed to provide an incredible learning experience for a , particular target audience, and in order to avoid any unnecessary confus ion about its scope or purpose by way of disgruntl ed emails, bad book reviews, or other misunderstandings that can come up, the remainder of this preface tries to help you determine whether you are part of that target audience. As busy professionals, we consider our time our most valuable asset, and we want you to know right from the beginning that we believe that the same is true of you Although we often fail, we really do try to honor our neighbors above ourselves as we walk out this life, and this preface is our attempt to honor you, the reader, by making it clear whether or not this book can meet your expectations Managing Your Expectations Some of the most basic assumptions this book makes about you as a reader are that you want to learn how to mine data from popular social web properties, avoid technology hassles when running sample code, and have lots of fun along the way. Although you could read this book solely for the purpose of learning what is possible, you should know up front that it has been written in such a way that you really could follow along with the many exercises and become a data miner once you’ve completed the few simple steps to set up a development environment. If you’ve done some programming before, you should find that it’s relatively painless to get up and running with the code examples. Even if you’ve never programmed before, if you consider yourself the least bit techsavvy I daresay that you could use this book as a starting point to a remarkable journey that will stretch your mind in ways that you probably haven’t even imagined yet To fully enjoy this book and all that it has to offer, you need to be interested in the vast possibilities for mining the rich data tucked away in popular social websites such as Twitter, Facebook, LinkedIn, and Instagram, and you need to be motivated enough to install Docker, use it to run this book’s virtual machine experience, and follow along with the book’s example code in the Jupyter Notebook, a fantastic webbased tool that features all of the examples for every chapter. Executing the examples is usually as easy as pressing a few keys, since all of the code is presented to you in a friendly user interface This book will teach you a few things that you’ll be thankful to learn and will add a few indispensable tools to your toolbox, but perhaps even more importantly, it will tell you a story and entertain you along the way. It’s a story about data science involving social websites, the data that’s tucked away inside of them, and some of the intriguing possibilities of what you (or anyone else) could do with this data If you were to read this book from cover to cover, you’d notice that this story unfolds on a chapterbychapter basis. While each chapter roughly follows a predictable template that introduces a social website, teaches you how to use its API to fetch data, and presents some techniques for data analysis, the broader story the book tells crescendos in complexity. Earlier chapters in the book take a little more time to introduce fundamental concepts, while later chapters systematically build upon the foundation from earlier chapters and gradually introduce a broad array of tools and techniques for mining the social web that you can take with you into other aspects of your life as a data scientist, analyst, visionary thinker, or curious reader Some of the most popular social websites have transitioned from fad to mainstream to household names over recent years, changing the way we live our lives on and off the web and enabling technology to bring out the best (and sometimes the worst) in us. Generally speaking, each chapter of this book interlaces slivers of the social web along with data mining, analysis, and visualization techniques to explore data and answer the following representative questions: Who knows whom, and which people are common to their social networks? How frequently are particular people communicating with one another? Which social network connections generate the most value for a particular niche? How does geography affect your social connections in an online world? Who are the most influential/popular people in a social network? What are people chatting about (and is it valuable)? What are people interested in based upon the human language that they use in a digital world? The answers to these basic kinds of questions often yield valuable insights and present (sometimes lucrative) opportunities for entrepreneurs, social scientists, and other curious practitioners who are trying to understand a problem space and find solutions. Activities such as building a turnkey killer app from scratch to answer these questions, venturing far beyond the typical usage of visualization libraries, and constructing just about anything stateoftheart are not within the scope of this book. You’ll be really disappointed if you purchase this book because you want to do one of those things. However, the book does provide the fundamental building blocks to answer these questions and provide a springboard that might be exactly what you need to build that killer app or conduct that research study. Skim a few chapters and see for yourself. This book covers a lot of ground One important thing to note is that APIs are constantly changing. Social media hasn’t been around all that long, and even the platforms that appear the most established today are still adapting to how people use them and confronting new threats to security and privacy. As such, the interfaces between our code and their platforms (the APIs) are liable to change too, which means that the code examples provided in this book may not work as intended in the future We’ve tried to create realistic examples that are useful for general purposes and app developers, and therefore some of them will require submitting an application for review and approval We’ll do our best to flag those with notes, but be advised API terms of service can change at any time. Nevertheless, as long as your app abides by the terms of service, it will likely get approved, so it’s worth the effort Python-Centric Technology This book intentionally takes advantage of the Python programming language for all of its example code. Python’s intuitive syntax, amazing ecosystem of packages that trivialize API access and data manipulation, and core data structures that are practically JSON make it an excellent teaching tool that’s powerful yet also very easy to get up and running. As if that weren’t enough to make Python both a great pedagogical choice and a very pragmatic choice for mining the social web, there’s the Jupyter Notebook, a powerful, interactive code interpreter that provides a notebooklike user experience from within your web browser and combines code execution, code output, text, mathematical typesetting, plots, and more. It’s difficult to imagine a better user experience for a learning environment, because it trivializes the problem of delivering sample code that you as the reader can follow along with and execute with no hassles Figure P1 provides an illustration of the Jupyter Notebook experience, demonstrating the dashboard of notebooks for each chapter of the book. Figure P2 shows a view of one notebook Figure P1. Overview of the Jupyter Notebook; a dashboard of notebooks Figure P2. The “Chapter 1 Mining Twitter” notebook Every chapter in this book has a corresponding Jupyter Notebook with example code that makes it a pleasure to study the code, tinker around with it, and customize it for your own purposes. If you’ve done some programming but have never seen Python syntax, skimming ahead a few pages should hopefully be all the confirmation that you need. Excellent documentation is available online, and the official Python tutorial is a good place to start if you’re looking for a solid introduction to Python as a programming language. This book’s Python source code has been overhauled for the third edition to be written in Python 3.6 The Jupyter Notebook is great, but if you’re new to the Python programming world, advising you to just follow the instructions online to configure your development environment would be a bit counterproductive (and possibly even rude). To make your experience with this book as enjoyable as possible, a turnkey virtual machine is available that has the Jupyter Notebook and all of the other dependencies that you’ll need to follow along with the examples from this book preinstalled and ready to go. All that you have to do is follow a few simple steps, and in about 15 minutes, you’ll be off to the races. If you have a programming background, you’ll be able to configure your own development environment, but our hope is that we’ll convince you that the virtual machine experience is a better starting point NOTE See Appendix A for more detailed information on the virtual machine experience for this book. Appendix C is also worth your attention: it presents some Jupyter Notebook tips and common Python programming idioms that are used throughout this book’s source code Whether you’re a Python novice or a guru, the book’s latest bugfixed source code and accompanying scripts for building the virtual machine are available on GitHub, a social Git repository that will always reflect the most uptodate example code available. The hope is that social coding will enhance collaboration between likeminded folks who want to work together to extend the examples and hack away at fascinating problems. Hopefully, you’ll fork, extend, and improve the source—and maybe even make some new friends or acquaintances along the way NOTE The official GitHub repository containing the latest and greatest bugfixed source code for this book is available at http://bit.ly/MiningtheSocialWeb3E Improvements to the Third Edition As mentioned earlier in this preface, the third edition of this book has brought on Mikhail Klassen as a coauthor Technology changes fast, and social media platforms along with it. When we began revising the second edition, it became clear that the book would benefit from an update to reflect all the changes that were taking place. The first and most obvious change was to update the code from Python 2.7 to a more recent version of Python 3.0+. While there are still diehard users of Python 2.7, moving to Python 3 has a lot of advantages, not the least of which is better support for Unicode. When dealing with social media data, which often includes emojis and text from other alphabets, having good support for Unicode is critical In a climate of increasing concerns over user privacy, social media platforms are changing their APIs to better safeguard user information by limiting the extent to which thirdparty applications can access their platforms—even applications that have been vetted and approved ce threshold # for any two consecutive words clusters = [] cluster = [word_idx[0]] i = 1 while i