uncharted_ big data as a lens on human culture-erez aiden

[...]... has no syntax and no sentences It is a 345-page-long list of words in alphabetical order It does not look or read like a novel In fact, when you read it, it appears to be complete nonsense We rarely read romance novels, but Reimer’s work is an exception An absolute pageturner, it fascinated us from cover to cover, from the dramatic beginning: Chapter One A AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAA... errors, and marred by numerous, frustrating gaps: missing pieces of information that any reasonable scientist would want to know These errors and omissions are often inconsistent, even within what is thought of as a single dataset That’s because big datasets are frequently created by aggregating a vast number of smaller datasets Invariably, some of these component datasets are more reliable than others, and... goes back for centuries So Google Books isn’t just big data, it’s long data Since they contain such long data, digitized books aren’t limited to painting a picture of contemporary humanity, as most big datasets are Books can also offer a portrait of how our civilization has changed over fairly long periods of time—longer than the length of a human life, longer even than the lifetimes of whole nations... years, as data storage technology improves, bandwidth increases, and our lives gradually migrate onto the Internet Big data just gets bigger and bigger and bigger THE DIGITAL LENS Arguably the most crucial difference between the cultural records of today and those of years gone by is that today’s big data exists in digital form Like an optic lens, which makes it possible to reliably transform and manipulate... and a strange, fascinating, and addictive approach to language, culture, and history that we call culturomics We’ll describe all sorts of observations that can be made using a culturomic approach We’ll talk about what our ngram data has revealed about how English grammar changes, how dictionaries make mistakes, how people get famous, how governments suppress ideas, how societies learn and forget, and... Stone Age ancestors of loose change—was inconvenient Instead, it was easier to take one really big stone and use the stylus to engrave lots of patterns on it, side by side Writing was born In retrospect, it might seem surprising that something as mundane as the desire to count sheep was the impetus for an advance as fundamental as written language But the desire for written records has always accompanied... intimate can you possibly be with a petabyte? A second major challenge is that big data doesn’t fit too well into what we typically think of as the scientific method Scientists like to confirm specific hypotheses, and to gradually assemble what they’ve learned into causal stories and eventually mathematical theories Blunder about in any reasonably interesting big dataset and you will inevitably make... finally let us graduate, which is about all you can hope for when you’re as underfed, underpaid, and overeducated as the typical PhD seeker As we were mulling this somewhat esoteric question, a revolution was occurring elsewhere that would sweep us up in its wake and lead millions of people to share our strange fascination At its core, this big data revolution is about how humans create and preserve a. .. transactions to become public information or to be made available to random grad students Search engine logs and e-mails are entitled to privacy and confidentiality Authors of books and blogs are protected by copyright And companies have strong proprietary interests in the data they control They may analyze their data with a view toward generating more ad revenue, but they are loath to share the heart... have a fascinating environment We met at Harvard’s Program for Evolutionary Dynamics, a haven of creativity and science founded by the charismatic mathematician and biologist Martin Nowak The PED (Program for Evolutionary Dynamics? Program for rEvolutionary Dynamics? Party Every Day?) is a place where mathematicians, linguists, cancer researchers, religious scholars, psychologists, and physicists congregate, . two years, as data storage technology improves, bandwidth increases, and our lives gradually migrate onto the Internet. Big data just gets bigger and bigger and bigger. THE DIGITAL LENS Arguably the. digitized books aren’t limited to painting a picture of contemporary humanity, as most big datasets are. Books can also offer a portrait of how our civilization has changed over fairly long periods. time—longer than the length of a human life, longer even than the lifetimes of whole nations. Books are a fascinating dataset for other reasons, too. They cover an extraordinary range of topics and