Cyberscholarship High Performance Computing meets Digital Libraries

Manuscript of paper: Journal of Electronic Publishing, January 2008 In April 2007, the US National Science Foundation (NSF) and the British Joint Information Systems Committee (JISC) held an invitational workshop on datadriven science and data-driven scholarship, co-chaired by Ronald Larsen and William Arms, who jointly authored the final report The report used the term cyberscholarship to describe new forms of research that become possible when high performance computing meets digital libraries [1] Elsewhere in this issue of the Journal of Electronic Publishing, Ronald Larsen describes the workshop and its conclusions In this article, William Arms gives a personal view of the motivation behind the workshop and the roles of libraries and publishing in achieving its goals Cyberscholarship: High Performance Computing meets Digital Libraries William Y Arms November 27, 2007 In a recent seminar, Gregory Crane of Tufts University made the simple but profound statement, "When collections get large, only the computer reads every word" [2] Thomas Jefferson's library, which formed the nucleus of the Library of Congress, had 6,487 volumes – a large private collection in its day [3] Edward Gibbon's famous library catalog had 2,675 records (which he wrote on the back of playing cards) [4] Avid readers, like Jefferson and Gibbon, can read and digest thousands of books, but modern research libraries are so large that nobody can read everything in them The marriage of high performance computing with digital libraries introduces a completely new notion of scale High performance computing can bring together vast quantities of material – datasets, manuscripts, reports, etc – which might never make their way into a traditional library A scholar reads only a few hundreds of documents; a supercomputer can analyze millions Of course, a person has a rich understanding of what is being read while a computer works at a very superficial level, but profound research is possible by simple analysis of huge amounts of information Computer programs can identify latent patterns of information or relationships that will never be found by human searching and browsing This new form of research can be called cyberscholarship Cyberscholarship becomes possible only when there are digital libraries with extensive collections in a broad domain Such collections are large even by supercomputing standards For instance, the USC Shoah Foundation's video collection of 52,000 interviews with survivors of genocide, notably the Nazi holocaust, is about 400 terabytes [5] The Library of Congress's National Digital Information Infrastructure and Preservation Program (NDIIPP) supports collaborative projects to preserve digital content [6] There is no published estimate for the aggregate size of these projects, but a —1— Manuscript of paper: Journal of Electronic Publishing, January 2008 rough estimate is at least one petabyte or 1,000 terabytes Many research libraries have begun to digitize the books in their collections, working with organizations such as the Open Content Alliance, Google, or Microsoft These collections are becoming very large Digitized copies of the eight million books in the Cornell library alone is estimated to be four petabytes of data A plausible view of today's developments sees these digital collections as the future of research libraries Already, many scientists never enter a library building or read the printed copy of a journal Their reports, working papers, conference proceedings, datasets, images, audio, videos, software, and journal articles are created in digital formats and accessed via networks, and a large proportion of older content has been digitized With books available in digital formats – either from the publishers or by scanning older materials – scholars from other disciplines, including many social scientists and humanities scholars, will also be able to work entirely from digital materials Perhaps the university library will cease to be the largest building on campus and become the largest computing center Examples of cyberscholarship How will scholars use these huge collections? How can high-performance computing support novel research on large bodies of text, data, and other digital materials? New forms of research need new types of libraries To understand why, here are three examples from different domains The first looks at data-driven science, the second has a focus on information discovery from heterogeneous sources, and the third is our own work at Cornell on mining the history of the Web The National Virtual Observatory The National Virtual Observatory has been a pioneer in data-driven science [7] Its goal is to bring together previously disjoint sets of astronomical data, in particular digital sky surveys that have made observations at various wavelengths Important astronomical results that are not observable in a single dataset can be revealed by combined analysis of data from these different surveys The datasets are archived at many locations and therefore the National Virtual Observatory provides coordinated access to distributed data Because the analyses are complex and often computationally intensive, the virtual observatory expects researchers to select data extracts from the archives and download them to their own computers For this purpose, the virtual observatory has developed an applications programming interface (API), an XML encoding scheme for astronomical data, and a number of applications that work with the API Protocols are provided for specific tasks, such as a simple image access protocol The National Virtual Observatory has been funded by a major NSF grant to astronomers Alex Szalay and Roy Williams They worked closely with the late Jim Gray, a leading computer scientist who specialized in very large —2— Manuscript of paper: Journal of Electronic Publishing, January 2008 databases Entrez The Entrez system from the National Center for Biotechnology Information (NCBI) is another early example of cyberscholarship [8] NCBI is a division of the National Library of Medicine within the National Institutes of Health (NIH) Entrez provides a unified view of biomedical information from a wide variety of sources including the PubMed citations and abstracts, the Medical Subject Headings, full text of journal articles and books, databases such as the protein sequence database and Genbank, and computer programs such as the Basic Local Alignment Search Tool (BLAST) for comparing gene and protein sequences Entrez provides very powerful searching across the 23 databases in the system For example, somebody who is interested in a particular genetic sequence can extract every occurrence of that sequence from all the information sources Genetic sequences are particularly convenient for information discovery because they are encoded by four letters, A, C, G, and T, representing the four nucleotides of a DNA strand, but Entrez is more than a cross-domain search service It provides an applications programming interface (API), so that researchers can use their own computers to explore this information, and the Entrez Programming Utilities, which provide a structured interface to all the databases Thus researchers have a flexible set of tools to discover unexpected patterns that are buried in the datasets The Cornell Web Lab The final example is the Cornell Web Lab [9] For many years, a group of us at Cornell University have been interested in analyzing the Web collection of the Internet Archive [10] Approximately every two months since 1996, the Internet Archive, led by Brewster Kahle, has collected a snapshot of the Web and preserved it for future generations The collection has more than 110 billion Web pages, or about 1.9 petabytes of compressed data The Wayback Machine provides basic access to the collection To use it, a user submits a URL and receives a list of all the dates for which there are pages with that URL If the user clicks on a date, the page is returned in a form as close to the original as possible This is a wonderful system, but it only goes so far It is organized for human retrieval of single pages, not for computer programs that analyze millions Researchers want more flexibility They want to ask questions such as, "What foreign government sites link to www.law.cornell.edu?" or "As the Web grows, how has its structure changed?" or "What was the origin of the myth that the Internet was doubling every six months?" To answer such questions, researchers need many more ways to select pages than by URL and date, as provided by the Wayback Machine, or by textual queries, as provided —3— Manuscript of paper: Journal of Electronic Publishing, January 2008 by search engines such as Google They also need powerful tools to transform raw data into the form that is used for analysis A colleague whose research is on Web clustering algorithms mentioned that his team spends 90 percent of its effort obtaining test data, and 10 percent using that data for research Much of the preliminary effort is spent finding a suitable set of pages to study; the remaining effort is in preparing the data for analysis: removing duplicates, clarifying ambiguous URLs, extracting terms and links, matching links to URLs, reconciling broken links or missing data, and much more Since the test collections are very large, these preparatory tasks and the subsequent analysis require sophisticated programming and large amounts of computation The Web Lab's strategy to meet such needs is to copy of a large portion of the Internet Archive's Web collection to Cornell, mount it on a powerful computer system, organize it so that researchers have great flexibility in how they use it, and provide tools and services that minimize the effort required to use the data in research Implications for publishers and libraries If these three examples are indeed representative of a new form of scholarship, the implications for publishers and libraries are profound, but there is limited experience to guide the developments Nobody is sure what services will be needed or how they will be provided Often it is easier to state the challenges than to articulate the solutions Here are some of the lessons that we have learned, based primarily on our experience in developing the Web Lab Market research and incremental development The first step in building the Web Lab was market research In fall 2004, a group of Cornell students interviewed 15 faculty members and graduate students from social sciences and computer science From these interviews, we gained an appreciation of potential areas of research and some of the methodologies that might be used by different disciplines Computer scientists study the structure and evolution of the Web itself A series of classic experiments have explored the structure of the Web, but the experiments require so much data that many have never been repeated; at most they have been repeated a few times and rarely at different dates in the evolution of the Web Social scientists study the Web as a social phenomenon of interest in its own right and for the evidence it provides of current social events, such as the spread of urban legends and the development of legal concepts across time In one experiment, computer scientists with strong ties to sociology analyzed a billion items of data from LiveJournal, to see how the tendency of an individual to join a community is influenced by the number of friends within the community and by how those friends are connected to each other [11] —4— Manuscript of paper: Journal of Electronic Publishing, January 2008 Figure 1: The research dialog Market research gives general guidance, but the detailed requirements can only be developed incrementally The Web Lab's strategy is to build a few generic tools and services, work with researchers to discover what they need, and implement new services steadily over time Figure shows the dialog between researchers and computer scientists The dialog begins with the researchers, e.g., sociologists studying the diffusion of ideas They have general ideas of studies they would like to do, based on concepts that are rooted in their discipline Those of us who are building the lab may not know how to create the precise data structures, algorithms, and computing systems that they would like to have, but we can suggest alternative approaches These suggestions may stimulate the researchers to devise new methods Eventually, an experiment emerges that is interesting from the discipline point of view and computationally feasible The experiment is carried out, both parties learn more, and the dialog continues Organizational implications This incremental development requires a flexible organization and close collaboration between researchers, publishers, and librarians Each of the three examples – the National Virtual Observatory, Entrez, and the Web Lab – has a large and complex computer systems under the control of one organization In theory it is possible to provide comparable functionality in systems with distributed management, but, even —5— Manuscript of paper: Journal of Electronic Publishing, January 2008 when the data and services are distributed, the practical challenges of coordination are so great that it is wiser to build a centralized organization specializing in an area of cyberscholarship and serving all universities The NSF has recently announced a program to create several centers, known as Datanets, where the collection and preservation of scientific data is closely integrated with the research that will make use of it From an earlier generation of libraries, OCLC provides an example of a centralized organization that manages data on behalf of a large number of members Content Cyberscholarship requires that the data exists and is accessible One reason that the NCBI and the National Virtual Observatory were able to make rapid progress is that molecular biology and astronomy have traditions of early and complete publication of research data, and well-managed archives to look after it For the Web Lab, the data is already collected and preserved by an established organization, the Internet Archive; our role is to provide access for researchers In other fields, however, the information is widely scattered, or never published; experimental data is poorly archived or even discarded In a well-known example, scholars were denied access to large portions of the Dead Sea Scrolls from the 1950s, when they were discovered, until 1991 Once captured, access to the information in machine-readable formats is crucial All the information in a field must be available to the cyber scholar under reasonable terms and conditions Open access is ideal but not the only possibility Policy issues are closely tied to access Even though the information in the Web Lab was originally placed on the Web with open access, there are important policy issues in using it for research The copyright issues are well known and outside the scope of this article Privacy considerations may be more daunting Data mining can violate privacy by bringing together facts about an individual Universities have strict policies on experiments that involve human subjects and Cornell applies these policies to data mining of the Web, but the situation becomes complicated when a digital library provides services to many universities or to non-academic researchers Finally, there is the question of custodianship These large digital library collections are important historical records The custodial duty is to preserve both the content and the identifying metadata It would be highly improper to destroy inconvenient data, or to index it in such a manner that damaging information is never found Tools and services Libraries provide tools and services to make the collections useful to their patrons, e.g., catalogs, bibliographies, classification schemes, abstracting and indexing services, and knowledgeable reference librarians These services have evolved over many years, based on a collective view of how libraries are used Conversely, the services that are available have shaped the type of research that is carried out in libraries —6— Manuscript of paper: Journal of Electronic Publishing, January 2008 What tools and services are required for cyberscholarship? As the Web Lab develops, we continually see new needs An early request was for focused Web crawling, which is used to find Web pages fitting specified criteria The request came from work in educational digital libraries to automate the selection of materials for a library [12] Other requests came from a study of how opinions about a company change across time One hypothesis is that anchor text (the linked text in Web pages) reflects such opinions An informal description of the experiment is to begin with the Web in 2005 and extract the anchor text from all external links that point to the company's Web site The text is processed to eliminate navigational terms, such as "click here", or "back" The frequency with which each word appears is counted and the process is repeated with data from subsequent years Trends in the frequencies provide evidence about how the company is perceived This simple example uses several tools that process massive quantities of data The first is a way to select pages by attributes such as a domain name, format type, date, anchor text, and links to and from each page For this purpose, the Web Lab has created a relational database with tables for all pages, URLs, and links Currently it contains metadata of the pages from four complete crawls and is about 50 terabytes in size Other tools are used to extract a clean set of links from a collection of pages and remove duplicates This is a messy task: many of the URLs embedded in Web pages refer to pages that were never collected or never existed; URLs have many variations; because each snapshot of the Web was collected over a period of weeks or months a URL may refer to a page that changed before it was collected Some tools and services are specific to a field of research, but there are two requirements that apply to every domain The first is an application program interface (API), so that computer programs can interact directly with the collections Conventional digital libraries have been developed on the assumption that the user is a person Much effort is placed in the design of user interfaces, but the computer interface is usually given low priority In contrast, the three examples in this article all assume that the primary form of access will be via computer programs Researchers want computer programs to act as their agents, searching billions of items for patterns of information that are only vaguely guessed at The next requirement is to be able to identify and download sub-collections Often, the scale of the full collection inhibits complex analyzes Instead, the methodology is to select part of the collection and download it to another computer for analysis For this purpose, digital library collections must be designed so that programs can extract large parts of them For example, Wikipedia encourages researchers to download everything: the software, the content, the discussions, and all the historical trails Currently, it is about one terabyte, a size that can be analyzed on a smallish computer —7— Manuscript of paper: Journal of Electronic Publishing, January 2008 Using high-performance computing Very large collections require high-performance computing Few scholars would recognize the Cornell Web Lab as a library, but the computer system looks very familiar to the supercomputing community An early version is described in [13] Powerful computer systems at the Internet Archive in San Francisco and Cornell University in upstate New York are connected by research networks – Internet2 and the National LambdaRail – which have so much capacity that complete snapshots of the Web are routinely transferred across the country The Internet Archive operates very large clusters of Linux computers, while Cornell has several smaller computer clusters and a large database server dedicated to the Web Lab One of the most interesting challenges is to help researchers who are not computing specialists use these systems Until very recently, high-performance computing was a very specialized area Even today, experienced computing professional are often discouraged by the complexities of parallel programming and managing huge datasets Fortunately this situation is changing In the 1960s, the Fortran programming language first gave scientists a simple way to translate their mathematical problems into efficient computer codes Cyber scholars need a similarly simple way to express massive data analysis as programs that run efficiently on large computer clusters The map/reduce programming paradigm developed by the Lisp community and refined and made popular by Google appears to fulfill this need [14] There is an excellent, open-source software system called Hadoop that combines map/reduce programming with a distributed file system [15] In the Web Lab, we have used it for tasks such as full text indexing, the anchor text study described above, and extraction of the graph of the links between Web pages While we are still inexperienced in running large jobs, the programming required of the researchers is reassuringly straightforward Computational limits Modern computers are so powerful that it is easy to forget that they have limits, but techniques that are excellent for moderately large digital libraries fail with very large collections The Web Lab tries to isolate cyber scholars from the complex systems that are serving them, but they have to recognize that, with billions of records and hundreds of terabytes of data, apparently simple operations can run forever We need better ways to help researchers recognize these limits, design experiments that not exceed them, and off-load appropriate tasks to more powerful computers For instance, users of the Web Lab want full-text indexes of the collections that they are studying The Internet Archive and Cornell University have considerable computing resources, but not enough to index a hundred billion Web pages Fortunately, no researcher wants to analyze every page in the history of the entire Web In one set of —8— Manuscript of paper: Journal of Electronic Publishing, January 2008 experiments, researchers are analyzing a sub-collection of 30 million pages extracted from the amazon.com domain This collection would be considered large in other contexts, but there is little difficulty in indexing it on a medium sized computer cluster We are experimenting to see if we can use one of the new NSF supercomputing centers to index the entire collection, but meanwhile indexes of sub-collections are sufficient for most purposes Expertise The limiting factor in the development of cyberscholarship is likely to be a shortage of expertise The convergence of high performance computing and large scale digital libraries requires technical skills that few universities possess During the dot-com boom, universities lost most of their expertise to start-up companies In particular, the development of automatic indexing moved from universities such as Cornell, Carnegie Mellon, and Stanford to the private sector Skill in managing large collections of semistructured data, such as the Web, is concentrated in a few commercial organizations, such as Google, Yahoo, Amazon, and Microsoft, and a single not-for-profit, the Internet Archive Recently Google, IBM, and others have begun initiatives to encourage universities to educate more students in the computing methods of massive data analysis Hopefully, some of this expertise will remain at the universities and become part of the academic mainstream Opportunities for change The underlying theme of this paper is that there is a symbiosis between the organization of library collections and the types of research that they enable When large digital collections meet high performance computing, opportunities for new kinds of scholarship emerge, but these opportunities need new approaches to how collections are organized and how they are used In a thought-provoking article on the future of academic libraries, David Lewis writes, "Real change requires real change" [16] In universities and libraries some people are cautious in accepting change, while others welcome it Some individuals were early adopters of Web searching, but others would not accept that automatic indexing could be more effective than indexing by skilled individuals [17] Some scholars have been outspoken about the evils of Wikipedia [18], or have written articles that are critical of digitized books [19], while their colleagues are enthusiastic about the opportunities that new approaches offer Overall, the academic community has benefited from early adoption of new technology, e.g., campus networks, electronic mail, and the Web Online library catalogs, led by the Library of Congress and OCLC, were pioneering computing systems in their day —9— Manuscript of paper: Journal of Electronic Publishing, January 2008 Information companies, such as Cisco and Google, have their roots in university projects Cyberscholarship provides similar opportunities For the academic community it provides new methodologies for research For libraries and entrepreneurs the potential is almost unlimited Acknowledgements Parts of this article come from a position paper for the NSF/JISC workshop in Phoenix, Arizona, April 2007: http://www.sis.pitt.edu/~repwkshop/papers/arms.html The Cornell Web Lab is funded in part by National Science Foundation grants CNS0403340, SES-0537606, IIS 0634677, and IIS 0705774 Notes and references [1] William Y Arms and Ronald L Larsen (co-chairs) The future of scholarly communication: building the infrastructure for cyberscholarship NSF/JISC workshop, Phoenix, Arizona, April 17 to 19, 2007 http://www.sis.pitt.edu/~repwkshop/NSF-JISCreport.pdf [2] Gregory Crane What you with a million books? Information science colloquium, Cornell University, March 2006 [3] See: Jefferson's Legacy, A brief history of the Library of Congress Library of Congress Web site http://www.loc.gov/loc/legacy/loc.html [4] Gibbon's Library, New York Times, June 19, 1897 [5] USC Shoah Foundation Institute for Visual History and Education http://www.usc.edu/schools/college/vhi/ [6] The National Digital Information Infrastructure and Preservation Program (NDIIPP) is described on the Library of Congress's digital preservation Web site: http://www.digitalpreservation.gov/ [7] Alex Szalay and Jim Gray, The World-Wide Telescope, Science, 293, 2037, 2001 The National Virtual Observatory's home page is: http://www.us-vo.org/ [8] The Entrez homepage is: http://www.ncbi.nlm.nih.gov/Entrez The best description of Entrez is on Wikipedia (November 11, 2007): http://en.wikipedia.org/wiki/Entrez [9] William Arms, Selcuk Aya, Pavel Dmitriev, Blazej Kot, Ruth Mitchell, and Lucia Walle, A research library based on the historical collections of the Internet Archive DLib Magazine, 12 (2), February 2006 — 10 — Manuscript of paper: Journal of Electronic Publishing, January 2008 http://www.dlib.org/dlib/february06/arms/02arms.html [10] The Internet Archive's home page is: http://www.archive.org/ [11] Lars Backstrom, Dan Huttenlocher, Jon Kleinberg, and Xiangyang Lan, Group formation in large social networks: membership, growth, and evolution Twelfth Intl Conference on Knowledge Discovery and Data Mining, 2006 [12] Donna Bergmark, Collection synthesis ACM/IEEE Joint Conference on Digital Libraries, July 2002 [13] William Arms, Selcuk Aya, Pavel Dmitriev, Blazej Kot, Ruth Mitchell, and Lucia Walle, Building a research library for the history of the Web ACM/IEEE Joint Conference on Digital Libraries, 2006 [14] Jeffrey Dean and Sanjay Ghemawat, MapReduce: simplified data processing on large clusters Usenix SDI '04, 2004 http://www.usenix.org/events/osdi04/tech/full_papers/dean/dean.pdf [15] Hadoop is part of the Lucene-Apache suite of open-source software See: http://lucene.apache.org/hadoop/ [16] David W Lewis, A model for academic libraries 2005 to 2025, Visions of Change, California State University at Sacramento, January 26, 2007 [17] The letter by Weinheimer in D-Lib Magazine, September 2000, is a good example of judging the new by the standards of the old http://www.dlib.org/dlib/september00/09letters.html#Weinheimer [18] See, for example: Noam Cohen, A history department bans citing Wikipedia as a research source New York Times, February 21, 2007 [19] See, for example: Robert B Townsend, Google Books: Is it good for history? American Historical Association, September 12, 2007 http://www.historians.org/perspectives/issues/2007/0709/0709vie1.cfm — 11 — ... paper: Journal of Electronic Publishing, January 2008 Using high- performance computing Very large collections require high- performance computing Few scholars would recognize the Cornell Web Lab... collections? How can high- performance computing support novel research on large bodies of text, data, and other digital materials? New forms of research need new types of libraries To understand... help researchers who are not computing specialists use these systems Until very recently, high- performance computing was a very specialized area Even today, experienced computing professional are