If You Can''''t Be With the Data You Love And the Risks of Loving the Data You''''re With If You Can''''t Be With the Data You Love And the Risks of Loving the Data You''''re With Po licy R ep o rt Neeta Fogg Pau[.]
Policy Report If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Neeta Fogg Paul Harrington Ishwar Khatiwada Irwin Kirsch Anita Sands Larry Hanover Center for Labor Markets and Policy, Drexel University ETS THE ETS CENTER FOR RESEARCH ON HUMAN CAPITAL AND EDUCATION Table of Contents Table of Contents Preface Acknowledgments .4 Introduction A Word about the Data .7 Divergent Measures: Educational Attainment vs Essential Skills Essential Skills Deficits of the Working-Age Population 11 Proxy Measures 11 Measuring and Comparing Skills Deficits Using PIAAC Data 11 The Interrelationship between PIAAC Literacy Scores and Educational Credentials 13 The Interrelationship between PIAAC Literacy Skills and Low Wages .18 What is the Utility of the Essential Skills Proxy Measure? 24 Implications 27 Appendix 28 About the Authors 29 This report was written by: Neeta Fogg Paul Harrington Ishwar Khatiwada Irwin Kirsch Anita Sands Larry Hanover The views expressed in this report are those of the authors and not necessarily reflect the views of the officers and trustees of Educational Testing Service Copyright © 2019 by Educational Testing Service All rights reserved ETS, the ETS logo, and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS) All other trademarks are the property of their respective owners September 2019 ETS Center for Research on Human Capital and Education Research and Development Educational Testing Service Rosedale Road Princeton, NJ 08541-0001 If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Preface Preface Policymakers and others in the United States, driven by a reasonable belief that when individuals have more education they will be better prepared for today's economy and have a better chance at favorable life outcomes, have recently redoubled efforts to improve rates of attaining credentials On the surface, these efforts appear to be paying off Today, a greater proportion of high school students are graduating on time than ever before, and among those who not, most end up with a high school degree by their early 20s What's more, the data show that the millennial generation has obtained the highest levels of postsecondary education of any previous generation in U.S history However, as Shakespeare once observed, "All that glitters is not gold." Mounting evidence, including this new paper, suggest that data on educational attainment and results from an array of skills assessments are telling divergent stories In fact, recent research that examines the literacy and numeracy skills of America's millennial generation finds high levels of skills deficits What's more, according to research on trends in fourth- and eighth-grade scores in the National Assessment of Educational Progress (NAEP; better known as the "Nation's Report Card"),[I] the pattern of increasing scores and decreasing achievement gaps among race-ethnicity groups, evident in the first few years of the 2000s, has leveled off Data on education in the United States has been formally collected since the late 19th century when Congress first passed legislation establishing a department to monitor the nation's educational progress Of course, what constitutes being educated—and closely related, the way to measure progress toward that goal—has varied over time From its beginnings until the middle of the 20th century, information on education inputs—such as numbers of teachers, schools, and students, as well as enrollment and attainment data—were used to understand the nation's educational progress But, with critical increases in the need for highly skilled workers rising rapidly toward the second half of the 20th century, especially after America's Sputnik moment—when the Soviet Union overtook our nation in the space race and concerns began to mount that public education wasn't meeting demands—a move toward including measures of skill proficiency, or outputs, was added.[II] While perhaps an oversimplification, one could say skills data emphasized the quality of educational progress, while attainment and enrollment data emphasized the quantity However, as we advance deeper into the 21st century, two troubling and paradoxical shifts appear to be emerging First, at a time when higher levels of skills are essential for the longterm success of individuals—and the nation as a whole—we appear to be losing ground despite rising enrollment and attainment Second, and in some ways more perplexing, we are seeing a move away from seeking to ensure that educational attainment comes with skills In this paper, the authors highlight the critical disadvantages of focusing on educational attainment as the sole or primary marker of productive ability They this by comparing two estimates of employed adults in the United States believed to have foundational skill deficits—one relies on educational attainment data as a proxy for skill deficiencies, while the other utilizes skills data from a large-scale assessment of adults The underlying theory behind using educational attainment as a proxy for skills is, quite reasonably, that the more education you have, the less likely, on average, you are to have a skills deficit With this estimate, for example, those who dropped out of high school would be classified as deficient, while those with a bachelor's degree would be assumed proficient The second estimate, on the other hand, is based on actual skills information from the Programme for the International Assessment of Adult Competencies (PIAAC), an international large-scale If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Preface assessment of adult skills in literacy, numeracy, and problem solving in technology-rich environments For this approach, lack of skills is determined by performance on a set of literacy and numeracy tasks, regardless of education level attained Through their analysis of attainment and skills data, the authors reveal that using educational attainment as a proxy for skill deficits results in large errors in terms of both the size and composition of those with low skills These findings suggest extreme caution when using educational attainment as the only or primary measure of the productive ability of American students and adults Indicators such as educational attainment are created, adopted, and used because they are believed to measure what we care about What's more, they provide a framework that guides our thinking, policies, and actions toward achieving important national, state, and local goals The waning ability of educational attainment to accurately measure progress on what we care about—the population having essential knowledge and skills, as shown in this and other research—has vast consequences felt by those without the skills they need despite a degree that signals otherwise It is lived by those who take initiative and pursue higher education but either aren't equipped to succeed or struggle in programs that little to foster skill development, although they acquire debt It also raises critical questions about the desirability of public policies that focus on educational credentials without acknowledging the levels of skills that are associated with those credentials Allowing students to attain degrees and diplomas with low levels of literacy skills, particularly as those skills become even more critical for successful navigation of our complex world, diminishes the fundamental promise of education —Paul Harrington and Irwin Kirsch References "Graduation Test Update: States that Recently Eliminated or Scaled Back High School Exit Exams" (Updated May 2019), National Center for Fair and Open Testing, http://www.fairtest.org/graduation-test-update-states-recently-eliminated Madeline Goodman and Anita Sands, Too Big to Fail: Millennials on the Margins (Princeton, NJ: Educational Testing Service, 2018), https://www.ets.org/research/report/opportunity-too-bigto-fail/millennial Madeline J Goodman, Anita M Sands, and Richard J Coley, America's Skills Challenge (Princeton, NJ: Educational Testing Service), https://www.ets.org/s/research/29836/ Susanna Loeb, "Low-Performing Students No Longer Making Gains on NAEP," Education Next, April 10, 2018 Notes I NAEP is a measure of the academic proficiencies of 4th-, 8th-, and 12th-grade students Surveys of math, reading, and writing of 4th and 8th graders are conducted every two years: See About NAEP: A Common Measure of Student Achievement, National Center for Educational Statistics, https://nces.ed.gov/nationsreportcard/about/ If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Preface II Beginning in the 1980s, a series of reports were published that focused on the links between educational performance, literacy skills, and the U.S economy including A Nation at Risk, (National Commission on Excellence in Education, 1983), Strategy for U.S Industrial Competitiveness (The Committee for Economic Development, 1984); A Nation Prepared: Teachers for the 21st Century (Carnegie Corporation Forum on Education and the Economy, 1986), and Toward a More Perfect Union (Ford Foundation Project on Social Welfare and the American Future, 1988) If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Acknowledgments Acknowledgments The authors would like to acknowledge the tireless work of Sallie Glickman, former Director of the Philadelphia Workforce Board Sallie has been a force for improving the foundational skills of disadvantaged adults around the nation In the nearly two decades we have worked with Sallie, she has tirelessly worked for a better understanding of the central role that skills should play in workforce programming for adults and in the design of strategies to bolster the skills of adults seeking prosperity in American labor markets We would also like to recognize the work of Johan Uvin, who serves as President of the Institute for Educational Leadership in Washington D.C Throughout his career at the state and federal level, and now in the nonprofit arena, Johan has brought great intellectual insight as well as program and policy savvy to adult skills development Both Sallie and Johan encouraged the authors to delve more deeply into the question of proxy measures of skill We would also like to thank Raija Vaisanen, Research Director at Commonwealth Corporation in Boston Raija provided us with very helpful comments in her review of our draft Her review was of key importance as it relied on her extensive labor-market research experience as well as her work in state and local education and with training organizations Lastly, we are grateful to John Sabatini for organizing a review process that has considerably strengthened this paper If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Introduction Introduction Measures of educational attainment such as high school graduation and college enrollment rates are at center stage in discussions of educational progress, frequently serving as benchmarks of high school performance.1 Almost across the board, those rates show encouraging signs A greater proportion of high school students are graduating on time than ever before, while most who not graduate on time end up with a high school degree by their early 20s, and an overwhelming majority of high school graduates are enrolling in college Race-ethnic groups have shared in the upward trend, with high school completion rates rising the most for Black and Hispanic students, and increases in the college enrollment rate of Black, and especially Hispanic, students outpacing even those of White high school graduates.2 If the goal is to increase and expand the overall level of human capital in the society, these measures paint a promising picture and bode well for Americans—and America's future However, when the lens is focused more sharply on the actual skills held by students and adults in the United States, the image we see shifts dramatically Many are leaving upper secondary and postsecondary education with degrees, yet they lack the skills they'll need for future success in education or the labor market That means that although both policymakers and researchers tend to rely on educational attainment as a gauge for what our students know, the data are telling us a misleading tale.3 American employers place a high value on skills and effectively find and reward individuals with them In particular, there are substantial independent earnings gains for those with a bachelor's or higher degree, and similarly large earnings gains for higher levels of literacy and numeracy proficiency, both for prime age full-time workers and for college graduates.4 The other side of the coin is that obtaining a degree while lacking the skills it purports to represent is a significant problem There is an earnings penalty to those who lack adequate levels of skills—in other words, skills matter Furthermore, new findings from PIAAC reveal that poor literacy and numeracy skills contribute to adverse life outcomes beyond just purely earnings, including a greater risk of unemployment and not having health coverage Over the past 20 years, we have witnessed significant gains in educational attainment, with little evidence of a commensurate gain in skills For example, recent research that examined the literacy and numeracy skill proficiencies of the American millennial generation found that while it has achieved the highest level of educational attainment of any previous generation in American history, the share of this cohort with deficits in essential skills is troublingly large.5 What's more, according to research on trends in fourth- and eighth-grade scores in NAEP,6 the pattern of increasing scores and decreasing achievement gaps among raceethnicity groups, evident in the first few years of the century, has leveled off—especially in the area of mathematics.7 An extreme example of this paradoxical relationship between attainment and skills has recently come to light in the City of Detroit The School District was sued by students who argued that the district had operated in a way as to deprive them of their constitutional right to "access to literacy."8 In 2017, NAEP reading and math scale scores of eighth-grade students in Detroit were lower than those of any other large city in the nation,9 with 96 percent in math and 93 percent of Detroit's eighth graders in reading scoring at or below basic levels.10 Even though NAEP achievement levels state that scores at the basic level denote only partial mastery of the knowledge and skills required for proficiency in reading and math, Detroit achieved a near 80 percent (78.2) on-time graduation rate in 2017.11 If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With Introduction The primary goal of this paper is to highlight the critical disadvantages of focusing on educational attainment data as a stand-in for the level and distribution of skills across the population, which has a real human cost when we fail to see that millions of Americans lack adequate skills despite their diplomas and degrees First, we'll provide additional background on the nature of the problem of overreliance on educational attainment data as well as what those statistics tell us as far as graduation and college enrollment rates Then we'll see what data from PIAAC, an international large-scale assessment of adult skills, independently shows We will then proceed to demonstrate the problems intrinsic in using educational data as a proxy for skills Then we will look through the lens of a methodology, derived from a reasonably well-crafted approach used in Detroit, that relies largely on educational attainment to see the story it tells about the size and composition of the American population lacking essential skills Finally, we will use the current gold standard—measurements of literacy and numeracy from PIAAC—and show the wide difference between the two measures.12 As we pursued this work, however, we were ever mindful of what the late, great Harvard economist James Medoff advised his graduate students: If you can't be with the data you love, then you should love the data you're with The pragmatics behind loving the data you're with are clear from the wide number of studies carried out by economists and social science researchers who utilize proxies to represent an issue of interest This makes some sense: Large-scale sample surveys of households that include direct measures of skills and that are statistically valid may be the data we love, but their development is quite costly and their availability quite scarce Indeed, proxy measures and approximations—the data you're with—dominate social and economic research despite the mixed results.13 If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With A Word about the Data A Word about the Data To bring a tailored meaning to this study, we evaluated the essential literacy standards skill levels for college graduates against a more stringent standard than we did in our analogous evaluation for those with no college degree The reason for using different PIAAC performance standards for these two groups is that different levels of skills are needed for occupations tied to college-level skills than for occupations that are not Although measures of the employment and earning advantages of earning a bachelor's degree abound, careful studies of these advantages find that gains accrue only to college graduates who find employment in college labor market occupations, generally in a professional, managerial, or technical field that utilizes their more advanced education College graduates who fail to find employment in the college labor market have earnings that are about equal to their high school graduate counterparts.14 Please also note that we limited our analysis of the PIAAC data to employed persons, as we were interested in measuring the size of skill deficiencies within the nation's workforce—those who have passed a basic labor market test and found work Adults who were unemployed or not in the labor force were excluded from our study.15 If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With ... graduate students: If you can''t be with the data you love, then you should love the data you'' re with The pragmatics behind loving the data you'' re with are clear from the wide number of studies carried... economic research despite the mixed results.13 If You Can''t Be With the Data You Love: And the Risks of Loving the Data You'' re With A Word about the Data A Word about the Data To bring a tailored... Welfare and the American Future, 1988) If You Can''t Be With the Data You Love: And the Risks of Loving the Data You'' re With Acknowledgments Acknowledgments The authors would like to acknowledge the