Using-validate-measures-of-high-school-academic-achievement_FULL

School of Economics Working Paper Series Using Validated Measures of High School Academic Achievement to Predict University Success Tim Maloney and Kamakshi Singh 2017/10 Using Validated Measures of High School Academic Achievement to Predict University Success Tim Maloney and Kamakshi Singh ∗ School of Economics Auckland University of Technology December 2017 Acknowledgement and Disclaimer: Access to the data used in this study was provided by a public university in New Zealand for the agreed purposes of this research project The interpretations of the results presented in this study are those of the authors and not reflect the views of this anonymous university This work was supported by the Faculty of Business, Economics and Law at Auckland University of Technology JEL Classifications: I21, I23 and I28 ∗ Corresponding Author: Tim Maloney, School of Economics, Auckland University of Technology, Auckland, NEW ZEALAND, tim.maloney@aut.ac.nz (+64-9-921-9823) and Kamakshi Singh, School of Economics, Auckland University of Technology, Auckland, NEW ZEALAND, kamakshi.singh1@gmail.com I Abstract Administrative data from a New Zealand university are used to validate the National Certificate of Educational Achievement (NCEA) Rank Score used in university admissions and scholarship decisions We find no statistical evidence to corroborate the specific weighting scheme used in this index For example, our regression analysis suggests that too much weight is attached to the lowest category of credits in predicting both successful completion outcomes and letter grades To show the potential importance of this validated measure of high school achievement, we run several simulations on these first-year student outcomes at this university We show that the use of an alternative, empirically-validated measure of NCEA results to select students would lead to only slight improvements in course completion rates and letter grades These higher entry standards would lead to declines in the proportions of Pacifica students, but minimal impacts on the proportion of Māori students enrolled at this university Keywords: Academic At-Risk Students, Academic Performance, Academic Success, Econometrics, Economics of Education II Introduction There has been a recent marked acceleration in worldwide enrolments in post-secondary education Between 1970 and 1990, the World Bank estimates that these enrolments, as a percentage of the five-year age group following the completion of high school, increased by one-third (from 10.2% to 13.6%) Between 1990 and 2010, however, this percentage more than doubled (from 13.6% to 29.3%) Using similar measures, tertiary enrolments in New Zealand have increased at a steadier but faster rate over this entire period, with participation increasing five-fold since 1970 Such substantial increases in higher educational participation suggest that less able or academically prepared individuals may be enrolling at university This relates to concerns by individuals and families in other countries over rising rates of academic failure, as well as the fiscal implications for the governments that subsidize these activities (e.g., see related discussions in Murray 2008, Johnson 2012, Raisman 2013 and Duncan 2015) As a result, empirical evidence on factors that are predictive of university failure may be particularly useful in both screening applicants and providing early interventions to improve academic outcomes Yet, such predictive risk analysis on university academic performance that focuses on the overall predictive power of these tools has been relatively rare (e.g., see Engler (2010a and 2010b), and Jia and Maloney (2015) for recent exceptions) The purpose of this study is to analyze a key summary measure of academic achievement from New Zealand high schools commonly used by universities in both screening applicants and providing student scholarships (commonly referred to as the ‘NCEA Rank Score’) Our concern is that this weighted index of academic achievement at school was arbitrarily constructed, and never empirically validated as to its efficacy in predicting relevant university academic outcomes We use regression analysis on administrative data from a large urban university in this country to show that alternative summary measures of high school academic achievement should be used if the objectives are to predict successful course completions or letter grades during the first year of study in bachelor’s degree programmes These alternative summary measures of academic achievement would improve the predictive Tables and figures downloaded from http://data.worldbank.org/indicator/SE.TER.ENRR The New Zealand tertiary sector covers private training establishments, workplace training, institutes of technology and polytechnics, wananga and universities accuracy of our tools for identifying both high-performing and at-risk students entering university The remainder of this paper is organized as follows Section provides a brief literature review and describes the nature of the current assessment system for high school academic achievement in New Zealand Section describes the data used in our analysis Section presents the main empirical results in this study Section provides further empirical results on the likely consequences of using validated summary measures of high school achievement on resulting university outcomes and the representativeness of key demographic groups Section uses simulation results to test the efficiency and equity implications of using Rank Ranks and empirically-validated alternative measures to select students at this university Section concludes and suggests possible future extensions to this study Review of the Relevant Literature and the NCEA System in New Zealand There is a substantial empirical literature on the determinants of academic outcomes at university Studies that focus on summary measures of high school academic achievement (e.g., Grade Point Average (GPA) or class rank) as predictors of subsequent university performance are the most relevant for this current project (e.g., see Johnes 1997, Betts and Morell 1999, Cohen et al 2004 and Angrist et al 2010) A high school GPA is essentially a cumulative index of letter grades Because the standards for assigning grades can vary across individual schools, school districts and academic disciplines, one could argue on this basis that GPA captures relevant high school academic achievement in predicting university performance with considerable measurement error Despite this concern, most empirical studies find that high school GPA positively and significantly influences subsequent university achievement Our concern is slightly different Even if individual grades were consistently applied based on clear performance standards, how we know that the ‘weights’ attached to this index are correct? At least in terms of their usefulness for predicting subsequent academic outcomes, are individual letter grades really ‘worth’ the numerical values conventionally assigned to them? Because Johnes (1997) examines the impact on entry qualifications on university programme completions in the United Kingdom, her analysis is probably more directly relevant to our present study This is because university entry in the UK is based on Advanced Level subject-based qualifications This national standards-based system provides more uniform and consistent indicators of academic achievement than a high school grades in the U.S (even if these could be broken down into subject areas) As expected, Johnes found that summary measures of entry qualifications were negatively and significantly associated with rates of degree programme non-completion New Zealand currently has a national standard-based assessment system for high school achievement The National Certificate of Educational Achievement (NCEA) system has been in place since 2002 It measures student performance against standards of achievement or competence in specific disciplines Assessments take place over the school year and in nationally administered examinations in the chosen subjects at the end of each calendar year Grades of ‘Excellence’, ‘Merit’, ‘Achieved’ or ‘Not Achieved’ are awarded in these standard These qualifications are normally offered over the last three years in high school, and are known as NCEA Levels 1, and 3, respectively Students must achieve 80 credits in approved standards to gain each qualification The awarding of University Entrance normally requires an NCEA Level qualification, including a minimum number of credits in three approved subjects, and a minimum number of credits in literacy and numeracy at lower NCEA levels A summary measure of these NCEA results known as the ‘Rank Score’ was eventually introduced based on the grades obtained in achieved standards for university entrance This index is based on the best 80 credits in approved subjects from NCEA Level 3, where each credit is awarded points for Excellence, points for Merit, points for Achieved, and points for Not Achieved Thus, the maximum Rank Score is 320 (80 Excellence credits at points each) According to this numerical scheme, an Achieved credit is worth exactly onehalf of an Excellence credit, while a Merit credit is worth exactly three-quarters of an Excellence credit This is why some U.S studies (e.g., Cohn et al 2004) also look at the predictive power of national standardized tests (e.g., the Scholastic Aptitude Test or SAT) on subsequent university outcomes Alternatives to the NCEA system exist Some schools use Cambridge International Examination or International Baccalaureate Diploma Programmes In many cases, students complete both NCEA credits and these alternative qualifications Approximately 85% of New Zealand high schools offer only the NCEA system For more background information on this NCEA system, see http://www.nzqa.govt.nz/qualificationsstandards/qualifications/ncea/understanding-ncea/ There are exceptions to this NCEA Level University Entrance requirement For example, Special Admissions status allows individuals aged 20 or older to enroll at university without this qualification For more information on this University Entrance standards see http://www.nzqa.govt.nz/qualificationsstandards/awards/university-entrance/ Over time, this Rank Score has been adopted for use in some capacity in all eight of the universities in New Zealand At least six of these universities explicitly use Rank Scores in their enrollment procedures The other two universities, Lincoln University and the University of Waikato, use this measure in awarding scholarships For example, in 2017 the University of Auckland set minimum Rank Score thresholds that would guarantee applicant placements in Bachelor’s degree programmes of 150 in Arts, 180 in Commerce, 230 in Architectural Studies, 250 in Health Sciences, 260 in Engineering, and 280 in Sciences (Biomedical Sciences) Because Rank Scores are already used in selecting students for admission into university, this may weaken any statistical association between NCEA results and the eventual academic performance of selected students at university For example, this argument has been made elsewhere that Graduate Record Examination (GRE) results may only weakly predict postgraduate performance in the US (e.g., see Moneta-Koehler et al 2017), because the GRE has already ‘done its job’ in selecting the most promising postgraduates Any further statistical relationship between these entry exams and postgraduate grades or completion rates may be relatively weak or nonexistent We accept that a similar issue may exist with NCEA results and early undergraduate success at university However, because of the wider range of student abilities and lesser restrictive standards for students entering Bachelor’s degree programmes, we anticipate that this statistical association will prove to be relatively stronger in this case A few studies in New Zealand have previously considered the usefulness of Rank Scores for predicting first-year university academic outcomes Shulruf at al (2008) used data on 2,877 first-year students at the University of Auckland from 2005 to estimate correlations between Rank Scores and first-year university GPA Like the present study, they speculated that this conventional summary measure of high school academic achievement may not have the highest possible predictive accuracy They experimented with a series of alternative summary measures of NCEA results that emphasized variants like ‘quantity’ (e.g., the total number of credits achieved) and ‘difficulty’ (e.g., recognizing the percentages of students who achieve subject-specific standards) The authors also showed how the predictive power These institutions are: Auckland University of Technology, Massey University, University of Auckland, University of Canterbury, University of Otago, and Victoria University of Wellington These are the six largest universities by full-time equivalent students, including more than 90% of all university enrollments in New Zealand in 2015 (http://www.universitiesnz.ac.nz/nz-university-system) of these alternative measures for first-year GPA results might vary by ethnicity and high school deciles They concluded that ‘quality’ measures like the current Rank Score are more predictive of first-year university GPA than alternative summary measures that emphasize total credits achieved or the relative difficulty of discipline areas Later studies by Scott (2008), Shulruf et al (2009), and Shulruf et al (2012) employed similar methodologies Our study is different from these previous analyses in that we ‘validate’ the weights attached to the different credit types based on objective assessments of their ability to predict first-year university academic achievement Simply put, the aforementioned authors did not use available data to test whether the 4-3-2 weighting scheme for NCEA Level credits is optimal from a predictive analytics perspective Data and Descriptive Statistics Anonymized, individual-level data were provided by a large urban university in New Zealand for the purposes of this study Data collected as part of the normal enrolment process were subsequently linked to the first-year outcomes of all students entering bachelor’s degree programmes in three consecutive years (2013 through 2015) Unlike survey data, administrative data provide more complete and accurate results from official high school and university records on academic performance We use first-year outcomes on individual courses as our unit of observation to avoid concerns about attrition bias in examining later course outcomes for students progressing on to subsequent years of study at this university Table provides definitions of the variables used in our analysis, and summary statistics for students with NCEA Level results > We concentrate on two dependent variables for our predictive risk analysis We first consider a dummy variable on the successful completion of a first-year course A value of one Deciles are used to target funding at disadvantaged schools in New Zealand Schools are allocated to deciles based on the socio-economic status of the communities from which most of their students are drawn Decile schools, for example, are the 10% of schools from the poorest and most disadvantaged communities For more information on the construction of these school deciles https://education.govt.nz/school/running-aschool/resourcing/operational-funding/school-decile-ratings/ indicates that a course was completed with a passing grade; zero otherwise Course completion rates in New Zealand universities are routinely monitored by the government, and fees subsidies can be forfeited if course completion rates fall below 60% Our second dependent variable is a more continuous measure on the course letter grade We suggest that letter grades offer an important additional dimension to this analysis Letter grades may be more closely aligned to the acquisition of knowledge, skills and human capital in the classroom, and subsequent returns in the labor market We convert letter grades to numerical equivalents for our regression analysis on the conventional nine-point scale used in New Zealand In some cases, we had to exclude course observations from our grade point analysis because no letter grades were assigned These generally occurred when courses were taken as ‘pass/fail’ Valid letter grades are available for nearly 96% of the courses in our samples We believe that course completions and grade points offer different summary measures of academic achievement at university Because both may be important in success in subsequent studies at university and eventually in the labour market, we think it is important to consider both outcome measures separately The mean course completion rate was 79.1% for the 78,617 first-year course observations for students in our sample with valid NCEA results 10 The mean course grade point is 3.63, which equates to a letter grade between a C+ and B- The independent variables used in our analysis are grouped into nine categories When the dummy variables are exhaustive and mutually exclusive, the italicized variable in a category is the omitted variable for our regression analysis For example, for the three annual cohorts of first-year students in bachelor’s degree programmes, 2013 is the excluded year We also know the prioritized ethnicity status as used at this university, country of origin, gender and age of our students 11 Course observations are almost three-times more likely to come from These letter grades and their numerical equivalents are A+=9, A=8, A-=7, B+=6, B=5, B-=4, C+=3, C=2, C-=1, and D=0 (or any failing or noncompletion grade) Of course, a GPA from this system can be converted to the four-point US scale by multiplying by four-ninths 10 There are several reasons why enrolled students might not have valid NCEA results They could have graduated from foreign high schools, completed schooling in New Zealand prior to the NCEA system, enrolled without this NCEA level qualification, or previously enrolled at another university 11 Students self-report up to three ethnic identities Anyone who reports being Māori is officially designated as Māori This prioritized ethnic designation then extends to Pacifica, Asian, European and Other in that order students that attended high schools in the top three deciles compared to the bottom three deciles 12 There are four possible types of university entry allowed in our dataset The default entry type is through the NCEA Level qualification External and internal entrance types exist for students previously admitted to another university or progressing on from lower-level predegree programmes at the current university, respectively The latter entry type represents ‘second chance opportunities’ for students who had not acquired University Entrance status coming out of high school (even though they may have obtained NCEA Level results) Special Admissions entry includes individuals who had not achieved University Entrance, but are allowed to enroll at university once when they reach their 20th birthdays (i.e., a possible at-risk group for poor university outcomes) We also have information on the degree programmes in which students initially enrolled at this university A series of eleven dummy variables capture these individual degree programmes 13 We also use a dummy variable to indicate the relatively rare event where student initially enrolled in more than one degree programme (i.e., a Double Degree) Since the course outcome is the unit of observation, we also condition on the academic level of each course Typical first-year courses in a bachelor’s degree programmes would be at Level Courses at Level are typically taken in a pre-degree programme, and are relatively rare in this sample Courses at Levels and would typically occur in the second and third years of study Finally, consider the NCEA Level results reported in Table The mean NCEA Rank Score is 173.4, and associated with 11.7 Excellence, 20.4 Merit, and 39.3 Achieved credits Totaling these means gives us approximately 71.4 credits, which is less than the maximum of 80 credits that can be used in calculating a Rank Score 12 This reflects both the distribution of secondary schools across these deciles, as well as the students who attend university from these school deciles Primary schools are more prevalent in the lower deciles, while high schools are more prevalent in the higher deciles As a result, university students are more likely to come from medium to high-decile high schools rather than from lower-decile high schools 13 These bachelor’s degree programmes are Arts (BA), Business (BBus), Computer and Information Systems (BCIS), Communication Studies (BCS), Design (BDes), Education (BEdu), Engineering Technology (BEngTech), Health Sciences (BHS), International Hospitality Management (BIHM), Sports and Recreation (BSR), and a residual category of several smaller degree programmes (Others) Students must enroll in degree programmes in their first year of study at this university gradually reduce the original sample by raising entry standards using these alternative measures, while retaining the same number of students For example, if a higher Rank Score was used to retain less than half of the original students, we estimate that the mean course completion rate would increase from the present 77.42% to 88.15%, and the current GPA from 3.621 to 4.577 The use of this alternative, empirically-validated measure would have minimal effects on these outcomes for the students selected This alternative measure of NCEA performance would only slightly increase the course completion rate to 88.22%, and GPA to 4.601 Higher entry standards of either form would lead to slight decreases in the proportions of Pacifica students and students from schools in the bottom three deciles However, these higher entry standards would have little impact on the proportion of Māori students enrolled at this university The primary purpose of this paper is to point out the importance of validating weights assigned to indices of prior academic achievement For example, if Rank Scores are used by New Zealand universities in making enrollment and scholarship decisions, then the weights attached to these credits should most likely reflect their contributions in predicting subsequent academic success Much more could be done on this topic Only the appropriate weights of broad categories of NCEA credits have been considered in this paper These weights could vary by the subject matter of these exams, the degree programmes or majors in which students first enroll, or interactions between the two sets of variables The key is that even conventional university administrative data can be used to objectively construct more efficient summary measures of past academic achievement based on statistical associations between finer details on this prior achievement and eventual outcomes at university 19 References Angrist, J., Oreopoulos, P and Williams, T (2010) When opportunity knocks, who answers? New evidence on college achievement award NBER Working Paper Series, Working Paper Number 16643, Cambridge MA Cohn, E., Cohn, S., Balch, D and Bradley, J (2004) Determinants of undergraduate GPAs: SAT scores, high-school GPA and high-school rank Economics of Education Review, 23: 577-586 Betts, J and Morell, D (1999) The determinants of undergraduate grade point average: the relative importance of family background, high school resources, and peer effects Journal of Human Resources, 34(2): 268-293 Duncan, A (2015) Toward a new focus on outcomes in higher education US Secretary of Education, Press Release, University of Maryland-Baltimore County speech, https://www.ed.gov/news/speeches/toward-new-focus-outcomes-higher-education Engler, R (2010a) School leavers’ progression to bachelors-level study Wellington: Ministry of Education http://www.educationcounts.govt.nz/publications/ 80898/school-leavers-progression-to-bachelors-level-study/summary Engler, R (2010b) Academic performance of first-year bachelors students at university Wellington: Ministry of Education http://www.educationcounts.govt.nz/publications/80898/academic-performance-offirst-year-bachelors-students-at-university/summary Jia, P and Maloney, T (2015) Using predictive modelling to identify students at risk of poor university outcomes Higher Education, 70: 127-149 Johnes, J (1997) Inter-university variations in undergraduate non-completion rates: a statistical analysis by subject of study Journal of Applied Statistics, 24(3): 343-361 Johnson, N (2012) The institutional costs of student attrition Working paper, Delta Cost Project, American Institutes for Research, Washington, DC Moneta-Koehler, L., Brown, A., Petrie, K., Evans, B and Chalkley, R (2017) The limitations of the GRE in predicting success in biomedical graduate school PLOS ONE, DOI:10.1371/journal.pone.0166742 Murray, V (2008) The high price of failure in California: how inadequate education costs schools, students, and society Working paper, Pacific Research Institute, San Francisco, CA Raisman, N (2013) The cost of college attrition at four-year colleges and universities Working paper, Educational Policy Institute, Virginia Beach, VA Scott, D (2008) How does achievement at school affect achievement in tertiary education? Working paper, Secondary to Tertiary Transitions Series, Ministry of Education, Wellington, New Zealand 20 Shulruf, B., Hattie, J and Tumen, S (2008) Student pathways at university: patterns and predictors of completion Studies in Higher Education, 33(3): 233-252 Shulruf, B., Li, M., McKimm, J and Smith, M (2012) Breadth of knowledge vs grades: what best predicts achievement in the first year of health sciences programmes? Journal of Educational Evaluation for Health Professions, 9(7): 1-9 Shulruf, B., Turner, R and Hattie, J (2009) A dual admission model for equity in higher education: a multicohort longitudinal study Procedia Social and Behavioral Sciences, 1: 2416-2420 21 Table 1: Variable Definitions and Descriptive Statistics Variables Course Completions Grade Points* if course successfully completed; otherwise Integers ranging from (D or failing grade) to (A+) Sample Means 0.7909 3.6286 NCEA Results Rank Score Rank Score for NCEA Level (i.e., best 80 credits using point values of 4, and for Excellence, Merit and Achieved credits, respectively) 173.447 Excellence Credits Merit Credits Achieved Credits Excellence NCEA Level credits obtained Merit NCEA Level credits obtained Achieved NCEA Level credits obtained 11.725 20.358 39.275 Enrolment Years 2015 2014 2013 if student enrolled in calendar year 2015; otherwise if student enrolled in calendar year 2014; otherwise Omitted category for students enrolled in the year 2013 0.3781 0.3552 0.2667 Prioritized Ethnicities Māori Pacifica Asian Other Ethnicities Undeclared European if student is Māori; otherwise if student is Pacifica; otherwise if student is Asian; otherwise if student is any other ethnicity; otherwise if student did not declare ethnicity; otherwise Omitted category for European ethnicity 0.1114 0.1550 0.2410 0.0620 0.0211 0.4095 Countries of Origin Asia Pacific Islands Other Countries New Zealand if student country of origin Asia; otherwise if student country of origin Pacific Islands; otherwise if student country of origin not listed; otherwise Omitted category for New Zealand country of origin 0.0635 0.0224 0.1039 0.8102 Demographic Factors Female if female student; male Part-time if student studying part-time; full-time Age Student age in years 0.6282 0.0541 18.9765 High School Deciles Decile Decile Decile Decile Decile Decile Decile Decile Decile 10 No Decile Decile 0.0387 0.0464 0.0776 0.1726 0.0648 0.0953 0.0954 0.1504 0.2253 0.0250 0.0732 if student from school decile 1; otherwise if student from school decile 2; otherwise if student from school decile 3; otherwise if student from school decile 4; otherwise if student from school decile 5; otherwise if student from school decile 7; otherwise if student from school decile 8; otherwise if student from school decile 9; otherwise if student from school decile 10; otherwise if school decile unknown; otherwise Omitted category school decile 22 Table Continued Entrance Types External Internal Special NCEA Level if student previously enrolled at another university; otherwise if students obtained pre-degree qualification from this university; otherwise if student entered with special admission (aged 20 or over without University Entrance); otherwise Omitted category for NCEA Level entrance 0.0352 0.0984 0.0253 0.8411 Bachelor’s Degree Programmes BBus if Business; otherwise BCIS if Computer Information Science; otherwise BCS if Communication Studies; otherwise BDes if Design; otherwise BEdu if Education; otherwise BEngTech if Engineering Technology; otherwise BHS if Health Science; otherwise BIHM if International Hospitality Management; otherwise BSR if Sports and Recreation; otherwise BSc if Science; otherwise Other Degrees if other small degree programmes; otherwise BA Omitted category for students enrolled in Bachelor of Arts Double Degree if enrolled in a double degree programme; otherwise 0.2234 0.0610 0.0880 0.0692 0.0306 0.0484 0.1801 0.0355 0.0682 0.0298 0.0673 0.1166 0.0181 Course Levels Level Level Level Level 0.0036 0.2054 0.0149 0.7762 if course level 4; otherwise if course level 6; otherwise if course level 7; otherwise Omitted category level course n 78,617 * There are fewer course observations on students with valid letter grades (n = 75,451) The reported grade point mean is conditional on courses with valid letter grades An unknown school decile is most often associated with a student who completed high school outside New Zealand There are very few of such students in our sample, because they must report valid NCEA results to be included in our analysis In a few cases, students completing high school within New Zealand not have a recorded school decile Most, but not all, private schools in New Zealand have a school decile 23 Table 2A: Maximum Likelihood Probit Regressions on Course Completions Full Set of Results Using NCEA Rank Scores Unrestricted Estimation Independent Variables Constant NCEA Results Rank Score/10 Coefficient 0.0613* Standard Error 0.0355 Restricted Estimation Marginal Effect - Coefficient -0.1066*** 0.0560*** Standard Error 0.0136 Marginal Effect - 0.0557*** 0.0011 0.0143*** Enrolment Years 2015 2014 -0.0597*** -0.0529*** 0.0142 0.0139 -0.0153*** -0.0135*** - - - Prioritized Ethnicities Māori Pacifica Asian Other Ethnicities Undeclared -0.2502*** -0.4050*** -0.0202 -0.2029*** 0.0233 0.0184 0.0176 0.0168 0.0225 0.0421 -0.0640*** -0.1036*** -0.0052 -0.0519*** 0.0060 - - - Countries of Origin Asia Pacific Islands Other Countries -0.0665*** -0.0325 -0.0475 0.0211 0.0286 0.1020 -0.0170*** -0.0083 -0.0122 - - - Demographic Factors Female Part-time Under Age 18 Age 19 Age 20 Age 21 Above Age 21 0.1360*** -0.1680*** 0.1266 -0.0176 -0.0373** -0.0368 0.1217*** 0.0123 0.0224 0.0929 0.0132 0.0168 0.0238 0.0455 0.0348*** -0.0430*** 0.0324 -0.0045 -0.0095** -0.0094 0.0311*** - - - High School Deciles Decile Decile Decile Decile Decile Decile Decile Decile Decile 10 No Decile -0.5701*** -0.1511*** -0.1953*** -0.1717*** 0.0113 -0.0743*** -0.2411*** -0.1368*** -0.1801*** 0.1071*** 0.0330 0.0319 0.0284 0.0266 0.0261 0.0280 0.0274 0.0258 0.0245 0.0411 -0.1458*** -0.0386*** -0.0499*** -0.0439*** 0.0029 -0.0190*** -0.0617*** -0.0350*** -0.0461*** 0.0274*** - - - 24 0.0008 0.0151*** Table 2A Continued Entrance Types External Entry Internal Entry Special Admission 0.2513*** 0.2938*** 0.1945*** 0.0303 0.0196 0.0342 0.0643*** 0.0751*** 0.0497*** - - - Degree Programmes BBus CIS BCS BDes BEdu BEngTech BHS BIHM BSR BSc Other Degrees Double Degree -0.1207*** -0.0291 0.3400*** 0.2921*** 0.6132*** -0.1224*** 0.1336*** 0.3283*** -0.1503*** -0.2020*** 0.0794*** 0.6842*** 0.0190 0.0264 0.0291 0.0300 0.0406 0.0290 0.0204 0.0340 0.0250 0.0326 0.0253 0.0705 -0.0309*** -0.0074 0.0870*** 0.0747*** 0.1568*** -0.0313*** 0.0342*** 0.0840*** -0.0384*** -0.0516*** 0.0203*** 0.1750*** - - - Course Levels Level Level Level 0.1520* 0.1334*** 0.3612*** 0.0857 0.0142 0.0490 0.0389* 0.0341*** 0.0924*** - - - n Pseudo R2 Statistic Pseudo Log-Likelihood 78,617 0.1082 -35,950.7 78,617 0.0604 -37,877.0 *** Statistically different from zero at a 1% level using a two-tailed t test Statistically different from zero at a 5% level using a two-tailed t test * Statistically different from zero at a 10% level using a two-tailed t test ** Notes: Estimated standard errors are adjusted for clustering with multiple course observations for individual students The NCEA Rank Score is divided by ten to make it easier to report and interpret these estimated effects 25 Table 2B: Maximum Likelihood Probit Regression Results on Course Completions Partial Results Using the Best 80 NCEA Credits Instead of Rank Scores Unrestricted Estimation Independent Variables NCEA Results Excellence Credits/10 Merit Credits/10 Achieved Credits/10 Restricted Estimation Coefficient Standard Error Marginal Effect Coefficient Standard Error Marginal Effect 0.1981*** 0.1732*** 0.0525*** 0.0060 0.0052 0.0045 0.0506*** 0.0442*** 0.0134*** 0.2058*** 0.1772*** 0.0166*** 0.0050 0.0042 0.0036 0.0549*** 0.0472*** 0.0044*** n Pseudo R2 Statistic Pseudo Log-Likelihood 78,617 0.1111 -35,832.9 78,617 0.0714 -37,432.5 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 𝛽𝛽𝐴𝐴 1,346.35 (0.0000) 10.82 (0.0010) 241.29 (0.0000) 2,907.51 (0.0000) 16.15 (0.0001) 907.50 (0.0000) H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 2𝛽𝛽𝐴𝐴 *** Statistically different from zero at a 1% level using a two-tailed t test Statistically different from zero at a 5% level using a two-tailed t test * Statistically different from zero at a 10% level using a two-tailed t test ** Notes: Estimated standard errors are adjusted for clustering with multiple course observations for individual students The three credit categories are divided by ten to make it easier to report and interpret these estimated effects The same additional 45 covariates included in the unrestricted estimation in Table 2A were included in the unrestricted estimation in this table Chi-squared statistics and p values on Wald tests involving these credit coefficients are reported at the bottom of this table 26 Table 2C: Maximum Likelihood Probit Regression Results on Course Completions Partial Results Using All Available NCEA Credits Instead of Rank Scores Unrestricted Estimation Independent Variables Restricted Estimation Coefficient Standard Error Marginal Effect Coefficient Standard Error Marginal Effect Excellence Credits/10 0.1756*** 0.0057 0.0448*** 0.1923*** 0.0050 0.0512*** Merit Credits/10 Achieved Credits/10 0.1622*** 0.0597*** 0.0049 0.0037 0.0414*** 0.0152*** 0.1747*** 0.0306*** 0.0042 0.0032 0.0466*** 0.0082*** NCEA Results n Pseudo R2 Statistic Pseudo Log-Likelihood 78,617 0.1122 -35,787.4 78,617 0.0722 -37,403.4 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 𝛽𝛽𝐴𝐴 718.12 (0.0000) 2.81 (0.0936) 121.51 (0.0000) 1,722.53 (0.0000) 5.29 (0.0214) 539.77 (0.0000) H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 2𝛽𝛽𝐴𝐴 *** Statistically different from zero at a 1% level using a two-tailed t test Statistically different from zero at a 5% level using a two-tailed t test * Statistically different from zero at a 10% level using a two-tailed t test ** Notes: Estimated standard errors are adjusted for clustering with multiple course observations for individual students The three credit categories are divided by ten to make it easier to report and interpret these estimated effects The same additional 45 covariates included in the unrestricted estimation in Table 2A were included in the unrestricted estimation in this table Chi-squared statistics and p values on Wald tests involving these credit coefficients are reported at the bottom of this table 27 Table 3A: Ordinary Least-Squares Regressions on Course Grade Points Full Set of Results Using NCEA Rank Score Unrestricted Estimation Independent Variables Constant Coefficient 1.5849*** Standard Error 0.0580 Restricted Estimation Coefficient 1.0331*** Standard Error 0.0257 NCEA Results Rank Score/10 0.1523*** 0.0019 Enrolment Years 2015 2014 0.0010 -0.0622*** 0.0221 0.0215 - - Prioritized Ethnicities Maori Pacifica Asian Other Ethnicities Undeclared -0.4221*** -0.9554*** -0.2706*** -0.5704*** -0.3302*** 0.0302 0.0301 0.0260 0.0376 0.0678 - - Countries of Origin Asia Pacific Islands Other Countries -0.1752*** -0.1172** -0.1408 0.0351 0.0480 0.1646 - - Demographic Factors Female Part-time Under Age 18 Age 19 Age 20 Age 21 Above Age 21 0.2616*** -0.1421*** 0.2990* -0.0269 0.0116 0.1069*** 0.4776*** 0.0200 0.0411 0.1670 0.0203 0.0276 0.0413 0.0828 - - High School Deciles Decile Decile Decile Decile Decile Decile Decile Decile Decile 10 No Decile -1.1681*** -0.2794*** -0.3473*** -0.2591*** -0.0141 -0.1306*** -0.3542*** -0.2925*** -0.3841*** 0.2078*** 0.0550 0.0510 0.0450 0.0410 0.0426 0.0409 0.0418 0.0376 0.0354 0.0641 - - 28 0.1491*** 0.0013 Table 3A Continued Entrance Types External Entry Internal Entry Special Admission 0.8068*** 0.5608*** 0.7796*** 0.0552 0.0329 0.0645 - - Degree Programmes BBus CIS BCS BDes BEdu BEngTech BHS BIHM BSR BSc Other Degrees Double Degree -0.7633*** -0.3522*** -0.7634*** 0.1146*** 1.2013*** -0.5667*** -0.0222 0.3760*** -0.6450*** -0.3991*** -0.0428 2.0524*** 0.0346 0.0474 0.0376 0.0432 0.0567 0.0532 0.0351 0.0514 0.0434 0.0607 0.0444 0.0735 - - Course Levels Level Level Level 0.4353*** 0.1249*** 0.5543*** 0.1672 0.0224 0.0713 - - n R2 Statistic 75,451 0.2163 75.451 0.1429 *** Statistically different from zero at a 1% level using a two-tailed t test Statistically different from zero at a 5% level using a two-tailed t test * Statistically different from zero at a 10% level using a two-tailed t test ** Notes: Estimated standard errors are adjusted for clustering with multiple course observations for individual students The NCEA Rank Score is divided by ten to make it easier to report and interpret these estimated effects 29 Table 3B: Ordinary Least-Squares Regressions on Course Grade Points Partial Results Using the Best 80 NCEA Credits Instead of Rank Scores Unrestricted Estimation Independent Variables NCEA Results Excellence Credits/10 Merit Credits/10 Achieved Credits/10 n R2 Statistic H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 𝛽𝛽𝐴𝐴 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 2𝛽𝛽𝐴𝐴 *** Coefficient Standard Error 0.5311*** 0.3488*** 0.0521*** 0.0083 0.0082 0.0082 Restricted Estimation Coefficient Standard Error 0.5341*** 0.3495*** -0.0032 0.0066 0.0065 0.0068 75,451 0.2307 75.451 0.1735 3,537.95 (0.0000) 453.57 (0.0000) 678.03 (0.0000) 5,749.34 (0.0000) 451.99 (0.0000) 1,383.12 (0.0000) Statistically different from zero at a 1% level using a two-tailed t test Statistically different from zero at a 5% level using a two-tailed t test * Statistically different from zero at a 10% level using a two-tailed t test ** Notes: Estimated standard errors are adjusted for clustering with multiple course observations for individual students The three credit categories are divided by ten to make it easier to report and interpret these estimated effects The same additional 45 covariates included in the unrestricted estimation in Table 3A were included in the unrestricted estimation in this table Chi-squared statistics and p values on Wald tests involving these credit coefficients are reported at the bottom of this table 30 Table 3C: Ordinary Least-Squares Regressions on Course Grade Points Partial Results Using All Available NCEA Credits Instead of Rank Scores Unrestricted Estimation Independent Variables NCEA Results Excellence Credits/10 Merit Credits/10 Achieved Credits/10 n R2 Statistic H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 𝛽𝛽𝐴𝐴 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 H0: 𝛽𝛽𝐸𝐸 = 𝛽𝛽𝑀𝑀 = 2𝛽𝛽𝐴𝐴 *** Coefficient Standard Error 0.4729*** 0.3381*** 0.0707*** 0.0062 0.0069 0.0063 Restricted Estimation Coefficient Standard Error 0.4939*** 0.3568*** 0.0319*** 0.0056 0.0061 0.0057 75,451 0.2309 75.451 0.1732 2,158.53 (0.0000) 214.54 (0.0000) 461.09 (0.0000) 3,397.22 (0.0000) 214.67 (0.0000) 894.14 (0.0000) Statistically different from zero at a 1% level using a two-tailed t test Statistically different from zero at a 5% level using a two-tailed t test * Statistically different from zero at a 10% level using a two-tailed t test ** Notes: Estimated standard errors are adjusted for clustering with multiple course observations for individual students The three credit categories are divided by ten to make it easier to report and interpret these estimated effects The same additional 45 covariates included in the unrestricted estimation in Table 3A were included in the unrestricted estimation in this table Chi-squared statistics and p values on Wald tests involving these credit coefficients are reported at the bottom of this table 31 Table 4: Estimated Effects of Using Rank Scores and Validated Measures to Select Students Paper Completion Rates and Composition of Student Body Proportion Sample Remaining Course Completions Māori Students Pacifica Students Bottom Three School Deciles Original Sample of Students 1.000 0.7742 0.1147 0.1487 0.1578 Rank Score > 110 Rank Score > 130 Rank Score > 150 Rank Score > 170 Rank Score > 190 0.8449 0.7631 0.6588 0.5600 0.4624 0.8003 0.8159 0.8396 0.8606 0.8815 0.1146 0.1152 0.1135 0.1125 0.1102 0.1359 0.1275 0.1186 0.1056 0.0936 0.1494 0.1422 0.1325 0.1212 0.1090 Mean Outcomes from Above Simulations 0.6578 0.8396 0.1132 0.1162 0.1309 Validated Score > 105.71 Validated Score > 122.51 Validated Score > 142.33 Validated Score > 161.28 Validated Score > 182.57 0.8449 0.7631 0.6588 0.5600 0.4624 0.8013 0.8190 0.8397 0.8596 0.8822 0.1161 0.1156 0.1142 0.1150 0.1140 0.1333 0.1239 0.1164 0.1054 0.0909 0.1416 0.1409 0.1312 0.1180 0.1068 Mean Outcomes from Above Simulations 0.6578 0.8404 0.1150 0.1140 0.1277 Notes: The original sample of students for this analysis is 9,520 The thresholds for the validated measures in the second panel were chosen to match the exact number of students selected using the Rank Scores in the first panel This validated measure is the probability of a course completion based only on the total number of Excellent, Merit and Achieved credits This measure comes from the regression results on the restricted estimation listed in Table 2C (i.e., this probability is Φ(0.1923×Excellence Credits + 0.1747×Merit Credits + 0.0306×Achieved Credits) where Φ(˖) is the Cumulative Density Function of Standard Normal) 32 Table 5: Estimated Effects of Using Rank Scores and Validated Measures to Select Students Grade Point Averages and Composition of Student Body Proportion Sample Remaining Grade Point Averages Māori Students Pacifica Students Bottom Three School Deciles Original Sample of Students 1.000 3.6209 0.1142 0.1486 0.1580 Rank Score > 110 Rank Score > 130 Rank Score > 150 Rank Score > 170 Rank Score > 190 0.8453 0.7636 0.6610 0.5622 0.4648 3.8241 3.9576 4.1491 4.3566 4.5774 0.1146 0.1149 0.1135 0.1121 0.1098 0.1354 0.1269 0.1175 0.1043 0.0923 0.1499 0.1425 0.1331 0.1212 0.1091 Mean Outcomes from Above Simulations 0.6594 4.1730 0.1130 0.1153 0.1312 Validated Score > 105.71 Validated Score > 122.51 Validated Score > 142.33 Validated Score > 161.28 Validated Score > 182.57 0.8453 0.7636 0.6610 0.5622 0.4648 3.8482 3.9837 4.1676 4.3673 4.6005 0.1165 0.1166 0.1146 0.1157 0.1142 0.1319 0.1219 0.1152 0.1066 0.0882 0.1484 0.1398 0.1318 0.1203 0.1066 Mean Outcomes from Above Simulations 0.6594 4.1935 0.1155 0.1128 0.1294 Notes: The original sample of students for this analysis is 9,346 The thresholds for the validated measures in the second panel were chosen to match the exact number of students selected using the Rank Scores in the first panel This validated measure is the probability of a course completion based only on the total number of Excellent, Merit and Achieved credits This measure comes from the regression results on the restricted estimation listed in Table 3C (i.e., this expected grade point is 0.4939×Excellence Credits + 0.3568×Merit Credits + 0.0319×Achieved Credits) 33