Defining developing and using curiculum indicators

Defining, Developing, and Using Curriculum Indicators Andrew C Porter John L Smithson CPRE Research Report Series RR-048 December 2001 Consortium for Policy Research in Education University of Pennsylvania Graduate School of Education © Copyright 2001 by the Consortium for Policy Research in Education Defining, Developing, and Using Curriculum Indicators Contents List of Figures iii Biographies v Acknowledgments v Introduction Defining Measures of the Enacted Curriculum .1 Distinguishing the Intended, Enacted, Assessed, and Learned Curricula The Enacted Curriculum The Intended Curriculum .2 The Assessed Curriculum The Learned Curriculum The Importance of a Systematic and Comprehensive Language for Description Developing Curriculum Indicators Content vs Pedagogy Issues in Developing a Curriculum Indicator System Do We Have the Right Language? The Possibility of a Third Dimension Who Describes the Content? .11 Response Metric 11 How Frequently Should Data Be Collected? .12 Validating Survey Data 13 Conducting Alignment Analyses .14 Alignment Criteria .14 Alignment Procedures 15 Using Curriculum Indicators .16 State, District, and School Use 16 Policy Analysis 17 Next Steps for Curriculum Indicators 18 Language and Instrumentation 18 Expansion of Subject Areas .18 Expanding the Taxonomy 19 Developing Electronic Instrumentation .19 Using Video .19 Extending Analyses and Use 20 Summary and Conclusion 20 References 23 Appendix A: Mathematics Topics .25 Appendix B: Science Topics .29 Appendix C: Mathematics Cognitive Demand 35 CPRE Research Report Series, RR-048 Defining, Developing, and Using Curriculum Indicators CPRE Research Report Series, RR-048 Defining, Developing, and Using Curriculum Indicators List of Figures Figure Example of Rotated Matrix Figure Changes in Categories of Cognitive Demand Over Time 10 Figure Developed and Potential Alignment Analyses .14 Figure Grade Eight Science Alignment Analysis 17 CPRE Research Report Series, RR-048 iii Defining, Developing, and Using Curriculum Indicators CPRE Research Report Series, RR-048 iv Defining, Developing, and Using Curriculum Indicators Biographies Acknowledgments Andrew Porter is professor of educational psychology and director of the Wisconsin Center for Education Research at the University of Wisconsin-Madison He has published widely on psychometrics, student assessment, education indicators, and research on teaching His current work focuses on curriculum policies and their effects on opportunity to learn This research was supported by a grant (No OERI-R308A60003) from the National Institute on Educational Governance, Finance, Policymaking, and Management (Office of Educational Research and Improvement, U.S Department of Education) to the Consortium for Policy Research in Education (CPRE) The opinions expressed herein are those of the authors and not reflect the views of the National Institute on Educational Governance, Finance, Policymaking, and Management; the U.S Department of Education; the Office of Educational Research and Improvement; CPRE; or its institutional members John Smithson is a research associate at the Wisconsin Center for Education Research, where he has worked for the past 10 years on developing indicators of classroom practice and instructional content He has worked on several federal- and state-funded research projects investigating changes in classroom instruction based upon various reform initiatives CPRE Research Report Series, RR-048 v Defining, Developing, and Using Curriculum Indicators CPRE Research Report Series, RR-048 vi Defining, Developing, and Using Curriculum Indicators Introduction A s education reform efforts have moved toward a standards-based, accountability-driven, and systemically-integrated approach to improving instructional quality and student learning, researchers and policymakers have become increasingly interested in examining the relationship between the curriculum delivered to students and the goals of state and district policy initiatives Assessing relationships between what is taught and what is desired to be taught has required the development of new methodologies The purpose of this report is to describe the progress of our work as we have worked to develop valid yet efficient measures of instructional content and its relationships to assessment and standards We have focused on mathematics and science, but done some work in language arts and history as well We hope this report is useful to researchers and policymakers who wish to track changes in the content of instruction or to determine relationships between curriculum policies and instructional content We begin with a brief review of the lessons learned in the Reform Up Close study, a Consortium for Policy Research in Education (CPRE) project funded by the National Science Foundation, then discuss the central issues involved in defining and measuring curriculum indicators, while noting how our approach has developed over the past 10 years This is followed by a discussion about using curriculum indicators in school improvement, program evaluation, and informing policy decisions Considerable attention is paid to new methods for determining alignment among instruction, assessments, and standards We conclude with a discussion of the next steps CPRE Research Report Series, RR-048 in the development and expansion of curriculum indicators Defining Measures of the Enacted Curriculum During the 1990-1992 school years, a team of researchers from the University of Wisconsin, led by Andrew Porter, and Stanford University, led by Michael Kirst, undertook an unprecedented large-scale look behind the classroom door (Porter, Kirst, Osthoff, Smithson, and Schneider, 1993) Incorporating an array of data collection tools, the researchers examined mathematics and science instructional content and pedagogy delivered to students in over 300 high school classrooms in six states Detailed descriptions of practice were collected, using daily teacher logs, for a full school year in more than 60 of these classrooms Interest in descriptions of classroom practice has grown steadily since the early 1990s, particularly as high-stakes tests have become a favored component of state and district accountability programs In such an environment it is essential that curriculum indicators provide reliable and valid descriptions of classroom practice Additionally, indicators should be versatile enough to serve the needs of researchers, policymakers, administrators, teachers, and the general public Our work described here has sought to develop measures and analyses that meet these demands Defining, Developing, and Using Curriculum Indicators Distinguishing the Intended, Enacted, Assessed, and Learned Curricula Classroom practice is the focal point for curriculum delivery and student learning So, it is not surprising that policymakers and researchers are interested in understanding the influence of the policy environment (including policies covering standards, assessments, accountability, and professional development) on classroom practice and gains in student achievement The importance of policies guiding curriculum has led us to expand our conceptual framework to consider the curricular implications In the Reform Up Close study, we discussed the intended versus the enacted curriculum, noting that the intention was that practice (the enacted curriculum) should reflect the curriculum policies of the state (the intended curriculum) More recently we have come to distinguish the intended from the assessed curriculum, and the enacted from the learned curriculum (Porter and Smithson, 2001) These distinctions come from the international comparative studies of student achievement literature that first distinguished among the intended, enacted, and learned curricula (McKnight et al., 1987; Schmidt et al., 1996) One could argue that the assessed curriculum is a component of the intended curriculum, and the learned curriculum an aspect of the enacted curriculum But we have found that these finer distinctions serve an important analytic role in tracing the chain of causality from education legislation to student outcomes CPRE Research Report Series, RR-048 The Enacted Curriculum The enacted curriculum refers to the actual curricular content that students engage in the classroom The intended, assessed, and learned curricula are important components of the educational delivery system, but most learning is expected to occur within the enacted curriculum As such, the enacted curriculum is arguably the single most important feature of any curriculum indicator system It has formed the centerpiece of our efforts over the last 10 years; we developed a comprehensive and systematic language for describing instructional content with the enacted curriculum in mind Descriptions of the enacted curriculum still lie at the heart of our work, but we have come to appreciate the importance of looking at the intended, assessed, and learned curricula in combination with the enacted curriculum in order to describe the context within which instruction occurs The Intended Curriculum By the intended curriculum we refer to such policy tools as curriculum standards, frameworks, or guidelines that outline the curriculum teachers are expected to deliver These policy tools vary significantly across states, and to some extent, across districts and schools There are two important types of information that should be collected when examining the intended curriculum The collected information should include the composition of the curriculum described in policy documents It is also important to collect measures that characterize the policy documents themselves For example, how consistent are the policies in terms of curricular expectations? How prescriptive Defining, Developing, and Using Curriculum Indicators are the policies in indicating the content to be delivered? How much authority the policies have among teachers? And finally, how much power have the policies in terms of rewards for compliance and sanctions for non-compliance? (Porter, Floden, Freeman, Schmidt, and Schwille, 1988; Schwille et al., 1983) Such policy analyses are distinct from alignment analyses, and both play a critical role in explaining the curriculum delivered to students The Assessed Curriculum Though assessments could be included in the definition of the intended curriculum, high-stakes tests play a unique role in standards-based accountability systems, often becoming the criteria for determining success or failure, reward or punishment Therefore, it is analytically useful to distinguish the assessed curriculum (represented by high-stakes tests) from the intended curriculum (represented by curriculum standards, frameworks, or guidelines) At a minimum, it can be informative to compare the content in the assessments with the content in the curriculum standards and other policy documents Such comparisons, in most cases, reveal important differences between the knowledge that is valued and the knowledge that is assessed, differences perhaps due to the limitations of resources and the technologies available for assessing student knowledge Lack of alignment leads to an almost inevitable tension between the intended and the assessed curriculum A curriculum indicator system should be able to reveal this tension and be able to characterize its nature within particular education systems CPRE Research Report Series, RR-048 The Learned Curriculum With the advent of standards-based reform and the popularity of accountability systems, student achievement scores are the apparent measure of choice in determining the success of educational endeavors Just as the assessed curriculum is, as a practical matter, restricted to reflecting a subset of the intended curriculum, achievement scores represent just a portion of the knowledge that students acquire as a result of their schooling experience Nonetheless, these measures invariably represent the bottom line for education providers under current reform initiatives Achievement scores may provide a reasonable summary measure of student learning, but, alone, they tell us little about the learned curriculum To be useful for monitoring, evaluating, and diagnosing purposes, indicator measures of the learned curriculum need to describe the content that has been learned as well as the level of proficiency offered by test scores In addition, student outcomes should be mapped on the curriculum to provide information about which parts of the curriculum have been learned by large numbers of students and which aspects require increased attention Several testing services provide skills analyses that tell how well students performed in various content areas While we applaud such efforts, it is not clear the extent to which such analyses are used by teachers, or the extent to which such analyses employ a sufficiently detailed language to meet the indicator needs of the system teachers and 23 teachers overall, your equation would look like this: 15 x 100 = 65% teachers are qualified or trained 23 When looking at statistics on teachers, it is also important to be able to calculate the number of female teachers (both qualified and unqualified) Frequently, there are far fewer female teachers than males teachers, which can have particularly adverse effects on girls’ school enrolment 3.7 Percentage of Female Teachers The operational purpose of calculating the percentage of female teachers is to assess gender equality among teachers Having a sufficient number of female teachers is essential in retaining female students and securing a safe learning environment To calculate the percentage of female teachers you divide the numerator, the total number of female teachers by the denominator, the total number of teachers For example, if there were female teachers and 23 teachers overall, your equation would look like this: 16 The denominator is the total (such as the ‘total number of refugee children aged 6-11’) that is at the bottom half of the fraction You require a denominator to calculate percentages in which you must divide the numerator by the denominator The numerator is the subset of a total at the top half of the fraction, such as the number of refugee girls enrolled in primary school You must divide the numerator by the denominator to calculate percentages Percentages and rates are particularly useful in calculating indicators, as opposed to absolute values, which are the numbers that have not been divided by a total and are not part of a fraction For example, an absolute value is “349 children are attending grade 1.” In contrast, an example of a rate is “78 per cent of schoolaged children are attending grade 1.” The rate shows you not only how many children are in school, but more significantly, how many children are not in school, which is an important factor in programming The absolute value of 349, on the other hand, only tells you how many children are in school, which makes programming decisions more difficult because it does not identify enrolment gaps As well, rates are very important when the camp population is fluctuating; either increasing due to an influx or decreasing due to repatriation or resettlement For example, if one month 349 students attend in grade and then next month only 302 students are in grade 1, you might assume that there is a high drop-out rate, which is usually a negative sign and suggests that something is wrong (for example, child labour, forced military recruitment, childhood illness, or poor education programming) However, if the camp population has drastically decreased in that month due to repatriation, you might find that the attendance rate of 78% is unchanged If for example you had 336 students in the school and 14 teachers, your calculation will look like this: 336 = 24 students per teacher 14 Similarly, if you are calculating student-to-textbook ratios, or student-to-desk ratios, you should include all children in the classroom, regardless of nationality or age Calculating Education Indicators 3.6 Percentage of Qualified or Trained Teachers The education indicators can be difficult to calculate for a number of reasons For certain indicators, such as the student to teacher ratio, you are required to include the local and over-aged children with the refugee children For other indicators, such as enrolment percentages, you are required to include just the school-aged refugee children An additional challenge is that you are required to generate data on certain age groups which, depending on your operation, can be difficult to obtain (i.e 15-24 year olds) If your operation uses proGres, you should be able to obtain the number of refugees in any given age group If you not currently have access to proGres data, it can be obtained by contacting the registration focal point in your operation A good idea, especially when asked to report on indicators such as the ‘percentage of 15-24 year olds enrolled in training,’ is to create a filter in proGres The operational purpose of calculating the percentage of qualified or trained teachers is to assess the quality of education being provided to the students Knowing the percentage of qualified teachers is particularly important in order to assess whether or not displaced children and adolescents are receiving quality education The percentage of qualified teachers is determined by dividing the numerator, the total number of qualified and trained teachers, by the denominator, the total number of teachers If for example there are 15 qualified or trained Refers to teachers who possess an official teacher qualification, or who have completed a ten-day teacher training course 15 This can be a particularly difficult indicator to calculate because the age group being asked for does not correspond easily with the age breakdown requested on population statistics at the beginning of the Standard and Indicator Reports However if your operation has access to the proGres data, you should be able to easily derive your denominator (i.e the total population of refugee adolescents aged 15-24) Depending on how your operation is structured, you may have multiple implementing partners who have data on the number of 15-24 year olds in training programmes To calculate this indicator, you should add together all the individuals aged 15-24 in all the training programmes from all the partners Given that you will have to report on this indicator year after year and monitor for changes, you can create a filter which will automatically reflect the individuals that meet the specified criteria In other words, by saving certain search criteria as a filter, you not need to re-enter the same search criteria the next time you want to perform the same search When an individual no longer meets the set criteria, they will be excluded by the filter and when other individuals meet the criteria, they will be included in the filtered data set For this particular indicator, it is recommended that your operation requests lists from implementing partners of individuals who have participated vocational training and other learning programmes This data should then be entered into proGres 3.5 Number of students per teacher 3.1 Enrolment Rates The operational purpose of knowing the number of students per teacher is to assess the quality of education, as well as the capacity of schools and teachers to provide quality education Unlike the enrolment indictors, this indicator requires that you include all students in the classroom (including over-aged and local children), not just schoolaged refugee children The numerator for this indicator is the total number of students at school, divided by the denominator, the total number of teachers in the school 14 The primary operational purpose of enrolment rates is to determine how many children are not in school In order to calculate the percentage of refugee children enrolled in grades 1-6 (primary school), you will need to divide the numerator, the number of female and male refugees enrolled in grades 1-6 aged 6-11, by the denominator, the total population of female and male refugees between the ages of 6-11 Because your data is likely disaggregated by gender, to get the overall enrolment, the number of refugee girls aged 6-11 enrolled in grades 1-6, and the number of refugee boys aged 6-11 enrolled in grades 1-6 needs to be added together to get the numerator For example, if there are 85 girls in grades 1-6 and 92 boys in grades 1-6, and overall there are 106 girls aged 6-11 and 103 boys aged 6-11, your calculation would look like this: This confusion can cause problems in two major regards Firstly, the percentage of female children who are not enrolled in school remains unknown Secondly, because the percentages reported are not correct, serious programming, policy and funding decisions may be made based on data which is not indicative of the actual situation 3.4 Percentage of adolescents aged 15-24 years enrolled in training (85+92) = 177 x 100 = 84.68, or 85 % of refugees (106 + 103) 209 aged 6-11 are enrolled in grades 1-6 To calculate girls’ enrolment separately, you divide the numerator, 85 female refugees aged 6-11 enrolled in primary school by the denominator, 106 female refugees who are between the ages of 6-11 Your calculation would look like this: 85 x 100 = 80% of girls are enrolled in primary school 106 The process is the same for calculating boys’ enrolment You divide the numerator, 92 boys aged 6-11 enrolled in primary school by the denominator, 103 boys aged of 611 Your calculation would look like this: 92 x 100 = 89% of boys are enrolled in primary school 103 The primary operational purpose of calculating the percentage of 15-24 year olds enrolled in training is to assess the percentage of adolescents who have access to programmes which can help them build skills for the future and facilitate the securing of livelihoods Calculating this indicator also requires that only refugees of this specific age group are included The percentage is determined by dividing the numerator, the number of refugees aged 15-24 enrolled in training, by the denominator, the total refugee population aged 15-24 If for example there were 37 refugees aged 15-24 enrolled in training and a total of 103 refugees aged 15-24, your equation would look like this: 37 x 100 = 36% of adolescents aged 15-24 are enrolled 103 in training 13 When calculating female enrolment rates such as in the above examples, what is sometimes (mistakenly) calculated is the gender parity in the classroom Taking the same example, we will determine if gender parity has been achieved with 85 girls and 92 boys enrolled in primary school This gives us a total of 177 children enrolled A quick way to determine whether or not gender parity has been met in the classroom is to divide the numerator, in this case the number of girls enrolled by the denominator, the total number of students enrolled Your calculation will look like this: 85 x 100 = 48% of the students are female 177 In the above example gender parity has almost been achieved, with 48 per cent of the students being female and 52 per cent of the students being male What sometimes occurs however is that this figure (i.e 48 per cent) is reported as the percentage of girls enrolled in school, and not as the gender parity As we are can see here, there is a large difference between 48 per cent girls’ enrolment and 80 per cent girls’ enrolment in school The above-process can also be applied to calculating the percentage of refugee children aged 6-11 with specific needs attending grades 1-6, and also the percentage of refugee adolescents’ aged 12-17 attending grades 7-12 It is also recommended that each school keep a book with a list of students who have dropped out of school This is important because if there is a high drop out rate, it can be determined whether this is due to repatriation or another cause (such as forced recruitment or early marriage) This information should be compared with the proGres data every months to see if the child has in fact simply repatriated, or once every month in areas of high repatriation Finally, while the indicators for enrolment have predetermined age groups for primary and secondary students, there is a certain degree of flexibility with these age groups If for example, children within your country of operation attend primary school from ages 711, then this age group can be used to reflect primary school enrolment However, this change must be accounted for in both the numerator and the denominator For example, if there were 92 students going to primary school aged 7-11, and 115 children aged 7-11, your equation would look like this: 92 x 100 = 80% of refugee children aged 7-11 attend 115 primary school 12 If the age categories need to be altered to reflect the situation in your country of operation this should be made note of in your comments column While modifying age groups can be done in order to adapt to each country programme, it should be stressed that age groups can be modified only if the numerator and the denominator match (i.e the number of 7-11 year olds in school must be divided by the total number of 7-11 year olds, and not divided by the number of 6-11 year olds) 3.2 Common Challenges with Enrolment Indicators The enrolment indicators are perhaps the most difficult indicators to calculate accurately for several reasons As we saw in the above example, the percentage of refugee children enrolled in primary school should be calculated by dividing the numerator, ‘the number of school-aged refugee students (i.e those aged 6-11),’ by the denominator, ‘the number of refugees aged 6-11’ What often happens however is that over-aged refugees and local children are included in the numerator because they too are attending a primary school in a refugee camp However, bearing in mind that the denominator, only accounts for refugee students aged 6-11, you often end up with enrolment figures of over 100 per cent We will use an example to more clearly demonstrate how this can create problems 10 In primary school there are 315 students enrolled (the numerator), and there are 267 registered refugee students aged 6-11 (the denominator) To calculate the percentage of enrolled refugee children at primary school your equation would look like this: 315 x 100 = 118% are enrolled in primary school 267 However, it is not possible for enrolment to be over 100 per cent What has happened is that over-aged refugee children and local children have been included in the numerator and not in the denominator To further illustrate how this can create analysis and programming challenges, let us say that of the children included in the numerator, only 183 were actually refugee students aged 6-11 As such, the real enrolment percentage for refugee children aged 6-11 enrolled in primary school would be: 183 x 100 = only 70% of refugee children aged 6-11 are 267 in fact enrolled in grades 1-6 3.3 Gender Parity (and the confusion with female enrolment rates) The primary operational purpose of gender parity is to determine if there is gender equality between girls and boys in the classroom, and if girls are being particularly discriminated against as regards access to education 11 If the age categories need to be altered to reflect the situation in your country of operation this should be made note of in your comments column While modifying age groups can be done in order to adapt to each country programme, it should be stressed that age groups can be modified only if the numerator and the denominator match (i.e the number of 7-11 year olds in school must be divided by the total number of 7-11 year olds, and not divided by the number of 6-11 year olds) 3.2 Common Challenges with Enrolment Indicators The enrolment indicators are perhaps the most difficult indicators to calculate accurately for several reasons As we saw in the above example, the percentage of refugee children enrolled in primary school should be calculated by dividing the numerator, ‘the number of school-aged refugee students (i.e those aged 6-11),’ by the denominator, ‘the number of refugees aged 6-11’ What often happens however is that over-aged refugees and local children are included in the numerator because they too are attending a primary school in a refugee camp However, bearing in mind that the denominator, only accounts for refugee students aged 6-11, you often end up with enrolment figures of over 100 per cent We will use an example to more clearly demonstrate how this can create problems 10 In primary school there are 315 students enrolled (the numerator), and there are 267 registered refugee students aged 6-11 (the denominator) To calculate the percentage of enrolled refugee children at primary school your equation would look like this: 315 x 100 = 118% are enrolled in primary school 267 However, it is not possible for enrolment to be over 100 per cent What has happened is that over-aged refugee children and local children have been included in the numerator and not in the denominator To further illustrate how this can create analysis and programming challenges, let us say that of the children included in the numerator, only 183 were actually refugee students aged 6-11 As such, the real enrolment percentage for refugee children aged 6-11 enrolled in primary school would be: 183 x 100 = only 70% of refugee children aged 6-11 are 267 in fact enrolled in grades 1-6 3.3 Gender Parity (and the confusion with female enrolment rates) The primary operational purpose of gender parity is to determine if there is gender equality between girls and boys in the classroom, and if girls are being particularly discriminated against as regards access to education 11 When calculating female enrolment rates such as in the above examples, what is sometimes (mistakenly) calculated is the gender parity in the classroom Taking the same example, we will determine if gender parity has been achieved with 85 girls and 92 boys enrolled in primary school This gives us a total of 177 children enrolled A quick way to determine whether or not gender parity has been met in the classroom is to divide the numerator, in this case the number of girls enrolled by the denominator, the total number of students enrolled Your calculation will look like this: 85 x 100 = 48% of the students are female 177 In the above example gender parity has almost been achieved, with 48 per cent of the students being female and 52 per cent of the students being male What sometimes occurs however is that this figure (i.e 48 per cent) is reported as the percentage of girls enrolled in school, and not as the gender parity As we are can see here, there is a large difference between 48 per cent girls’ enrolment and 80 per cent girls’ enrolment in school The above-process can also be applied to calculating the percentage of refugee children aged 6-11 with specific needs attending grades 1-6, and also the percentage of refugee adolescents’ aged 12-17 attending grades 7-12 It is also recommended that each school keep a book with a list of students who have dropped out of school This is important because if there is a high drop out rate, it can be determined whether this is due to repatriation or another cause (such as forced recruitment or early marriage) This information should be compared with the proGres data every months to see if the child has in fact simply repatriated, or once every month in areas of high repatriation Finally, while the indicators for enrolment have predetermined age groups for primary and secondary students, there is a certain degree of flexibility with these age groups If for example, children within your country of operation attend primary school from ages 711, then this age group can be used to reflect primary school enrolment However, this change must be accounted for in both the numerator and the denominator For example, if there were 92 students going to primary school aged 7-11, and 115 children aged 7-11, your equation would look like this: 92 x 100 = 80% of refugee children aged 7-11 attend 115 primary school 12 Because your data is likely disaggregated by gender, to get the overall enrolment, the number of refugee girls aged 6-11 enrolled in grades 1-6, and the number of refugee boys aged 6-11 enrolled in grades 1-6 needs to be added together to get the numerator For example, if there are 85 girls in grades 1-6 and 92 boys in grades 1-6, and overall there are 106 girls aged 6-11 and 103 boys aged 6-11, your calculation would look like this: This confusion can cause problems in two major regards Firstly, the percentage of female children who are not enrolled in school remains unknown Secondly, because the percentages reported are not correct, serious programming, policy and funding decisions may be made based on data which is not indicative of the actual situation 3.4 Percentage of adolescents aged 15-24 years enrolled in training (85+92) = 177 x 100 = 84.68, or 85 % of refugees (106 + 103) 209 aged 6-11 are enrolled in grades 1-6 To calculate girls’ enrolment separately, you divide the numerator, 85 female refugees aged 6-11 enrolled in primary school by the denominator, 106 female refugees who are between the ages of 6-11 Your calculation would look like this: 85 x 100 = 80% of girls are enrolled in primary school 106 The process is the same for calculating boys’ enrolment You divide the numerator, 92 boys aged 6-11 enrolled in primary school by the denominator, 103 boys aged of 611 Your calculation would look like this: 92 x 100 = 89% of boys are enrolled in primary school 103 The primary operational purpose of calculating the percentage of 15-24 year olds enrolled in training is to assess the percentage of adolescents who have access to programmes which can help them build skills for the future and facilitate the securing of livelihoods Calculating this indicator also requires that only refugees of this specific age group are included The percentage is determined by dividing the numerator, the number of refugees aged 15-24 enrolled in training, by the denominator, the total refugee population aged 15-24 If for example there were 37 refugees aged 15-24 enrolled in training and a total of 103 refugees aged 15-24, your equation would look like this: 37 x 100 = 36% of adolescents aged 15-24 are enrolled 103 in training 13 This can be a particularly difficult indicator to calculate because the age group being asked for does not correspond easily with the age breakdown requested on population statistics at the beginning of the Standard and Indicator Reports However if your operation has access to the proGres data, you should be able to easily derive your denominator (i.e the total population of refugee adolescents aged 15-24) Depending on how your operation is structured, you may have multiple implementing partners who have data on the number of 15-24 year olds in training programmes To calculate this indicator, you should add together all the individuals aged 15-24 in all the training programmes from all the partners Given that you will have to report on this indicator year after year and monitor for changes, you can create a filter which will automatically reflect the individuals that meet the specified criteria In other words, by saving certain search criteria as a filter, you not need to re-enter the same search criteria the next time you want to perform the same search When an individual no longer meets the set criteria, they will be excluded by the filter and when other individuals meet the criteria, they will be included in the filtered data set For this particular indicator, it is recommended that your operation requests lists from implementing partners of individuals who have participated vocational training and other learning programmes This data should then be entered into proGres 3.5 Number of students per teacher 3.1 Enrolment Rates The operational purpose of knowing the number of students per teacher is to assess the quality of education, as well as the capacity of schools and teachers to provide quality education Unlike the enrolment indictors, this indicator requires that you include all students in the classroom (including over-aged and local children), not just schoolaged refugee children The numerator for this indicator is the total number of students at school, divided by the denominator, the total number of teachers in the school 14 The primary operational purpose of enrolment rates is to determine how many children are not in school In order to calculate the percentage of refugee children enrolled in grades 1-6 (primary school), you will need to divide the numerator, the number of female and male refugees enrolled in grades 1-6 aged 6-11, by the denominator, the total population of female and male refugees between the ages of 6-11 and then next month only 302 students are in grade 1, you might assume that there is a high drop-out rate, which is usually a negative sign and suggests that something is wrong (for example, child labour, forced military recruitment, childhood illness, or poor education programming) However, if the camp population has drastically decreased in that month due to repatriation, you might find that the attendance rate of 78% is unchanged If for example you had 336 students in the school and 14 teachers, your calculation will look like this: 336 = 24 students per teacher 14 Similarly, if you are calculating student-to-textbook ratios, or student-to-desk ratios, you should include all children in the classroom, regardless of nationality or age Calculating Education Indicators 3.6 Percentage of Qualified or Trained Teachers The education indicators can be difficult to calculate for a number of reasons For certain indicators, such as the student to teacher ratio, you are required to include the local and over-aged children with the refugee children For other indicators, such as enrolment percentages, you are required to include just the school-aged refugee children An additional challenge is that you are required to generate data on certain age groups which, depending on your operation, can be difficult to obtain (i.e 15-24 year olds) If your operation uses proGres, you should be able to obtain the number of refugees in any given age group If you not currently have access to proGres data, it can be obtained by contacting the registration focal point in your operation A good idea, especially when asked to report on indicators such as the ‘percentage of 15-24 year olds enrolled in training,’ is to create a filter in proGres The operational purpose of calculating the percentage of qualified or trained teachers is to assess the quality of education being provided to the students Knowing the percentage of qualified teachers is particularly important in order to assess whether or not displaced children and adolescents are receiving quality education The percentage of qualified teachers is determined by dividing the numerator, the total number of qualified and trained teachers, by the denominator, the total number of teachers If for example there are 15 qualified or trained Refers to teachers who possess an official teacher qualification, or who have completed a ten-day teacher training course 15 teachers and 23 teachers overall, your equation would look like this: 15 x 100 = 65% teachers are qualified or trained 23 When looking at statistics on teachers, it is also important to be able to calculate the number of female teachers (both qualified and unqualified) Frequently, there are far fewer female teachers than males teachers, which can have particularly adverse effects on girls’ school enrolment 3.7 Percentage of Female Teachers The operational purpose of calculating the percentage of female teachers is to assess gender equality among teachers Having a sufficient number of female teachers is essential in retaining female students and securing a safe learning environment To calculate the percentage of female teachers you divide the numerator, the total number of female teachers by the denominator, the total number of teachers For example, if there were female teachers and 23 teachers overall, your equation would look like this: 16 The denominator is the total (such as the ‘total number of refugee children aged 6-11’) that is at the bottom half of the fraction You require a denominator to calculate percentages in which you must divide the numerator by the denominator The numerator is the subset of a total at the top half of the fraction, such as the number of refugee girls enrolled in primary school You must divide the numerator by the denominator to calculate percentages Percentages and rates are particularly useful in calculating indicators, as opposed to absolute values, which are the numbers that have not been divided by a total and are not part of a fraction For example, an absolute value is “349 children are attending grade 1.” In contrast, an example of a rate is “78 per cent of schoolaged children are attending grade 1.” The rate shows you not only how many children are in school, but more significantly, how many children are not in school, which is an important factor in programming The absolute value of 349, on the other hand, only tells you how many children are in school, which makes programming decisions more difficult because it does not identify enrolment gaps As well, rates are very important when the camp population is fluctuating; either increasing due to an influx or decreasing due to repatriation or resettlement For example, if one month 349 students attend in grade will influence what action UNHCR and its partners take in order to improve the living situations of refugees and other persons of concern In order for an indicator to be useful, it must be calculated in the same way every time This is important so that statistics are comparable across different places and in the same place over different times The reason for pairing standards with indicators is that while an indicator is the measurement itself, a standard defines the point for that particular measurement which must be reached or maintained Standards are useful because they provide an indication of where programming is falling short of reaching operational goals and they quantify this shortfall The difference between the standard and the indicator measured in your operation is called a “gap.” For example, if the standard for primary enrolment is set at 100 per cent, but the indicator measures enrolment at only 70 per cent, there is a gap of 30 per cent Sometimes due to cultural, geographic or other circumstances, it may not be possible for your area to achieve all of the standards If this is the case, this should be noted in the ‘comments column’ of the Standards and Indicators Report There are also two other useful terms worth clarifying for the purposes of calculating indicators which require the use of fractions These terms are the denominator and the numerator x 100 = 39% of teachers are female 23 Let us say for example that of the female teachers only were qualified If you wanted to know the percentage of female teachers who were qualified, and also the percentage of qualified female teachers overall, your two equations would look like this: x 100 = 66.67, or 67% of female teachers are qualified x 100 = 26% of all teachers are qualified female 23 teachers A similar process can be followed to calculate the percentage of qualified male teachers, and also what percentage of the teaching population is comprised of qualified male teachers Why are Accurate Calculations Important and what is the Data Used For? Accurate and systematic calculation and collection of education statistics is important for several reasons Accurate calculation makes it possible to assess the education situation and determine where the gaps are Also, it is very difficult to compare whether progress has been made each year in improving both access to and 17 quality of education, if the data cannot be compared over time Similarly, programming decisions that are not based on the indicators may not in fact target the largest education gaps, because these gaps have not been properly identified The education standard and indictor information is also important to secure funding from donors and potential donor countries and organizations If, for example, all of the countries in region A are reporting over 100 per cent enrolment and/or 100 per cent of qualified teachers, it would appear as if these countries are doing very well as regards access to and quality of education In reality however, enrolment may only be at 70 per cent and only 60 per cent of the teachers may be qualified teachers Having the right statistics is extremely beneficial, as they serve as a ‘signal’ when there are major gaps as regards refugee education If and when gaps are identified, more resources can be channeled and allocated accordingly The data collected can also be used to seek out new and strengthen existing partnerships If UNHCR notices for example that there is a very low percentage of 15-24 year olds enrolled in training programmes, then UNHCR and relevant partners can use this information to address this gap and strengthen non formal education activities Introduction Many questions are often asked when calculating education statistics for Standard and Indicator Reports, such as ‘When I include the local population? Why are enrolment percentages reaching over 100 per cent? Is gender parity the same as the percentage of girls enrolled in school?’ This handbook has been developed because education indicators can be both difficult and confusing to calculate, especially when working in locations with large population movements This handbook aims to bring clarity on the calculation of these indicators, by defining keys terms, giving concrete examples of calculations using fractions, and discussing the rational behind accurate and systematic education data collection Standards and Indicators and Other Key Terms Standards and Indicators are often spoken of in relation to one another, but also they need to be understood on their own in order to be useful Indicators are the quantitative or qualitative parameters (or measures) that determine, over time, performance of functions, processes, and outcomes In other words, indicators are measures of progress: they are measurable values that The definitions used in this section are derived from the UNHCR Operational Data Management Learning Programme 18 Table of Contents Introduction……………………………………… Standards and indicators and other key terms…… 3 Calculating education standards and indicators… 3.1 Enrolment rates…………………………………7 3.2 Common challenges with enrolment indicators 3.3 Gender parity (and the confusion with female enrolment rates)……………………………… 10 3.4 Percentage of adolescents aged 15-24 years enrolled in training…………………………….12 3.5 Number of students per teacher……………… 13 3.6 Percentage of qualified or trained teachers… 14 3.7 Percentage of female teachers…………………15 Why are accurate calculations important and what is the data used for? 16 Helpful Resources General information on UNHCR statistics, including standards and indicators can be found on the website: www.unhcr.org UNHCR Education http://www.unhcr.org/protect/405027d34.html Internal Resources Practical Guide to the Systematic use of Standards and Indicators in UNHCR Operations (Second Edition) February 2006 The Operational Data Management Learning Programme http://intranet.hcrnet.ch/ Available by clicking on ‘Staff Development,’ ‘Learning Activities,’ and then ‘Operations.’ Helpful resources…… ………………………… 17 19