Introduction
Open Educational Resources (OER) are teaching, learning, and research materials that are either in the public domain or available under licenses that allow free use and modification These resources promote accessibility and collaboration in education, enabling educators and learners to share and adapt content freely By utilizing OER, individuals can enhance their educational experiences while contributing to a more open and inclusive learning environment.
(William and Flora Hewlett Foundation)
The use of Open Educational Resources (OER) is increasingly being recognized as one of the most significant educational movements thus far in the 21st century The
NMC Horizon Report: 2013 Higher Education Edition, for example, counts it as the top Key Trend in terms of likely significance for higher education from 2013-18:
Openness, encompassing ideas such as open content, open data, and open resources, is increasingly recognized as a valuable principle, emphasizing transparency and easy access to information This shift was notably embraced at the 2012 World Open Educational Resources Congress, highlighting the growing importance of accessible educational materials and data.
Resources Congress, UNESCO’s 2012 Paris OER Declaration calls on governments around the world to adopt policies and support capacity development to promote the use of OER (UNESCO, 2012).
Open Educational Resources (OER) present an appealing solution for learners facing financial constraints or those in areas with restricted access to localized educational materials However, it is essential to explore the effectiveness of these resources and identify the most productive usage and adoption models.
Since 2002, the Hewlett Foundation has played a crucial role in promoting high-quality, free, and open educational resources (OER), providing significant funding for various programs dedicated to OER development and research This report highlights one of those key initiatives.
This evaluation report assesses the achievements and challenges faced by the OER Research Hub within the wider scope of OER research It highlights the hub's status at the time of its funding, its current position, and outlines critical questions that need to be addressed for the future of OER research.
About This Evaluation
In September 2014, the William and Flora Hewlett Foundation engaged SRI
International is conducting a review of its investment in the OER Research Hub (OERRH), assessing its achievements and challenges, the value it brings to the global OER research community, and providing future recommendations This evaluation aims to adapt to the evolving landscape of Open Educational Resources (OER) and OER research.
The evaluation used an assortment of methods:
To gain insights into OERRH's objectives, operational methods, and offerings, we engaged with the team through interviews, reviewed relevant reports, and analyzed their online presence, including the OERRH website and Evidence Hub, while also conducting web searches on their dissemination products.
To understand the perspective of the partners that worked with OERRH, we interviewed 8 OERRH research fellows and other representatives of partner organizations.
To contextualize the efforts of the OER Research Hub within the larger OER community, we conducted interviews with 10 experts in OER and OER research, showcasing diverse backgrounds and roles within the field.
This report highlights the evaluation findings and outlines future requirements for OER research The following section provides an overview of the field, emphasizing its status when the OER Research Hub commenced its activities in 2012.
OER Research: The Lay of the Land
Over the past decade, there has been a remarkable increase in the accessibility and organization of open educational resources (OER), leading to a surge in related research Since 2009, the Babson Survey Research Group has incorporated OER into its annual report on online learning in higher education Furthermore, various OER tools and platforms, such as the Concord Consortium’s Molecular Workbench, the University of Colorado Boulder’s PhET simulations, and Carnegie Mellon’s Open Learning Initiative, have become focal points for extensive academic investigation.
2015, the OER Knowledge Cloud (https://oerknowledgecloud.org/), a repository of research related to OER, had over 1,000 postings
Most research on Open Educational Resources (OER) tends to focus on exploration, description, or advocacy rather than rigorous empirical testing of their effectiveness While there has been some progress in recent years, several knowledgeable interviewees noted that the OER research field remains underdeveloped, characterizing it as "early stage" and overly optimistic, often highlighting positive stories without providing a balanced perspective One interviewee pointed out that much of the discourse around OER is still largely opinion-based, indicating a need for stronger empirical evidence.
To test this claim, we analyzed a total of 78 OER-related articles that were published between 2012 and 2014 in the International Review of Research in Open and
Distributed Learning (IRRODL), a prominent open journal in the OER field Coding classified the articles into categories as shown in Table 1:
Policy study Study focusing on OER-related policies in place in one or more settings
Other articles with no empirical outcomes
Published articles that offer a theory, history or position, or that describe interventions or technologies, rather than describing empirical research
Over a three-year period from 2012 to 2014, the journal published a total of 33 articles in the final category, which consisted of non-empirical research This was followed by 28 implementation studies and 12 empirical studies lacking comparison groups Notably, only two impact studies featuring a control condition were published during this timeframe.
Over the past three years, the dataset indicates a noticeable maturation in both the content and methodology of research on Open Educational Resources (OER) As illustrated in Figure 1, there has been a decline in the number of OER articles lacking empirical outcomes, alongside a rise in implementation studies This shift suggests a growing interest in exploring the specific conditions and parameters surrounding the use of OER.
Figure 1 Frequency of OER Study types in IRRODL, 2012-14
A more comprehensive review of the literature was published by the Open
In 2015, a review by Hilton identified trends in the efficacy of Open Educational Resources (OER) compared to traditional learning materials The review focused on empirical studies published in peer-reviewed journals, highlighting nine studies that specifically assessed OER effectiveness Notably, eight of these studies were published in 2012 or later, underscoring the growing interest in OER research.
The maturation of OER research is understandably slow due to its inherent challenges Traditional research designs often struggle with OER's fundamental characteristics, such as the ability for users to remix and adapt content freely, which complicates the establishment of controlled conditions necessary for efficacy studies Additionally, the context of implementation is crucial in OER research, as variations in adaptation can alter both the effectiveness and the nature of the educational resources themselves, adding further complexity to the research process.
A significant challenge in Open Educational Resources (OER) research is the lack of agreement among researchers and practitioners regarding the definition of OER Survey findings indicate that a considerable number of faculty members and chief academic officers are uncertain about what OER entails.
A recent survey by the Babson Research Group revealed that two-thirds of faculty members are unfamiliar with Open Educational Resources (OER), highlighting a significant gap in awareness (Allen & Seaman, 2015, p 29) Among those who are somewhat knowledgeable about OER, there remains a lack of clarity regarding its definition, as the term encompasses a wide array of materials, from course syllabi to individual learning objects and complete courses.
There is a significant lack of awareness and consensus regarding the essential characteristics that define an educational resource as "open." While many people emphasize the "free to use" aspect of open educational resources (OER), some definitions encompass resources available at very low costs A critical omission in many definitions is the permission to adapt and repurpose these resources Additionally, global awareness of intellectual property issues varies, with some regions unfamiliar with the concept of restrictions on adaptation This lack of clarity on whether adaptability is a fundamental feature of OER greatly influences what qualifies as open For instance, Carnegie Mellon’s Open Learning Initiative offers free access to its courses, yet users face limitations in modifying the content.
The widespread unfamiliarity with the term Open Educational Resources (OER) leads to varied interpretations among survey respondents, affecting their perceptions of OER's benefits, effectiveness, and drawbacks.
The low visibility of the term Open Educational Resources (OER) has led to challenges in locating empirical research specifically focused on OER instances Often, well-known educational products like Khan Academy, OLI Statistics courses, or Phet simulations overshadow the OER label itself Consequently, if research does not highlight unique aspects of OER compared to general digital learning resources, the term may be absent from publication titles or abstracts, making it difficult for scholars to identify relevant studies.
A significant challenge in Open Educational Resources (OER) research is the inconsistency in the characteristics being studied While it is widely recognized that free and unrestricted access enhances the dissemination of educational materials, the benefits of licenses that permit modification and adaptation are still underexplored Understanding how often these licenses are utilized and the patterns of their application presents a critical area for future investigation (Smith, 2013).
In 2012, the OER Research Hub identified a lack of synthesized findings and coordinated priorities among OER researchers, who were largely working independently to address the challenges of this emerging field This situation influenced the Hub's focus, guiding the challenges they tackled, the tasks they undertook, and the priorities that defined their research efforts.
The OER Research Hub
The OER Research Hub, established in 2012 with funding from the William and Flora Hewlett Foundation, aims to investigate the impact of Open Educational Resources (OER) on learning and teaching practices Based at the Open University in the UK, a prominent institution for distance learning and research, the initiative seeks to enhance the visibility and quality of OER research in the educational field.
OpenLearn, the Open University's platform for Open Educational Resources (OER) courses, along with OLnet—a three-year research initiative in collaboration with Carnegie Mellon University—established a strong foundation for the Hub This positioning enabled the Hub to effectively coordinate and promote research on OER from its inception.
Activities conducted by the OER Research Hub are designed to serve a number of different roles in the youthful OER research community described in the previous section The OERRH:
The OER Research Hub conducts global research on Open Educational Resources (OER), focusing on 11 key hypotheses identified in collaboration with the Hewlett Foundation To explore these important questions, the Hub developed a modular survey tailored to address specific hypotheses in various settings This survey is primarily distributed through research partnerships with diverse organizations worldwide.
1 Use of OER leads to improvement in student performance and satisfaction.
2 The open aspect of OER creates different usage and adoption patterns than other online resources.
3 Open education models lead to more equitable access to education, serving a broader base of learners than traditional education.
4 Use of OER is an effective method for improving retention for at-risk students.
5 Use of OER leads to critical reflection by educators, with evidence of improvement in their practice.
6 OER adoption at an institutional level leads to financial benefits for students and/or institutions.
7 Informal learners use a variety of indicators when selecting OER.
8 Informal learners adopt a variety of techniques to compensate for the lack of formal support, which can be supported in open courses.
9 Open education acts as a bridge to formal education, and is complementary, not competitive, with it.
10 Participation in OER pilots and programs leads to policy change at the institutional level.
11 Informal means of assessment are motivators to learning with OER.
The OERRH is dedicated to enhancing research capacity among OER researchers through its fellowship programs, which foster capacity development and research partnerships Six linked fellows, representing various partner organizations, have participated in approximately two-week visits to collaborate on their defined research projects while gaining insights from the OERRH team and experts at the Open University Additionally, the OERRH has offered five open fellowships focused on broader research objectives, such as OER policy To further support research capacity building, the OERRH provides accessible tools like the OERRH survey template and engages in ongoing consulting activities for a diverse range of organizations.
We enhance the research capabilities of partner organizations in the Open Educational Resources (OER) sector by offering expert consultation on research design, assisting with data collection and analysis, and acting as supplementary research team members for those with limited internal resources.
The OER Impact Map is a comprehensive and searchable database that compiles over 130 pieces of research evidence related to Open Educational Resources (OER) as of December 2014 This database organizes findings by geographic location and their relevance to 11 key hypotheses A visual representation of this evidence, illustrating support and opposition for each hypothesis, is provided in Figure 2 from the Impact Map.
Figure 2 OER Impact Map synthesis of evidence for and against hypotheses
The OERRH team actively shares Open Educational Resources (OER) research through various platforms, including blogs, conference presentations, social media, and online courses Their visibility at major events for OER practitioners and researchers enhances the dissemination of their findings and insights.
The OERRH team has achieved several significant milestones that lay the groundwork for the organization's ongoing development This section includes insights gathered from interviews with OERRH partners, fellows, and collaborators.
The OER Research Hub has built a solid reputation among a community of OER researchers and practitioners as leader, expert, and role model
Consensus among collaborators suggests that the
The OER Research Hub (OERRH) serves as a vital resource for expertise in open research, earning widespread appreciation from partner institutions for its guidance and models Recognized as leaders in fostering a community of practice for OER research, OERRH fills a crucial gap in coordination within this emerging field For newer research organizations aiming to establish international credibility and secure funding, the partnership with OERRH provides a significant “seal of approval.”
“They’re probably the world leaders in research on OER… No, not probably, actually.”
“They’ve been unbelievably helpful to us because of their
[respected position in the field]… Our credibility would have suffered had
OERRH in the field is also evidenced by the fact that the group’s leader has recently released a second book on the state of OER (Weller, 2014)
OERRH provides active support for less-experienced
OER researchers are aiming to enhance their research efforts, and the OERRH team is actively facilitating connections with these researchers through various channels The most impactful collaborations so far have been established directly with the researchers themselves.
OERRH fellowship recipients from partner institutions.
Many participants in research roles, particularly those who are new or the only resource in their organizations, have greatly benefited from the guidance of an independent and experienced team Several fellows highlighted the valuable insights and knowledge they gained through their interactions with this supportive network.
The OERRH team, along with connections made at the Open University, has gained valuable insights into accessibility issues for learners with disabilities that they might not have encountered locally This collaboration has particularly benefited those in advocacy roles, who regularly engage with educators and OER users about the advantages and resources of Open Educational Resources (OER), providing them with fresh perspectives on their work.
OERRH has developed tools for conducting integrated and customized research across various national contexts, focusing on 11 key hypotheses By creating a modularized survey as a standardized instrument and leveraging partnerships through the OER Research Hub, the program fosters international research collaboration in both the Global South and Global North Team members collaborate with partner organizations to analyze survey results and adapt research methodologies to address local concerns For instance, in South Africa, the primary barrier to OER adoption is file size for accessible downloads, while in Western countries, quality tends to be the more significant issue.
“It was like a mini accelerator [for early career OER researchers]”
“When I approach people about open education, it’s a different conversation I have different questions to ask.”
The OER Research Hub prioritizes openness and inclusivity in its practices, striking a balance between these principles and the need for rigor This commitment enhances their role as a leading voice and exemplary model in the field.
The concept of "openness" has influenced initial design choices that conflict with the rigorous research standards advocated by the William and Flora Hewlett Foundation This tension is evident in two specific areas, which are detailed below.
Future Needs for OER Research
This section outlines future research directions to assist policymakers, educators, and producers of open educational resources (OER) in addressing key educational challenges It summarizes the OER research needs identified by experts, introduces classification systems for OER and related research studies, and offers recommendations for enhancing the curation, synthesis, and usability of a next-generation research hub.
What We Heard from Stakeholders
In the interviews we conducted, stakeholders articulated a diverse set of needs for OER research For example:
There is a critical need for comprehensive, controlled studies to assess the learning outcomes and cost-effectiveness of Open Educational Resources (OER) compared to traditional and other digital materials across diverse educational contexts Additionally, stakeholders emphasize the importance of a credible synthesis of existing research to effectively demonstrate to education policymakers that OER represents a viable alternative to commercially published resources.
Educators exploring Open Educational Resources (OER) are looking for research on the effectiveness of various OER products, as well as studies that provide insights on effectively integrating OER into their instructional practices and educational institutions.
Those seeking to adopt some specific example of OER are looking also for research on best practices for implementation in contexts like their own.
Funders interested in investing in Open Educational Resources (OER) seek research that highlights the unique features of OER compared to other learning materials, particularly its adaptability and reusability in various contexts This raises important questions regarding the types of adaptations made to OER and whether these modifications generally enhance the effectiveness of the resources compared to using them in their original form.
Policymakers aiming to enhance educational access and reduce costs through Open Educational Resources (OER) seek research-backed strategies that promote OER adoption and effective utilization Additionally, they face broader policy considerations, particularly in contexts like the recent overhaul of education standards, exemplified by the adoption of the Common Core State Standards.
Many U.S states, schools, and districts utilizing Open Educational Resources (OER) can adapt to new educational standards more rapidly than those relying on commercial instructional materials.
There is a consensus among experts that while there are diverse opinions on the contributions of Open Educational Resources (OER) research, it is essential to enhance the rigor of OER studies To gain credibility within wider policy and research circles, OER research must progress beyond mere advocacy.
Recent analysis of trends in the Open Educational Resources (OER) research landscape over the last three years indicates that progress is occurring, albeit slowly A significant barrier to advancement is the absence of a unified understanding of what defines OER, along with the essential factors that must be considered to interpret varying OER outcomes and associated costs effectively.
Open Educational Resources (OER) encompass a wide range of educational tools, from middle school science apps to online college textbooks and course management systems, each presenting unique implementation challenges, outcomes, and costs However, many studies group these diverse resources under the umbrella of "OER," complicating efforts to evaluate their specific usage and value This aggregation can obscure the distinct impacts and implementation issues associated with different types of OER To address this complexity, we propose the use of feature codes to create more specific OER subcategories, facilitating better planning and research organization Our analysis identifies seven dimensions along which OER and their applications differ, along with alternative features for each dimension.
Recent meta-analyses by SRI have highlighted that the effectiveness of technology-based learning resources is influenced by various factors, including Grain Size, Implementation Modality, Subject Area, and occasionally, Type of Learning (Means, Peters, & Zheng, 2014; U.S Department of Education, 2010).
Types of OER Research Studies
In addition to having a classification system for OER, the field would benefit from a shared language to describe the major different kinds of OER research studies In
- Cost effectiveness (requires both learning outcome and cost data)
Empirical studies measuring outcomes without a counterfactual examine similar results as impact studies; however, they do not include the necessary comparison or control group to accurately assess the impact of Open Educational Resources (OER).
Implementation Studies examine the way in which OER is implemented and include case studies and comparisons of different ways of implementing OER.
Policy Studies are descriptive studies of OER policies and policy changes without outcome data or quantitative data on implementation variables.
Research on the effects of digital Open Educational Resources (OER) should leverage existing studies on digital learning interventions The literature on digital learning effectiveness provides valuable insights and examples of OER, even if these resources are not always explicitly labeled as such.
Through our interviews with OER experts and our experience in digital learning research and evaluation, we have identified the primary research needs in the OER field.
Impact studies that utilize meaningful control groups and objective external measures of learning are essential for understanding the effectiveness of Open Educational Resources (OER) Historically, controlled studies have primarily compared outcomes between students using digital OER and those using traditional non-digital materials However, as textbook publishers increasingly integrate digital resources into their offerings, researchers must reconsider whether this comparison remains relevant for policymakers The challenge for OER developers lies in the fact that commercial publishers are now providing select resources for free, leveraging analytics and student data dashboards to generate revenue Consequently, education administrators often face a choice between OER and commercially available eBooks or learning software rather than traditional print textbooks While commercial materials typically undergo professional design and some quality review, they seldom face rigorous efficacy evaluations In contrast, OER often receives funding from foundations or government agencies that prioritize research on effectiveness Nonetheless, the academic experts creating OER may lack formal training in learning science or instructional design Moving forward, research that compares various types of OER or evaluates OER against similar commercial materials is likely to gain importance for decision-makers.
Cost impact studies that analyze both cost and effectiveness data are essential for estimating the cost per unit of desired student outcomes, such as achieving a 5% improvement in course completion rates These studies must specify the openness level of Open Educational Resources (OER), as the ability to modify and distribute instructional materials directly influences scaling potential and the overall cost per student It's crucial to realistically estimate costs in comparison conditions; for instance, without free alternatives, how many students would actually purchase a textbook at full price? While many students typically borrow or buy used textbooks at lower prices, some OER advocates may conduct cost comparisons that inaccurately assume all students pay for new books Furthermore, few studies address the transaction costs incurred by students who invest time searching for alternatives to full-priced textbooks.