Learning Analytics Assisting Universities with Student Retention

Learning Analytics: Assisting Universities with Student Retention 2015 Final Report - Part Appendices Charles Darwin University Batchelor Insti tute of Indigenous Terti ary Educati on Griffi th University Murdoch University The University of Newcastle Deborah West Charles Darwin University http://www.letstalklearninganalytics.edu.au/ Support for the production of this report has been provided by the Australian Government Office for Learning and Teaching The views expressed in this report not necessarily reflect the views of the Australian Government Office for Learning and Teaching With the exception of the Commonwealth Coat of Arms, and where otherwise noted, all material presented in this document is provided under Creative Commons AttributionShareAlike 4.0 International License http://creativecommons.org/licenses/by-sa/4.0/ The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided) as is the full legal code for the Creative Commons Attribution-ShareAlike 4.0 International License http://creativecommons.org/licenses/bysa/4.0/legalcode Requests and inquiries concerning these rights should be addressed to: Office for Learning and Teaching Department of Education and Training GPO Box 9880, Location code N255EL10 Sydney NSW 2001 2015 ISBN 978-1-76028-462-6 PRINT ISBN 978-1-76028-463-3 PDF ISBN 978-1-76028-464-0 DOCX Table of contents Appendix A - DVC Certification Appendix B – References Appendix C – Demographic Frequency Data of Academic Level Survey Participants Appendix D - Relevant OLT Projects 13 Appendix E – Impact Statement 15 Appendix F – Completed Dissemination Activities 18 Appendix G – Scheduled Dissemination Activities 19 Appendix H – High Level Framework Summary 20 Appendix I – Discussion Questions 22 Appendix J – Evaluation Report 26 Learning Analytics: Assisting Universities with Student Retention 12 Appendix A - DVC Certification Learning Analytics: Assisting Universities with Student Retention Appendix B – References ACODE (2014) Benchmarks for Technology Enhanced Learning Retrieved 12 May, 2015 from http://www.acode.edu.au/pluginfile.php/579/mod_resource/content/3/TEL_Benchmarks.p df Australian Government Department of Education and Training (2014) 2013 Student Data: Appendix - Attrition, Success and Retention Retrieved 26 May, 2015 from https://www.education.gov.au/selected-higher-education-statistics-2013-student-data Arnold, K., Lonn, S., & Pistilli, M (2014) An exercise in institutional reflection: The Learning Analytics Readiness Instrument (LARI) Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ‘14), Indianapolis, IN, USA doi:10.1145/2567574.2567621 Arnold, K & Pistilli, M (2012) Course signals at Purdue: Using learning analytics to increase student success, Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), Vancouver, British Columbia, Canada doi:10.1145/2330601.2330666 Baer, L., Norris, D., Duin, A., & Brodnick, R (2013) Crafting transformative strategies for personalized learning/analytics Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460354 Baker, R., & Yacef, K (2009) The state of educational data mining in 2009: A review and future visions Journal of Educational Data Mining, (1), 3-17 Balacheff, N., & Lund, K (2013) Multidisciplinarity vs multivocality: The case of “Learning Analytics” Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460299 Beer, C., Jones, D & Clark, D (2012) Analytics and complexity: Learning and leading for the future Proceedings Future Challenges: Sustainable Futures, ascilite, Wellington, NZ Retrieved April 13, 2015 from http://eprints.usq.edu.au/23092/2/Beer_Jones_Clark_ascilite_2012_PV.pdf Behrendt, L., Larkin, S., Griew, R., & Kelly, P (2012) Review of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander People: Final Report Canberra: Australian Government Retrieved May, 12, 2012 from http://docs.education.gov.au/documents/review-higher-education-access-and-outcomesaboriginal-and-torres-strait-islander-people-0 Bradley, D., Noonan, P., Nugent, H., & Scales, B (2008) Review of Australian Higher Education: Final Report Canberra: Department of Education, Employment and Workplace Relations Retrieved May, 12, 2015 from www.mq.edu.au/pubstatic/public/download.jsp? id=111997 Learning Analytics: Assisting Universities with Student Retention Buckingham Shum, S., & Ferguson, R (2012) Social learning analytics Educational Technology & Society, 15(3), 3-26 Camilleri, V., de Freitas, S., Montebello, M., & McDonagh‐Smith, P (2013) A case study inside virtual worlds: Use of analytics for immersive spaces Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460341 Campbell, J., DeBlois, P., & Oblinger, D (2007) Academic analytics: A new tool for a new era, Educause Review, July/August 2007 Retrieved May 12, 2015 from https://net.educause.edu/ir/library/pdf/ELIWEB0710.pdf Campbell, J., and Oblinger, D (2007) Academic analytics: White paper Educause Retrieved 12 May, 2015 from http://net.educause.edu/ir/library/pdf/pub6101.pdf Chaloux, B., & Miller, G (2013) E-learning and the transformation of higher education Chapter (pp 3-23) in G Miller, M Benke, B Chaloux, L Ragan, R Schroeder, W Smutz, Wayne, K Swan, (Eds.) Leading the e-Learning transformation in higher education: meeting the challenges of technology and distance education Stirling, VA: Stylus Publishing Chatti, M., Dyckhoff, A., Schroeder, U., & Thüs, H (2012) A reference model for learning analytics International Journal of Technology Enhanced Learning, 4(5/6), 318–331 Chowdry, H., Crawford, C., Dearden, L., Goodman, A., & Vignoles, A (2013), Widening participation in higher education: analysis using linked administrative data Journal of the Royal Statistical Society: Series A (Statistics in Society), 176: 431–457 doi: 10.1111/j.1467985X.2012.01043.x Clarke, J., Nelson, K., & Stoodley, I (2013) The place of higher education institutions in assessing student engagement, success and retention: Proceedings Higher Education Research and Development Society of Australasia, Auckland, New Zealand Retrieved May 12, 2015 from http://eprints.qut.edu.au/60024/2/60024.pdf Clow, D (2013) MOOCs and the funnel of participation Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460332 Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S (2014) Current state and future trends: A citation network analysis of the learning analytics field Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ‘14), Indianapolis, IN, USA doi:10.1145/2567574.2567585 Dawson, S., Tan, J., & McWilliam, E (2011) Measuring creative potential: Using social network analysis to monitor a learners’ creative capacity Australian Journal of Educational Technology, 27(6), 924-942 Learning Analytics: Assisting Universities with Student Retention Dekker, G., Pechenizkiy, M and Vleeshouwers, J (2009) Predicting students’ drop out: a case study Proceedings of the 2nd International Conference on Educational Data Mining (EDM '09), Cordoba, Spain http://www.educationaldatamining.org/EDM2009/uploads/proceedings/dekker.pdf Diaz, D (2000) Comparison of student characteristics, and evaluation of student success, in an online health education course Unpublished doctoral dissertation, Nova Southeastern University Retrieved 12 May 2015 from http://home.earthlink.net/~davidpdiaz/LTS/pdf_docs/dissertn.pdf Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) (2012a) 2011 full year higher education student statistics Appendix – Equity Groups Canberra: Australian Government Retrieved 18 June, 2013 from http://www.innovation.gov.au/HigherEducation/HigherEducationStatistics/StatisticsPublicati ons/Pages/2011StudentFullYear.aspx Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) (2012b) 2011 full year higher education student statistics Appendix - attrition, progress and retention Canberra: Australian Government Retrieved 14 June 2013 from http://www.innovation.gov.au/HigherEducation/HigherEducationStatistics/StatisticsPublicati ons/Pages/2011StudentFullYear.aspx Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) (2013a) Students: selected higher education statistics 2012 (Appendix 2: equity data) Canberra: DIICCSRTE Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) (2013b) Moving to an enhanced indicator of higher education students’ socio-economic status Canberra: DIICCSRTE EDUCAUSE (2010) Things you should know about analytics Retrieved May 12, 2015 from http://www.educause.edu/ir/library/pdf/ELI7059.pdf EduTech Wiki (2013) Learning analytics Retrieved 14 June, 2013 from http://edutechwiki.unige.ch/en/Learning_analytics#Introduction Ferguson, R., (2012) Learning analytics: drivers, developments and challenges International Journal of Technology Enhanced Learning, 4(5/6), 304–317 Fiaidhi, J (2014) The next step for learning analytics IT PRO/Institute of Electrical and Electronics Engineers, September/October 2014 Retrieved May 12, 2015 from https://www.computer.org/csdl/mags/it/2014/05/mit2014050004.pdf Frankola, K (2001) Why online learners drop out Retrieved May 12, 2015 from http://www.workforce.com/articles/why-online-learners-drop-out Learning Analytics: Assisting Universities with Student Retention Gallagher, M (2014) Micro-economic reform of the Australian higher education industry: Implications of the Abbott Government’s Budget of 13 May 2014 Address to the EduTECH Higher Education Leaders Congress, Brisbane, Australia Retrieved May 10, 2015 from https://go8.edu.au/sites/default/files/docs/article/edutech_presentation_-_4_june_2014pdf_version.pdf Gašević, D., Mirriahi, N., Long, P., & Dawson, S (2014) Editorial – inaugural issue of the Journal of Learning Analytics Journal of Learning Analytics, 1(1), 1–2 Goldstein, P., & Katz, R (2005) Academic analytics: The uses of management information and technology in higher education ECAR Research Study, Vol Retrieved May 10, 2015 from http://www.educause.edu/library/resources/academic-analytics-uses-managementinformation-and-technology-higher-education Hanna, D (2000) Higher education in an era of digital competition: choices and challenges Madison, WI: Atwood Publishing Heathcote, E., & Dawson, S (2005) Data mining for evaluation, benchmarking and reflective practice in a LMS Proceedings e-Learn 2005: World conference on e-Learning in corporate, government, healthcare & higher education, Vancouver, Canada Retrieved April 13, 2015 from http://eprints.qut.edu.au/2368/1/2368.pdf Holman, C., Aguilar, S., & Fishman, B (2013) GradeCraft: what can we learn from a gameinspired learning management system Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460350 James, R., Krause, K-L., & Jennings, C (2010) The first year experience in Australian universities: Findings from 1994 to 2009 Centre for the Study of Higher Education, Melbourne: University of Melbourne Retrieved May 10, 2015 from http://www.cshe.unimelb.edu.au/research/experience/docs/FYE_Report_1994_to_2009.pdf Kennedy, G., Corrin, L., Lockyer, L., Dawson, S., Williams, D., Mulder, R., Khamis, S & Copeland, S (2014) Completing the loop: returning learning analytics to teachers Proceedings Rhetoric and Reality: Critical perspectives on educational technology, 31st ascilite Conference, Dunedin, New Zealand Retrieved May 12, 2015 from http://www.academia.edu/12074024/Completing_the_loop_returning_learning_analytics_t o_teachers Kift, S (2009) Articulating a transition pedagogy to scaffold and to enhance the first year student learning experience in Australian higher education: Final report ALTC Senior Fellowship Retrieved April 14, 2015 from http://www.altc.edu.au/resource-transitionpedagogy-report-qut-2009 Learning Analytics: Assisting Universities with Student Retention Kift, S., Nelson, K., & Clarke, J (2010) Transition pedagogy: A third generation approach to FYE - A case study of policy and practice for the higher education sector The International Journal of the First Year in Higher Education, 1(1), 1-20 Koshy, P (2014) Student equity performance in Australian higher education: 2007 to 2012 National Centre for Student Equity in Higher Education (NCSEHE), Perth: Curtin University Macfadyen, L., & Dawson, S (2010) Mining LMS data to develop an ‘early warning system’ for educators: A proof of concept Computers & Education, 54(2), 588-599 Mirriahi, N., & Dawson, S (2013) The pairing of lecture recording data with assessment scores: A method of discovering pedagogical impact Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460331 Mirriahi, N., Gašević, D., Long, P., & Dawson, S (2014) Scientometrics as an important tool for the growth of the field of learning analytics Journal of Learning Analytics, 1(2), 1‐4 Nelson, K., Clarke, J., Stoodley, I., & Creagh, T (2014) Establishing a framework for transforming student engagement, success and retention in higher education institutions Final Report 2014, Canberra, Australia: Australian Government Office for Learning & Teaching Nelson, K., Kift, S., & Clarke, J (2012) A transition pedagogy for student engagement and first-year learning, success and retention In I Solomonides, A Reid & P Petocz (Eds.) Engaging with learning in higher education Faringdon, UK: Libri Publishing Norris, D., Baer, L & Offerman, M (2009) A national agenda for action analytics National Symposium on Action Analytics, St Paul, MN Retrieved May 12, 2015 from http://lindabaer.efoliomn.com/uploads/settinganationalagendaforactionanalytics101509.pdf Norris, D., & Baer, L (2012) A toolkit for building organizational capacity for analytics Washington, DC: Strategic Initiatives, Inc Retrieved 12 May, 2015 from https://docs.google.com/a/student.rmit.edu.au/file/d/0B0CqeHHcpYzMEZPY2hCRHpyLVU/edit Norris, D., & Baer, L (2013) Building organizational capacity for analytics Boulder, CO: Educause Retrieved 10 May, 2015 from https://net.educause.edu/ir/library/pdf/PUB9012.pdf Ochoa, X., Suthers, D., Verbert, K., & Duval, E (2014) Analysis and reflections on the third Learning Analytics and Knowledge Conference (LAK 2013) Journal of Learning Analytics, 1(2), 5‐22 Pechenkina, E & Anderson, I (2011) Background paper on Indigenous Australian higher education: trends, initiatives and policy implications Retrieved May 12, 2015 from www.innovation.gov.au/HigherEducation/IndigenousHigherEducation/ReviewOfIndigenousH igherEducation/Pages/Research.aspx Learning Analytics: Assisting Universities with Student Retention Raca, M., Tormey, R., & Dillenbourg, P (2014) Sleeper’s lag: A study on motion and attention Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ‘14), Indianapolis, IN, USA doi:10.1145/2567574.2567581 Rivera, J., and Rice, M (2002) A comparison of student outcomes & satisfaction between traditional & web based course offerings Online Journal of Distance Learning Administration, 5(3) Romero, C., & Ventura, S (2010) Educational data mining: A review of the state-of-the-art IEEE Transaction on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 40(6), 601-618 Siemens G (2012) Learning analytics: Envisioning a research discipline and a domain of practice, Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), Vancouver, British Columbia, Canada http://learninganalytics.net/LAK_12_keynote_Siemens.pdf Siemens, G., Dawson, S., & Lynch, G (2013) Improving the productivity of the higher education sector: policy and strategy for systems-level deployment of learning analytics Society for Learning Analytics Research/Australian Government Office for Learning and Teaching Retrieved 14 April, 2014 from http://www.itl.usyd.edu.au/projects/SoLAR_Report_2014.pdf Siemens, G & Long, P (2011) Penetrating the Fog: Analytics in learning and education EDUCAUSE Review, 46(4) July/August Retrieved 20 May, 2013 from http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education Suthers, D., & Verbert, K (2013) Learning analytics as a “middle space.” Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium doi:10.1145/2460296.2460298 Thomas, L (2002).Widening participation in post‐compulsory education Education & Training, 44(4/5), 241 – 242 doi:10.1108/et.2002.44.4_5.241.2 Tinto, V (2009) Taking student retention seriously: rethinking the first year of university ALTC FYE Curriculum Design Symposium, Queensland University of Technology, Brisbane, Australia Retrieved May 8, 2015 from http://www.fyecd2009.qut.edu.au/resources/SPE_VincentTinto_5Feb09.pdf Tsui, J., Lau, J., & Shieh, L (2014) Septris and SICKO: Implementing and using learning analytics and gamification in medical education Educause Learning Initiative Retrieved May 10, 2015 from https://net.educause.edu/ir/library/pdf/ELIB1401.pdf Universities Australia (2013) A smarter Australia: An agenda for Australian higher education 2013-2016 Canberra, ACT: Universities Australia Retrieved April 21, 2015 from Learning Analytics: Assisting Universities with Student Retention 10 Appendix E – Impact Statement Anticipated Changes at: Impact on: Project Completion Team Members    Project team members immediate Students2 Spreading the Word  Project members all have influence within their institution around the development of learning analytics infrastructure, strategies and/or implementation Some team members have gained additional input to the development of learning analytics in their institution Team members have influenced vendors roadmap and development of proprietary analytics packages Presentations have been requested and completed in Malaysia and Singapore  Article for special edition of journal requested by editors  One external months postcompletion  University of Newcastle to scope a studentfocused analytics solution targeting retention outside of the larger MIS and from within Blackboard 12 months postcompletion 24 months postcompletion1  University of Newcastle implementation of student-focused analytics solution (for students) Follow up ‘use case’ workshop run at one conference to provide examples of reports and gather further feedback  Identified applicability of Learning analytics is at a very early stage and a fast moving field so it is difficult to project how this work would be useful in this time frame beyond being something that others would build on only one of the project team members has a direct teaching role; most of us are located at a central level so are likely to influence students in our institution more broadly Learning Analytics: Assisting Universities with Student Retention 15 Anticipated Changes at: Impact on: Project Completion institution has engaged in a workshop to consider use cases to inform the development of learning analytics  additional institutions have requested workshops  Request to work with Malaysian universities to extend this study in Malaysia  Several papers are scheduled at various conferences in the coming months Narrow  Learning analytics opportunistic reports on adoption in retention have partner been trialled by institutions some academics in (impacting on partner some students) institution although they are available to all academics Narrow  Use case systemic workshops have adoption (in been run in two partner partner institutions institutions impacting on  Use cases have all students) been used to inform the development of reports at partner institution  Framework has been used to inform development (or move forward on learning analytics months postcompletion 12 months postcompletion use case workshops are run in outside institutions 24 months postcompletion framework in Malaysia  Framework is applied robustly in institutions to progress learning analytics initiatives  Advancement of the use of learning analytics for retention in partner institutions  The use of learning analytics reports is included in policy procedure in at least one partner institution Learning Analytics: Assisting Universities with Student Retention 16 Anticipated Changes at: Impact on: Project Completion months postcompletion 12 months postcompletion 24 months postcompletion in partner institutions Broad opportunistic adoption (in outside institutions impacting on some students) Broad systemic adoption (in outside institutions impacting on all students)  One LMS vendor has done initial mapping of academic use cases to available reports in their product  LMS vendor has adopted technical implementation approach by one institution as a standard  Use cases have been used as model for implementation in institutions  Increase in the number of academics using learning analytics reports to assist with student retention  Framework utilised by a number of outside institutions to inform the ongoing development of learning analytics for retention  Potential addition of reports to vendors product  Additional projects to extend the work of the project in the sector (e.g next iteration of another version of a framework) Learning Analytics: Assisting Universities with Student Retention     17 Additional projects to extend the work on the project Learning analytics becomes a core element of retention plan reporting and/or informing of retention strategies Application of learning analytics work to various disciplines Learning analytics emerges as a ubiquitous Appendix F – Completed Dissemination Activities Event Date Event Title, Location Brief Description of Event Number of participants 10 June 2014 OLT Conference Sydney Workshop on Learning Analytics with Prof Phil Long and Prof Gregor Kennedy >80 July 2014 First Year Experience Conference, Darwin Joint paper with Ms Cassandra Colvin (UniSA team) >100 21 August Learning Analytics presentation, CDU Presentation to introduce learning analytics to the academic community >50 26-28 August 2014 Blackboard Learning and Teaching Conference, Gold Coast Workshop on Learning Analytics Use Cases >80 October Nanyang Technological University, Singapore (webinar) Presentation on learning analytics for staff PD in learning and teaching >200 October 2014 Flinders University Use case workshop with Flinders staff >20 11 November IRU, MRUN workshop, Universiti of Malaya, KL Presentation on project to IRU/MRUN members >80 18 November 2014 Charles Darwin University Use case workshop with CDU staff >20 20 November 2014 Australian Learning Analytics Summer Institute Panel session >80 23-26 November 2014 ASCILITE Conference, Dunedin NZ Joint Poster Presentation on Project with UniSA team 13 February 2015 Newcastle University Use case workshop with Newcastle staff 16-20 March 2015 Learning Analytics Knowledge Conference, USA Joint poster presentation on Project with UniSA team April 2015 Let’s Talk Learning Analytics National Forum Project Forum Learning Analytics: Assisting Universities with Student Retention >60 >150 18 Appendix G – Scheduled Dissemination Activities Event Date Event Title, Location Brief Description of Event 16-19 June 2015 35th Annual Conference of the Society for Teaching and Learning in Higher Education (STLHE), Vancouver, Canada Conference presentation on project findings 17-19 June 2015 10th eLearning Forum, Singapore Conference paper presentation on project findings 22-24 June, 2015 AACE EdMedia 2015: World Conference on Educational Media and Technology, Montreal Canada Best Practice session on project findings 6-9 July 2015 HERDSA Conference, Melbourne Conference paper presentation on project findings 25-27 August 2015 Blackboard Teaching and Learning Conference Follow up workshop on use cases 27 -30 October 2015 ISSOTL conference Melbourne Paper presentation (if accepted) Learning Analytics: Assisting Universities with Student Retention 19 Appendix H – High Level Framework Summary See Next Page The framework can also be downloaded as a stand-alone pdf from here Learning Analytics: Assisting Universities with Student Retention 20 Learning Analytics: Assisting Universities with Student Retention 21 Appendix I – Discussion Questions See Next Three Pages The discussion questions can be downloaded as a stand-alone pdf from here Learning Analytics: Assisting Universities with Student Retention 22 Learning Analytics: Assisting Universities with Student Retention 23 Learning Analytics: Assisting Universities with Student Retention 24 Learning Analytics: Assisting Universities with Student Retention 25 Appendix J – Evaluation Report Evaluation Reflections Charles Darwin University – SP13-3268 Learning Analytics: assisting universities with student retention Background The overall aim of this project was to develop a framework which cross references the tools available through learning management systems and learning analytics with variables and actions to improve retention of at risk students The project addressed this aim through an extended literature review, two national surveys, in-depth interviews and case studies The focus of this project is on the use of learning analytics by higher education institutions in Australia, for student retention The data gathered was related to academic and executive views and uses of learning analytics The guiding focus of the evaluation was to determine if the project’s aims were achieved, and outcomes delivered, within budget and on time The intended outcomes were to: - Gain a better understanding of the maturity of the learning analytics implementation and application within Australia - Explore and identify the range of ways analytics are being used in the sector - Increase awareness of the role, function and potential uses of analytics for student retention - Increase awareness of the elements that need to be considered for learning analytics implementation These outcomes were intended to be accomplished via the following demonstrable outputs:  Overview of use/potential use and limitations of learning analytics  Framework for evaluating use of analytics for student retention  Case studies on the use of the framework to evaluate analytics  National forum Learning Analytics: Assisting Universities with Student Retention 26 Evidence The first interactions between the Project and Evaluation Teams were at the OLT workshop for the 2013 Strategic Commissioned Projects, conducted in April 2014 Within the Learning Analytics Cluster, there were two projects, with this project being led by Associate Professor Deborah West from Charles Darwin University, with team members Henk Huijser, David Heath, Alf Lizzio, Carol Miles, Danny Toohey, Bill Searle and Jürg Bronnimann In order to identify that the project’s aims were achieved and outcomes delivered both formative and summative evaluation strategies were utilised The Evaluation team was provided with access to the key documentation from the project team and were included in significant project team communications In addition, a member of the evaluation team was a participant in virtual and face to face project team and reference group meetings Specifically, an evaluation team member participated in the day project team meeting in September 2014 Throughout the lifecycle of the project the evaluation team provided input and advice The Evaluator found several key factors that contributed to the successful achievement of the project aim and goals These factors include: ● Regular meetings of the project team with the Evaluator and Reference Group from the beginning of the project, which were well supported by project plan updates and reports on activities This ensured that the team were provided formative feedback to further enhance the proposed project outcomes ● Active and sustained communications between project team members from the partner institutions ● Strong project management, as demonstrated by extensive and appropriate documentation and insightful input to the project from the project manager ● Appropriate knowledge of institutional structures and priorities, ensuring that the activities undertaken related to institutional strategies and requirements in this emerging field ● Establishment and maintenance of a genuine team approach to the project This was exemplified in the breadth of contribution made by project team members at national forum ● Diversity of skill set in the project team, which ensured a range of perspectives, breadth of analytical skills, and variation of insight into the project communication requirements Learning Analytics: Assisting Universities with Student Retention 27 Project Management It has been documented that effective project management has the following elements: ● ● ● ● ● Identifying requirements, Establishing clear and achievable outcomes, Balancing the competing demands for quality, scope, time and cost, Managing the expectations of various stakeholders, and Adapting plans to overcome challenges From a Project Management perspective, the project was well managed and all stakeholder groups were involved There was effective and significant communication with all members of the project team and involvement of the reference group assisted with project execution and promulgation of project outputs From the outset it was evident that this was an active, enthusiastic and well-led project with clear project goals and strategies The leadership from Associate Professor Deborah West along with the capable and conscientious oversight of David Heath as Project Manager, were key factors in the success of this project This facilitated a collaborative team environment with a clear and ongoing focus on the project deliverables Inclusion of a member of the Evaluation Team in key project discussion provided formative evaluation and input throughout the project and was facilitated by the project leader in a positive and generative manner Achievement of Outcomes The key summative evaluation questions centred on whether the project was able to identify the usage of learning analytics tools, generate a meaningful framework, establish documented case studies and undertake the national forum There were over 353 valid survey responses from academic staff, and the data within these surveys were able to be analysed and mapped to identify the context and tensions that are currently present in the use of learning analytics to address student retention Institutional data was derived from 23 interviews across 15 institutions, and the completion of 24 institutional level surveys (22 from Australia and from New Zealand) It should be noted that this extent of response at sector level is a positive reflection on the execution of the institutional level survey by the project team This was supplemented with five case studies Survey inform also provided insight into the current state of LMS use in the sector Learning Analytics: Assisting Universities with Student Retention 28 The final phase of the project featured the national forum, which included input from the University of South Australia Learning Analytics OLT Strategic Grant project team The national forum was a well attended (148 attendees) event, that generated informed discussion on the Learning Analytics field in general, and the project outputs specifically The national forum in particular was used to elicit the final input to the institutional Learning Analytics framework The forum and overall project is http://www.letstalklearninganalytics.edu.au/ supported by the website at: To date (June 2015) there have been 13 workshops/presentations undertaken, addressing and disseminating the project and its outcomes Over 900 attendees have participated at these events The project team have reported that a further presentations are scheduled by the end of 2015 Summary The project activities, and in particular the national forum, ensured that a large number of stakeholders (institutions, academic managers and educators) were not only party to the collection of the sector data, but also engaged with the rapidly changing agenda of Learning Analytics, and its specific application to student retention The form of the surveys and the associated data provide important insights into the tension between institutional level managerial use of learning analytics data and the explicitly educational use of the data The variation in perspectives identified by the project, and the means by which a whole-of-institution capability is advanced, is a key insight to the effective maturation of learning analytics in the sector The institutional framework is now a tested tool for extending and informing the necessary dialogue, at multiple levels, within institutions, and it can be expected that the framework will be actively used across the sector Overall the evaluators appreciated the opportunity to work with the capable and enthusiastic project team The extent of interest in the forum, the active nature of the forum, and the ongoing positive inter-institutional dialogue in the weeks following its conclusion is reflective of the value of this project Learning Analytics: Assisting Universities with Student Retention 29 ... here Learning Analytics: Assisting Universities with Student Retention 22 Learning Analytics: Assisting Universities with Student Retention 23 Learning Analytics: Assisting Universities with Student. .. downloaded as a stand-alone pdf from here Learning Analytics: Assisting Universities with Student Retention 20 Learning Analytics: Assisting Universities with Student Retention 21 Appendix I – Discussion... Evaluation Report 26 Learning Analytics: Assisting Universities with Student Retention 12 Appendix A - DVC Certification Learning Analytics: Assisting Universities with Student Retention Appendix