This article analyses the CDIO selfevaluation rubrics of which each of the rubrics is associated to the CDIO standards toward evaluating the business administration training programme at Ha Tinh University in Vietnam. The analysis is done based on our own experience in doing selfevaluation of the business administration training programme at Ha Tinh University. The structure of the original rubric is kept (i.e. each standard is evaluated on a 05 point scale). Outcomes of the analysis are proposals for clarifications, precisions and extensions of the rubrics for better training quality assuarance of such training programme.
Journal of Critical Reviews ISSN- 2394-5125 Vol 7, Issue 3, 2020 Review Article EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELF-EVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM Ngoc Hai Tran1*, Nhan Van Phan2 Senior lecturer, Institute of Continuing Education, Ha Tinh University, No 447, 26 March St., Ha Tinh city, Vietnam - PhD candidate, Vietnam National Institute of Educational Sciences, No 101, Tran Hung Dao St., Ha Noi, Vietnam; E-mail of the corresponding author: haingoc74@gmail.com; Asso.Prof., Senior lecturer, Vietnam National Institute of Educational Sciences, No 101, Tran Hung Dao St., Ha Noi, Vietnam; Email: thucnghiem106@yahoo.com Received: 22.11.2019 Revised: 23.12.2019 Accepted: 12.01.2020 Abstract This article analyses the CDIO self-evaluation rubrics of which each of the rubrics is associated to the CDIO standards toward evaluating the business administration training programme at Ha Tinh University in Vietnam The analysis is done based on our own experience in doing self-evaluation of the business administration training programme at Ha Tinh University The structure of the original rubric is kept (i.e each standard is evaluated on a 0-5 point scale) Outcomes of the analysis are proposals for clarifications, precisions and extensions of the rubrics for better training quality assuarance of such training programme Keywords: Self-evaluation, quality assurance, 12 CDIO Standards, business administration © 2019 by Advance Scientific Research This is an open-access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/) DOI: http://dx.doi.org/10.31838/jcr.07.03.68 INTRODUCTION Assessment of universities and training programmes are in focus for relevant authorities at higher education institutions in the world Accreditation standards like ABET, EUR-ACE or CEAB are popular for universities A number of papers from the CDIO conferences have focused on alignment of the CDIO self-evaluation and other accreditation systems (see, for example, Cloutier, Hugo, & Sellens, 2011; Dai et al., 2017; Ha, Nguyen, Nayyar & Liu, 2019; Malmqvist, 2016) evaluation concept, where a higher education institution or a training programme could self-evaluate how well it is doing on a six-point scale In Vietnam, improving the quality of higher education has been in focus for the last few decades (Tran, Hallinger & Truong, 2018; Vu & Phung, 2015) The Ministry of Education and Training of Vietnam provided the main guidelines for increasing the competitiveness of Vietnam higher education They called for improvements in the quality of education and emphasize the importance of Higher Education Institutions to provide education that answers to the competence requirements of working life and employers’ increasing demands More recently, improving the education and training system quality has been set as a key target in Vietnam’s strategy to become a smart, sustainable and developed economy (Do, 2015; Tran et al., 2018; Tran et al., 2020) In this process, selfevaluation plays a key role in higher education institutions The process of creating the CDIO self-evaluation rubric for the business administration training programme at Ha Tinh University was done in 2015 - 2018 We took part in the final evaluation of the rubric used for the self-evaluation of the programe In the past, the CDIO organization discussed if accreditations should be promoted This was not found to be relevant; the way forward selected is to make a self- For the last recent years, the CDIO initiatives have been asking higher education institutions and training programmes to a self-evaluation in order to support the continued improvement of the CDIO implementation at the institution/programme (Ha et al., 2019; Truong, Ha & Le 2019) It has become a de facto action to perform when applying for the membership of CDIO Now three years have passed and we have had real experience with using the rubric for several times and several programmes at our three institutions In this article we will evaluate the self-evaluation from the view of a business administration programme responsible - are the descriptions understandable, are the descriptions on the same level, we need more extensions, clarifications, precisions… THE GENERAL GUIDELINES FOR THE SELF-EVALUATION OF BUSINESS ADMINISTRATION TRAINING PROGRAMME AT HA TINH UNIVERSITY The self-evaluation is done on a six point scale There is a general idea that the evaluation of the levels of all 12 standards should be compliant to Table Table Generic description of the six levels Level Rubric Evidence related to the standard is regularly reviewed and used to make improvements Journal of critical reviews 354 EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM There is documented evidence of the full implementation and impact of the standard across programme components and constituents Implementation of the plan to address the standard is underway across the pro- gram components and constituents There is a plan in place to address the standard There is an awareness of need to adopt the standard and a process is in place to address it There is no documented plan or activity related to the standard All twelve standards have a six-levelled hierarchy The hierarchy is intended to be defined in such a way that being on level n also implies that the requirements for levels 0,1,…, n-1 is met EVALUATION OF THE RUBRICS OF THE TWELVE STANDARDS In the following subsections we will evaluate and discuss each of the rubrics for the twelve standards A higher education institution or a programme typically looks for arguments for the given level and the main benefit is not the level itself but the insights one gets from doing the evaluation Rubric of standard - The context The rubric of standard is consistent with the general rubric and we not see any need to propose changes to the rubric Level Table Rubric of standard Original rubric Suggested change to the rubric Evaluation groups recognize that CDIO is the NO CHANGE PROPOSED context of the business administration programme and use this principle as a guide for continuous improvement There is documented evidence that the CDIO principle NO CHANGE PROPOSED is the context of the business administration proramme and is fully implemented CDIO is adopted as the context for the business NO CHANGE PROPOSED administration programme and is implemented in one or more years of the programme There is an explicit plan to transition to a CDIO context NO CHANGE PROPOSED for the business administration programme The need to adopt the principle that CDIO is the NO CHANGE PROPOSED context of business administration training is recognized and a process to address it has been initiated There is no plan to adopt the principle that CDIO is the NO CHANGE PROPOSED context of business administration training programme Rubric of standard - CDIO Syllabus Outcomes The learning outcomes described in the standard are learning outcomes for a given business administrationprogramme In our country, these are given by the law and as such the institutions not have any control over these It is naturally possible to make them more specific, but they have to be compliant with the overall learning outcomes What is normally seen as a good idea - and a help for students, lecturers and course designers is learning outcomes on a course level and on a semester/term level, such that there will be alignment Level between the learning outcomes, the teaching and the exam We therefore suggest that the rubrics reflect this by changing all programme learning outcomes to course and/or programme learning outcomes When doing learning outcomes, an important issue is the depth of learning that a student needs to have; there is a big difference between being able to recite what a differential equation is, being able to apply it or create new theory relating to differential equations We suggest that this is needed to be on level Table Rubric of standard Original rubric Suggested change to the rubric Internal and external groups regularly review Internal and external groups regularly and revise programme learning outcomes, review and revise course and programme based on changes in stakeholder needs learning outcomes … Programme learning outcomes are aligned with Programme as well as course learning institutional vision and mission, and levels of out- comes are aligned with institutional proficiency are set for each outcome vision and mission Programme learning outcomes are vali- dated Course and/or programme learning with key programme stakeholders, including outcomes are validated … and levels of faculty, students, alumni, and industry proficiency are set for each outcome representatives A plan to incorporate explicit statements of A plan to incorporate explicit statements programme learning outcomes is accepted by of course and/or programme learning programme leaders, business administration outcomes… Journal of critical reviews 355 EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM faculty, and other stakeholders The need to create or modify programme learning outcomes is recognized and such a process has been initiated There are no explicit programme learning outcomes that cover knowledge, personal and interpersonal skills, and product, process and system building skills Rubric of standard - Integrated curriculum We propose some changes to the rubric of standard In level 0, we want to emphasize the curriculum and not just refer to the whole programme In level 2, we raise the requirement Level 4 There are no explicit course and/or programme learning outcomes … instead of an approval of a curriculum plan we propose an approval of the integrated curriculum The programme reaches level when the integrated curriculum is in use Table Rubric of standard Original rubric Suggested change to the rubric Internal and external stakeholders regularly review NO CHANGES NEEDED the integrated curriculum and make recommendations and adjustments as needed There is evidence that personal, interpersonal, NO CHANGE NEEDED product, process, and system building skills are addressed in all courses responsible for their implementation Personal, interpersonal, product, process, The approved integrated curriculum is in and system building skills are integrated into one or use more years in the curriculum A curriculum plan that integrates disciplinary learning, The curriculum that integrates learning personal, interpersonal, product, process, and system outcomes of personal, interpersonal, building skills is approved by appropriate groups product, process, and system building skills is approved The need to analyze the curriculum is recognized and NO CHANGES PROPOSED initial mapping of disciplinary and skills learning outcomes is underway There is no integration of skills or mutually supporting The curriculum has no courses that integrate disciplines in the programme learning outcomes of personal, interpersonal, product, process, and system building skills Rubric of standard - Introduction to the training programme The rubric of standard has some inconsistencies in levels one and two We propose modification according to Table In Level The need to create or modify course and/or programme learning outcomes … level one, the need for an introductory course is recognized In level two, the plan for the introductory course is done and the implementation has been initiated Table Rubric of standard Original rubric Suggested change to the rubric The introductory course is regularly evaluated and NO CHANGE PROPOSED revised, based on feedback from students, instructors, and other stakeholders There is documented evidence that students have NO CHANGE PROPOSED achieved the intended learning outcomes of the introductory business administration course An introductory course that includes learning NO CHANGE PROPOSED experiences and introduces essential personal and interpersonal skills has been implemented A plan for an introductory business administration A plan for an introductory business course introducing a framework for practice has been administration course introducing a approved framework for practice has been approved and a process to implement the plan has been initiated The need for an introductory course that provides the The need for an introductory course that framework for business administration practice is provides the framework for business recognized and a process to address that need has been administration practice is recognized initiated There is no introductory business administration NO CHANGE PROPOSED course that provides a framework for practice and introduces key skills Journal of critical reviews 356 EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM Rubric of standard – Design - Implement experiences We don’t propose any changes to the rubric of standard The rubric is consistent and understandable Level Table Rubric of standard Original rubric Suggested change to the rubric The design-implement experiences are regularly NO CHANGE PROPOSED evaluated and revised, based on feedback from students, instructors, and other stake-holders There is documented evidence that students have NO CHANGE PROPOSED achieved the intended learning outcomes of the design-implement experiences At least two design-implement experiences of NO CHANGE PROPOSED increasing complexity are being implemented There is a plan to develop a design-implement NO CHANGE PROPOSED experience at a basic and advanced level A needs analysis has been conducted to iden- tify NO CHANGE PROPOSED opportunities to include design-implement experiences in the curriculum There are no design-implement experiences in the NO CHANGE PROPOSED business administration programme Rubric of standard – Business administration workspaces If we suppose that there is no need to remodel or build anything new in order to have satisfactory business administration workspaces, then it is difficult to score higher than level for standard 6, if we were to strictly follow the rubric This is not an unrealistic scenario for business administration, where normal university buildings could provide adequate workspaces Therefore, we suggest that the rubric of standard is changed according to Table Table Rubric of standard Original rubric Suggested change to the rubric Internal and external groups regularly evaluate the NO CHANGE PROPOSED impact and effectiveness of workspaces on learning and provide recommendations for improving them Business administration workspaces fully support NO CHANGE PROPOSED all components of hands-on, knowledge, and skills learning Plans are being implemented and some new or If business administration workplaces initially were remodelled spaces are in use deemed unsatisfactory, plans are now being implemented and some new or remodeled spaces are in use Plans to remodel or build additional workspaces If business administration workplaces are deemed have been approved by the appropriate bodies unsatisfactory, plans to remodel or build additional business administration workspaces have been approved by the appropriate bodies The need for business administration workspaces NO CHANGE PROPOSED to support hands-on, knowledge, and skills activities is recognized and a process to address the need has been initiated Business administration workspaces are NO CHANGE PROPOSED inadequate or inappropriate to support and encourage hands-on skills, knowledge, and social learning Level Rubric of standard - Integrated learning The rubric of standard correlates in general with the generic rubric, but there is a discrepancy at level where full implementation is expected although the generic rubric dictates that implementation should be underway In addition, the formulation in relation to level of the rubrics makes it Level unclear if one is supposed to show evidence of the positive effects of integrated learning in general or specifically the actual implementations of integrated learning at the current study programme We suggest that this is clarified according to Table Table Rubric of standard Original rubric Suggested change to the rubric Courses are regularly evaluated and NO CHANGE PROPOSED revised regarding their integration of learning outcomes and activities There is evidence of the impact of integrated There is evidence of the impact of the learning experiences across the curriculum implementation of integrated learning experiences across the curriculum Journal of critical reviews 357 EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM Integrated learning experiences are implemented Integrated learning experiences are being in courses across the curriculum implemented in courses across the curriculum Course plans with learning outcomes and NO CHANGE PROPOSED activities that integrate personal and interpersonal skills with disciplinary knowledge has been approved Course plans have been benchmarked NO CHANGE PROPOSED with respect to the integrated curriculum plan There is no evidence of integrated learn- ing of NO CHANGE PROPOSED disciplines and skills Rubric of standard - Active learning The rubric of standard roughly correlates with the generic rubric of Table However, the formulation of the rubric regarding active learning methods makes it unclear if one should review the impact of active learning in general or if it is the current implementation of active learning methods that should be quality assured There is, of course, a wide acceptance of the fact that active learning enhances student learning, but the issue here is to establish if the current Level Table Rubric of standard Original rubric Suggested change to the rubric Internal and external groups regularly review Internal and/or external groups regularly review the impact of active learning methods and make the implementation of active learning activities recommendations for continuous improvement across the curricula and make recommendations for continuous improvement There is documented evidence of the impact of There is documented evidence that active learning active learning methods on has been implemented all across student learning the curriculum Active learning methods are being implemented NO CHANGE PROPOSED across the curriculum There is a plan to include active learning NO CHANGE PROPOSED methods in courses across the curriculum There is an awareness of the benefits of active There is an awareness of the benefits of active learning, and benchmarking of active learning learning and a process is in place to introduce it methods in the curriculum is in process across the curricula There is no evidence of active experiential NO CHANGE PROPOSED learning methods Rubric of standard - Enhancement of faculty competence The original formulation of the rubric of standard is difficult to relate to if we assume that the staffs initially has a high competence in personal and interpersonal skills etc The Level programme has components of active learning incorporated into its courses One further question is the need for both external and internal groups to meet (level five) - what additional value does it give to have both internal and external groups; could the same level of quality be achieved with just e.g internal (to the study programme) groups to meet? These uncertainties are visible in levels one, four and five of the rubric We suggest that the rubric for standard is changed according to Table formulation of levels two and three assumes that there is no such competence present amongst the staff We suggest that the rubric for standard is changed according to Table 10 Table 10 Rubric of standard Original rubric Suggested change to the rubric Faculty competence in personal, interpersonal, product, NO CHANGE PROPOSED process, and system building skills is regularly evaluated and updated where appropriate There is evidence that the collective faculty is competent NO CHANGE PROPOSED in personal, interpersonal, product, process, and system building skills The collective faculty participates in faculty Where needed, the faculty participates in development in personal, interpersonal, product, faculty development in personal, process, and system building skills interpersonal, product, process, and system building skills There is a systematic plan of faculty development in Where needed, there is a systemat- ic plan of personal, interpersonal, product, process, and system faculty development in personal building skills interpersonal, product, process, and system building skills There is an awareness of the benefits of active learning, NO CHANGE PROPOSED and benchmarking of active learning methods in the curriculum is in process There is no evidence of active experiential learning NO CHANGE PROPOSED methods Rubric of standard 10 - Enhancement of faculty teaching competence Journal of critical reviews The rubric of standard 10 suffers from the same underlying assumption as the rubric of standard 9: If the staff already have 358 EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM a high competence in teaching we can not assess the level to a value of without firstly creating a plan etc We suggest that Level Table 11 Rubric of standard 10 Original rubric Suggested change to the rubric Faculty competence in teaching, learning, and NO CHANGE PROPOSED assessment methods is regularly evaluated and updated where appropriate There is evidence that the collective faculty is NO CHANGE PROPOSED competent in teaching, learning, and assessment methods Faculty members participate in faculty Where needed, faculty members participate in development in teaching, learning, and faculty development in teaching, learning, and assessment methods assessment methods There is a systematic plan of faculty development Where needed, a systematic plan of faculty in teaching, learning, and assessment methods development in teaching, learning, and assessment methods is developed A benchmarking study and needs analysis of NO CHANGE PROPOSED faculty teaching competence has been conducted There are no programmes or practices to enhance NO CHANGE PROPOSED faculty teaching competence Rubric of standard 11 - Assessment In the rubric of standard 11, there is an inconsistency with in relation to the generic rubric as presented in Table The rubric of standard 11 states at level that a full implementation Level 4 of assessment methods is needed while the generic rubric states that implementation should be underway for a level agreement with the rubric We suggest that the rubric is changed according to Table 12 Table 12 Rubric of standard 11 Original rubric Suggested change to the rubric Internal and external groups regularly review the use of NO CHANGE PROPOSED learning assessment methods and make recommendations for continuous improvement Learning assessment methods are used effectively in NO CHANGE PROPOSED courses across the curriculum Learning assessment methods are implemented across the Learning assessment methods are curriculum implemented in key courses of curriculum There is a plan to incorporate learning assessment methods NO CHANGE PROPOSED across the curriculum The need for the improvement of learning as- sessment NO CHANGE PROPOSED methods is recognized and benchmarking of their current use is in process Learning assessment methods are inadequate or NO CHANGE PROPOSED inappropriate Rubric of standard 12 - Programme evaluation The rubric of standard 12 is consistent with the generic rubric, but we would like to propose some modifications to the Level the rubric for standard is changed according to Table 11 the wordings to make it clearer and easier to understnad All the proposed changes are shown in Table 13 Table 13 Rubric of standard 12 Original rubric Suggested change to the rubric Systematic and continuous improve- ment is Systematic and continuous improvement is based based on programme evaluation results from on continuous programme evaluation results multiple sources and gathered by multiple methods Programme evaluation methods are being used NO CHANGE PROPOSED effectively with all stakeholder groups Programme evaluation methods are being Programme evaluation methods are being implemented across the programme to gather implemented across the programme to gather data from students, faculty, pro- gram leaders, data from majority of the stakeholders (such as alumni, and other stake-holders students, faculty, programme leaders, alumni, working life representatives) A programme evaluation plan exists A continuous programme evaluation plan exists The need for programme evaluation is NO CHANGE PROPOSED recognized and benchmarking of evaluation methods is in process Programme evaluation is inadequate or Programme evaluation is inadequate, inconsistent inconsistent or non-exsisting Journal of critical reviews 359 EVALUATING THE CDIO-BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM RECOMMENDATIONS FOR FUTURE RESEARCH The self-evaluation rubrics are important for evaluating a training programme at university Therefore it is recommended to make quality assurance of it This paper can be seen as a first step, following this one we need to make a much broader evaluation of the understandability and usefulness of it We did a pilot study of this at Ha Tinh University and presented at the regional meeting in Ho Chi Minh city in January 2019; however such an evaluation needs more time in order to be of good quality (i.e the participants were shown the rubric of a given standard in less than one minute and afterwards asked about their views on the rubric) and this was a small sample that cannot be generalised for a larger population in Vietnam Higher Education institutions CONCLUSION Doing a self-evaluation for management administration training programme at Ha Tinh University is seen as a major quality improvement factor This naturally puts big requirements on the self-evaluation rubrics such that the rubrics help relevant stakeholders at Ha Tinh University in their reflection on the quality of the training programme In this paper, we have evaluated the twelve rubrics of the CDIO standards refering to the management administration programme Our goal was to analyze the rubrics and their understandability, consistency and usability for the CDIO self-evaluation for that programme In general, the rubrics are understandable, consistent and usable However, our analysis found several possible changes to the rubrics that could further improve the usability of the rubrics and could support the self- evaluation of CDIO-based management administration programme at Ha Tinh University Leadership in Education DOI: 10.1080/13603124.2019.1708472 Truong V Truong, Binh D Ha & Bao N Le 2019 The Effects of Industry 4.0 on Teaching and Learning CDIO Project at Duy Tan University Proceedings of the 15th International CDIO Conference, Aarhus University, Denmark 10 Vũ Anh Dũng & Phùng Xuân Nhạ 2015 Adaptation of CDIO-Based Learning Outcomes for Non-Business administration Disciplines: A Case study of Higher Educational System in an Emerging Country Journal of Business administrationTechnology and Education, (1), 101-112 11 PV Kamala Kumari, S Akhila, Y Srinivasa Rao and B Rama Devi "Alternative to Artificial Preservatives." Systematic Reviews in Pharmacy 10.1 (2019), 99-102 Print doi:10.5530/srp.2019.1.17 REFERENCES Cloutier, G., Hugo, R., & Sellens, R 2011 Mapping the relationship between the CDIO syllabus and the CEAB graduate attributes: An update Proceedings of the 7th International CDIO Conference, Copenhagen, Denmark Dai, B., Xu, W., Lan, B., Wang, T., & Hang, Z 2017 The evaluation method of the CDIO syllabus achievements based on the examination scaling point Proceedings of the 13th International CDIO Conference, Calgary, Canada Đỗ Thế Hưng 2015 Dạy học theo hướng tiếp cận CDIO đào tạo giáo viên kĩ thuật trình độ đại học, Luận án tiến sĩ Khoa học Giáo dục Retrieved from http://vnies.edu.vn/upload/Boiduong/dothehung.pdf Gunnarsson, S., Herbertsson, H., & Orman, H 2019 Using course and programme matrices as components in a quality asssuarance system Proceedings of the 15th International CDIO Conference, Aarhus University, Denmark Malmqvist, J 2016 A comparison of the CDIO and Eurace quality assuarance systems Proceedings of the 12th International CDIO Conference, Singapore Polytechnic, Singapore Nhu-Hang Ha, Duc-Man Nguyen, Anand Nayyar & Chia-An Liu 2019 Enhancing students’ softskills by implementing CDIO-based integration teaching model Proceedings of the 15th International CDIO Conference, Aarhus University, Denmark Tran, Hai-Ngoc, Philip Hallinger, & Dinh-Thang Truong 2018 The heart of school improvement: A multi-site case study of leadership for teacher learning in Vietnam School Leadership and Management, 38 (1), 80-101 https://doi.org/10.1080/13632434.2017.1371690 Tran, H N., Nguyen, D C., Nguyen, G V., Ho, T N., Bui, T Q T., & Hoang, N H (2020) Workplace conditions created by principals for their teachers’ professional development in Vietnam International Journal of Journal of critical reviews 360 ... 357 EVALUATING THE CDIO- BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM Integrated learning experiences are implemented Integrated.. .EVALUATING THE CDIO- BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM There is documented evidence of the full implementation... and introduces key skills Journal of critical reviews 356 EVALUATING THE CDIO- BASED BUSINESS ADMINISTRATION TRAINING PROGRAMME USING THE CDIO SELFEVALUATION RUBRICS AT HA TINH UNIVERSITY IN VIETNAM