3. Redesign the learning outcomes for the MEP
4.2. Evaluation of the current status of learning outcomes and the expected LOs
This part discusses in detail what each group of stakeholders expects from the graduates of the MEP and the gap between the current and expected learning outcomes. Since this study goes in the direction of outcomes-based learning, all the results shall be considered but the results from the groups of alumni and employers shall be given more attention in the formation of the new learning outcomes.
Graduating Students
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
In the study, all senior-year-students or graduating students were involved in the survey to fill out the questionnaires and to write down their ideas on the learning outcomes of the MEP. There were 250 questionnaires distributed to the students, 250 gave feedback but only 248 questionnaires were qualified for data processing.
The results received from the students are very encouraging; they support the re- designing of the Learning Objectives for better training and practice during the time they were at the university.
The results show that there is a gap between the current proficiency level of the students and the expected level in the view of the students (Appendix B). With regard to the Disciplinary knowledge and reasoning topics, there are gaps between the current proficient level and the expected proficiency level of the students.
For topics that belong to the Knowledge of Underlying Mathematics and Sciences, the gaps are small ranging from 0.37 to 0.60.
On the other hand, in the group of Core engineering fundamental Knowledge, the gaps are a bit bigger ranging from 0.52 to 0.82. The biggest gap of the group is for Tolerance and the smallest gap of the group is for the topic Electrical Engineering.
In Advanced Engineering Fundamental Knowledge, Methods and Rules, the gaps are bigger than the first two, the gaps range from the smallest which is 0.83 and the biggest is 0.93. However in the fourth group, other supplementary subjects, the gaps are smaller.
It can be inferred from the research that the students expected more of the knowledge related to their profession – engineering.
On the topics of Personal and Professional Skills and Attributes, there are also gaps between current and expected proficiency level. The survey results show an
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
interesting result that students wish to be updated with the current world of engineering with a gap which is the biggest within the group (1.04).
On the topic of Interpersonal Skills: Teamwork and Communication, the gaps are much bigger as students think they were not very good at this matter and they hope to be more competent in the field. Four (4) out of eleven (11) topics has a gap of more than 1 and the smallest is 0.86. The students evaluated their English
communication at low level (2.14) and they expected to be more skillful in the field (3.36). This reflects the real situation that students learned English in their first two years and then in their last 2 years, hence, they lack exposure to use the language so there is a limited improvement in its use. Using t-test to test the difference between the current and expected proficiency level of MEP graduates, the
researcher found a difference between the current and expected level of proficiency (t= -8.75, p=0.000).
Table 13
The Current and Expected Proficiency Level of Designing as viewed by Students
4.4. Designing
Current Proficiency
Level (WM)
Expected Proficiency Level (WM)
Quantity
Difference t-value Sig.
4.4.1 The Design Process 2.13 3.08 0.95 -10.45 0.00 4.4.2 The Design Process Phasing
and Approaches 2.02 3.09 1.07 -12.48 0.00
4.4.3 Utilization of Knowledge in
Design 2.18 3.15 0.97 -10.89 0.00
4.4.4 Disciplinary Design 2.16 3.13 0.97 -11.07 0.00 4.4.5 Multidisciplinary Design 2.00 2.96 0.96 -10.24 0.00 4.4.6 Design for Sustainability,
Safety, Aesthetics, Operability and other Objectives
1.96 3.01 1.05 -11.67 0.00
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
In relation to the application of knowledge to benefit the society section: Conceiving, designing, implementing and operating systems in the enterprise, societal and environmental context, students rated the current skills at rather low level ranging from 1.80 to 2.20 (see Appendix B). It means that some of the students now have experienced or been exposed to the practice of and some are able to participate in and contribute to the designing, implementing and operating systems in the
enterprise, societal and environmental context, while few of them are capable of understanding and explaining.
On the topic of Designing, students are more skillful than the other topics; they rate the current proficiency level from 1.96 to 2.18. Table 13 shows that the gaps are rather big between current and expected proficiency level. All are bigger than 1 point which requires lots of changes in the training of the teachers and the learning of the students, as well as the methods of training, time for practicing, since this topic requires these skills. It is notable that only topic 4.4.6 receives point lower than 2.
However, since the skills that the students possess are better, they require more improvement and most of the expected proficiency levels is above 3, only topic 4.4.5 Multidisciplinary Design gets the score of 2.96. T-test confirmed the difference between reality and the expected proficiency that the MEP should provide students with skills at higher level.
The topics under 4.3 Conceiving system engineering and management, 4.5.
Implementing and 4.6. Operating the system, are rated below 2. This explained the fact that students lack a lot of practice in the field of doing experimental work, therefore, they are weak in the skills of using engineering knowledge to real context of enterprise and society. The t-test has once again confirmed the difference
between current proficiency level and the expected one.
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn Table 14
The Current and Expected Proficiency Level of Operation as viewed by Students
4.6. Operating
Current Proficiency level (WM)
Expected proficient
level (WM)
Quantity
Difference t-value Sig.
4.6.1 Designing and Optimizing Sustainable and Safe Operations
1.83 2.86 1.04 -11.64 0.00 4.6.2 Training and Operations 1.84 2.96 1.12 -12.52 0.00 4.6.3 Supporting the System Life
Cycle 1.77 2.81 1.04 -10.92 0.00
4.6.4 System Improvement and
Evolution 1.86 2.96 1.10 -11.79 0.00
4.6.5 Disposal and Life-End Issues 1.80 2.85 1.05 -11.21 0.00 4.6.6 Operations Management 1.90 2.96 1.07 -10.98 0.00
Table 14 shows that the gaps are rather huge between current and expected proficient level. All are bigger than 1 point which requires lots of changes in the training of the teachers and the learning of the students as well as the methods of training, and time for practicing, since this topic requires skills in practicing.
In conclusion, based on the final year of students, they have had enough knowledge of basic mathematics and sciences. They need more advanced engineering
fundamental knowledge, other personal and professional skills, as well as, skills in communicating with others and doing teamwork. They rated themselves weak in the application of knowledge into real situation like designing, implementing and
operating engineering system within the enterprise, social context. This confirmed that students lack required skills before graduating. What students expected from the program is that the program would provide them the skills at the 3rd level that they should be able to understand and give explanation as compared to the students at MIT, USA. Moreover, they expect that on the 4th and 5th level, the program would
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
train them to be future engineers who are adept in practice or implementation or able to lead or innovate in the field.
Faculty
The faculty plays an important role during the training and learning of the students. They provide direct effect on the outcomes of the MEP in producing quality engineers especially in obtaining knowledge and acquiring skills.
The researcher did study on 50 faculty members and received 48 feedback, all of them are qualified for data processing.
In general, the faculty expected much more from the graduate of the MEP than the students did on almost every topic.
Table 15 shows the comparison of the current level of proficiency and the expected level with regard to Knowledge of underlying mathematics and sciences. Among the 7 topics surveyed, Statistics and probability has the lowest score (1.73) at current level of proficiency and Information Technology has the highest score (2.67). It may be inferred that the graduate’s knowledge of Information and Technology is better than that of statistics and probability. The topics that need most improvement are Statistics and probability. Method of calculation, where there are gaps between current and expected proficiency level, are 1.17 and 1.13, respectively. T-test results confirmed the difference between current and expected proficiency level of the graduates.
Table 15
Results of Survey on Knowledge of Underlying Mathematics and Sciences
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
There are 12 items (subjects) that the students will take in the curriculum related to core engineering fundamental knowledge (Appendix). Among the 12 topics, the lowest point is given to topic Electrical Engineering (1.90) in current proficiency level.
The highest point for current proficiency level belongs to Materialogic (2.48).
The highest expected topic is Materialogic (3.48) and the lowest is for Electronics Engineering (2.79).
In comparison between the current proficiency level and the expected proficiency level, the faculty seem to expect a lot from the graduate which has been proven by the results of the t-test. Seven (7) topics over 12 topics have a gap bigger than 1 point that means a great deal from the current to expected level.
The result of the survey also showed those faculty members evaluate the current proficiency level of the graduate lower than that of the students. It is the reason why the gap between the current and expected level is bigger than that of the students.
Topic
Current Proficien cy Level (WM)
Expected Proficienc y Level
(WM)
Quantity Differen
ce
t-
value Sig.
1.1.1 Analytic Mathematics (the derivative, differentiation, integral, differential equations, ...)
2.19 3.10 0.92 -8.60 .00
1.1.2 Algebra 2.19 3.13 0.94 -8.57 .00
1.1.3 Statistics and Probability 1.73 2.90 1.17 -
10.52 .00 1.1.4 Method of calculation 1.90 3.02 1.13 -8.67 .00 1.1.5 Physics: mechanical,
thermal, optical, electrical...
2.08 3.02 0.94 -7.73 .00
1.1.6 Chemistry 1.88 2.60 0.73 -6.73 .00
1.1.7 Information Technology 2.67 3.35 0.69 -7.03 .00
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
With regard to Advanced engineering, the researcher has listed a number of subjects related to advanced engineering fundamental knowledge, and methods and tools that are being taught in MEP-TNU (The design - manufacturing process, Manufacturing techniques (Mechanical technology), Equipment and Support Tools, Measurement Techniques, and Automation of production). In evaluating the current proficiency of the MEP students, all topics ranged from 2.31 to 2.77. The lowest point belongs to Equipment and Support Tools (2.31). Manufacturing techniques
(Mechanical technology) got the highest point (2.77).
The results suggest that the faculty expect that all the topics need improvement with respect to the student’s knowledge. The lowest point for the expected proficiency level is 3.25 and the highest is 3.69 for Automation of production and Measurement Techniques, respectively.
The gaps between the expected level of proficiency and the current proficiency levels range from 0.85 to 1.06. T-test has confirmed the difference between the two and that the students should be more proficient in their advanced professional knowledge.
Regarding other supplement subjects, faculty highly evaluate the knowledge of CAD of graduates (2.92) and the gaps between the expected and current proficiency levels ranged from 0.71 to 0.92. T- test confirmed that there was a significant difference between the current and expected level of proficiency (t value ranges from – 5.07 to – 8.68 with p=0.000). (See Appendix B).
Regarding Personal and Professional skills and attributes including analytical reasoning and problem solving, experimentation, investigation and knowledge
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
discovery, system thinking, attitudes, thoughts and learning, ethics, equity and other responsibilities, the results are as follows:
Table 16
Results on Analytical Reasoning and Problem Solving
Topic
Current Proficiency Level (WM)
Expected Proficiency
Level (WM)
Quantity
Difference t-value Sig.
2.1 Analytical Reasoning and Problem Solving 2.1.1 Problem Identification and
Formulation 2.08 3.25 1.17 -7.88 .00
2.1.2 Modeling 2.19 3.33 1.15 -6.75 .00
2.1.3 Estimation and Qualitative
Analysis 2.02 3.35 1.33 -10.41 .00
2.1.4 Analysis With Uncertainty 1.98 3.21 1.23 -7.85 .00 2.1.5 Solution and
Recommendation 2.15 3.42 1.27 -8.46 .00
2.2 Experimentation, Investigation and Knowledge Discovery
2.2.1 Hypothesis Formulation 2.27 3.46 1.19 -8.28 .00 2.2.2 Survey of Print and Electronic
Literature 2.48 3.69 1.21 -8.33 .00
2.2.3 Experimental Inquiry 2.19 3.48 1.29 -7.85 .00 2.2.4 Hypothesis Test and Defense 2.06 3.46 1.40 -9.70 .00 Table 16 shows that graduates’ current proficient levels of all topics do not reach point 3, there is even one topic rated lower than two which is Analysis With Uncertainty (1.98). The highest is for Modeling (2.08). With regard to expected proficiency level, the highest score is 3.42 for Solution and Recommendation and the lowest score is 3.21 for Analysis With Uncertainty. As far as analytical reasoning and problem solving is concerned, the stakeholders expected from the graduates that
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
their competences should be higher at the third level that they could be skilled in the practice and implementation and 28% to 38% of respondents share the idea.
With reference to the competences in experimentation, investigation and knowledge discovery, the survey reveals that the students’ competences are below the expected level. Table 16 shows the results on evaluation of the experimentation, investigation and knowledge discovery competences and there are differences between the current and expected proficient level. All topics have the difference of more than 1 point in which Hypothesis Test and Defense has the biggest gap of 1.40.
Table 17 shows the results on system thinking which is composed of four topics.
Among these topics, Trade-offs, Judgment and Balance in Resolution obtained better evaluation at current level of proficiency. The gap between current and expected proficiency levels is big, all are bigger than 1 (one) and the biggest gap was Thinking Holistically (gap value = -1.52).
Further, it shows the results on attitudes, thoughts and learning including 7 topics.
What the faculty evaluated on the current competences of the students is a round level 2 that is to be able to participate in and contribute to but what they expect them is at the 3rd level that is to be able to understand
Table 17
Results on System Thinking and Attitudes, Thoughts and Learning
Topic
Current Proficiency Level (WM)
Expected Proficiency Level (WM)
Quantity
Difference t-value Sig.
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
2.3.1 Thinking Holistically 2.02 3.54 1.52 -10.48 .00 2.3.2 Emergence and Interactions
in Systems 2.06 3.38 1.31 -8.07 .00
2.3.3 Prioritization and Focus 2.00 3.48 1.48 -8.43 .00 2.3.4 Trade-offs, Judgment and
Balance in Resolution 2.13 3.46 1.33 -8.19 .00 2.4.1 Initiative and the Willingness
to Make Decisions in the Face of Uncertainty
1.98 2.92 0.94 -8.99 .00
2.4.2 Perseverance, Urgency and Will to Deliver,
Resourcefulness and Flexibility
2.06 3.02 0.96 -7.69 .00
2.4.3 Creative Thinking 2.15 3.08 0.94 -7.29 .00
2.4.4 Critical Thinking 2.13 3.15 1.02 -10.43 .00
2.4.5 Self-awareness, Metacognition and Knowledge Integration
2.15 3.08 0.94 -6.54 .00
2.4.6 Lifelong Learning and
Educating 2.33 3.33 1.00 -6.49 .00
2.4.7 Time and Resource
Management 2.25 3.27 1.02 -7.18 .00
and explain and at the 4th level to be skilled in the practice and implementation. The gaps that the graduates should overcome range from 0.94 to 1.02, while much improvement should be done for Critical Thinking and Time and Resource Management (Appendix 12).
Table 18
Results on Ethics, Equity and Other Responsibilities
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn Topic
Current Proficiency Level (WM)
Expected Proficiency Level (WM)
Quantity
Difference t-value Sig.
2.5.1 Ethics, Integrity and Social
Responsibility 2.35 3.50 1.15 -7.26 .00
2.5.2 Professional Behavior 2.23 3.46 1.23 -7.66 .00 2.5.3 Proactive Vision and
Intention in Life 2.23 3.56 1.33 -7.84 .00
2.5.4 Staying Current on the World
of Engineering 2.40 3.60 1.21 -8.93 .00
Table 18 shows the results on evaluation of ethics, equity and other responsibilities of an engineer. The current proficiency levels ranged from 2.23 to 2.40 which are more than one (1) point lower than the expected levels (from 3.46 to 3.60). The graduates should have more proactive vision and intention in life (the gap is 1.33). In general, the faculty required more from the graduates than what they possess at the moment regarding ethics, equity and other responsibility of the engineer. It is
interesting to note that the students and faculty share the same idea that graduate should study more about engineering, to be updated to the modern world.
The Interpersonal skills including teamwork and communication are considered very important skills that an engineer should have in order to work effectively as a partner and a leader. In reality, the matter takes much more consideration of the faculty and students when they are at school than before. But the results of the survey show that what they have tried is not enough and the requirement of the faculty is higher in every topic. The competences at current level was below 3 with 2.00 as the lowest which is Communications Structure, while all expected levels are above 3 with 3.69, the highest level is Technical and
Multidisciplinary Teaming. From the results, it can be seen that the university did not satisfy the requirements of the society on the skills that the graduates have. The
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn
gaps between the current and expected proficiency level are all higher than one (1) and the t-test results have once again confirmed the differences between them.
As an engineer to be professional, he should be able to conceive, design, implement and operate systems in the enterprise, societal and environmental
context. However, to what extent does the groups of faculty expect from the graduate and at which level they are at the moment?
As far as Roles and Responsibility of Engineers is concerned, the average current proficiency level is 1.98 and the expected proficiency level is 3.25. Presently, students can take part in or be able to understand and explain the topic (58%), however, the expected level is be able to practice, implement and create (91.8%). In this category, students’ current achievement level does not meet the requirements of the group of stakeholder. The stakeholders also expect the students’ to have a higher proficiency level.
Table 19 shows the results on external, societal and environmental context. It can be seen that the current proficiency level is not very high but the expected level is neither high nor low. The highest score for current level belongs to the impact of engineering on society and the environment, The Historical and Cultural Context (2.15) and the highest expected score belongs to The Impact of Engineering on Society and the Environment (3.33). The lowest current score is Society’s Regulation of Engineering (1.88) and the expected level is (3.17).
Table 19
Results on External, Societal and Environmental Context
Topic Current
Proficiency
Expected Proficiency Level (WM)
Quantity
Difference t-value Sig.
Số hóa bởi Trung tâm Học liệu – ĐHTN http://www.lrc.tnu.edu.vn Level
(WM) 4.1.1 Roles and Responsibility of
Engineers 1.98 3.25 1.27 -8.82 .00
4.1.2 The Impact of Engineering on
Society and the Environment 2.15 3.35 1.21 -9.02 .00 4.1.3 Society’s Regulation of
Engineering 1.88 3.17 1.29 -10.16 .00
4.1.4 The Historical and Cultural
Context 2.15 3.13 0.98 -6.89 .00
4.1.5 Contemporary Issues and
Values 2.08 3.33 1.25 -10.16 .00
4.1.6 Developing a Global
Perspective 2.02 3.17 1.15 -9.03 .00
With regard to the competence and knowledge of the graduate on enterprise and business context, the researcher defined four topics, namely: Appreciating Different Enterprise Cultures - the average current proficiency level is 2.17 and the expected proficiency level is 3.15; Working in different kinds of organizations seems to need more improvement in comparison to current status (2.00) when it got the expected level of 3.23; and Adapting to different working environment is also expected much from stakeholders so that the engineer will be more flexible and adaptable to the fast changing world and to be ready for international integration.
Table 20 shows the results of the survey on Enterprise and business context and designing which includes 10 topics. The expected proficiency levels are one point above the current proficiency level of the graduates. The highest score is 3.60 for the desired competence of Disciplinary Design and the lowest desired score is 3.00 for Defining Function, Concept and Architecture, System Engineering, Modeling and Interfaces and Development Project Management topics.