EUROCONTROL Customer Satisfaction Report IANS A Centre of Excellence in ATM Training Report Copyright Notice © 2006 European Organisation for the Safety of Air Navigation (EUROCONTROL) All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise Any other use is subject to prior written consent by EUROCONTROL Request shall be addressed to: Director of the Institute of Air Navigation Services, 12, rue Antoine de Saint-Exupéry, L-1432 Kirchberg/Luxembourg The EUROCONTROL Institute of Air Navigation Services aims to provide the services that you want and to make your stay in the Institute as enjoyable as possible All Institute personnel are there to ensure that your stay at the Institute is successful However, if you have a complaint (or a compliment) please tell us If you are not satisfied with the service we provide or you would like to propose an improvement then please fill out the form at http://www.eurocontrol.int/ians/complaint.html, or contact IANS.complaints@eurocontrol.int directly ii © Copyright 2006 EUROCONTROL Table of Contents INTRODUCTION ATM TRAINING ATC TRAINING 12 E-LEARNING 13 STAKEHOLDER FEEDBACK 14 COMPLAINTS 15 CONCLUSION 16 Printed on 30/11/2004 Produced by the Business and Administration Unit Customer Satisfaction Report Introduction This is the 7th report that the Institute devotes to the assessment of the level of satisfaction of its customers The study on which this report is based has been conducted in the framework of the quality improvement programme implemented at the Institute since end of 1996 • Measuring customer satisfaction on a regular basis is one of the key elements of the quality strategy implemented by the Institute since 1997 • Our objective is to understand exactly how our customers perceive the Institute and, based on their level of satisfaction, to improve our services and products This report proposes a summary and an analysis of the data collected in 2005 The customer segments assessed in 2005 IANS applies a policy to measure the satisfaction of its customers with all its products and services These include: • • • • • • The ATM course participants The ATC course participants The E-Learning Students Training Sponsors Consultancy Bodies User Groups • A Student Evaluation Form filled in by the students at the end of a course This evaluation form is used for both the ATM course participants and ATC course participants An E-Learning Evaluation Form filled in by a student upon completion of an E-Learning Module A generic Stakeholder Feedback Form filled in by various stakeholders The form is available at http://www.eurocontrol.int/ians/feedback.html Customer Visits IANS visits a sample of its customers on a yearly basis The customer visits serve as a main source of feedback on existing products and a source of information for improvements and new products The customer visits include visits to Air Navigation Service Providers and civil and military State Authorities Complaint Procedure and Complaint Form IANS uses an anonymous form, available on its web-site for the collection of complaints related to its products, services and staff A reference to the complaint procedure and associated form is made in all products and services The complaint form is available at http://www.eurocontrol.int/ians/complaint.html Assessment Tools The assessment is based on a set of assessment tools and measurement methods During 2005 further work has been done to increase the integrity of the data collection Evaluation Forms To measure the satisfaction of its customers with the products and services IANS uses the following evaluation forms: © Copyright 2006 EUROCONTROL ATM Training The Satisfaction of Students with the ATM Training Products is measured through the use of a Course Evaluation Form Measurement Method At the end of each (classroom) course, students are presented with an evaluation form containing a number of criteria to be assessed Each criterion can be given a result between and meaning unacceptable (1), unsatisfactory (2), good (3) an excellent (4) The complete set of criteria are grouped in five main categories: • • • • • Training Content Instructional Team Documentation Equipment Administration The first three categories are directly related to the course The last two categories are directly related to the supporting environment and organisation Figure Student Evaluation Form Printed on 30/11/2004 Produced by the Business and Administration Unit Customer Satisfaction Report From Forms to Results We map the marks of the students on a scale from (all students mark Unacceptable) to +100 (all students mark Excellent) When all students mark Good then the result is 75 and when students mark, on average, between Good and Excellent then the result will be around 85 Likewise, if students fill in 'all-Uncacceptable' then the result will be If students fill in 'all-UnSatisfactory' then the result will be 25 Measuring Satisfaction Marks from Students Unsatisfactory Satisfactory Good Excellent 25 75 100 All All Result All All ©Copyright 2003 EUROCONTROL Figure How we measure satisfaction IANS aims to meet our objective of being a Centre of Excellence in ATM Training For us this means we aim to achieve results on any criterion between 75 and 100 As an example: the overall average (all courses, all criteria, 3090 forms) for IANS in 2005 is 87.5 – compared to 85 in 2004 The highest result on a single criterion is 95 (Subject Knowledge of the Instructor) The lowest result remains pre-course information (76.5 – compared to 70 in 2004) Results The results presented below are extracted from the automated student satisfaction evaluation system Results can be presented per category, per training domain and per criterion We have selected the most significant results to be included in this report © Copyright 2006 EUROCONTROL Size of the Population 3500 This graph shows the number of evaluation forms received and processed by IANS In 2005 the Institute continued to optimise the processing of the forms 3000 2500 2000 This led to another significant increase of the number of processed forms in 2005 1500 1000 500 Number of Forms 2002 2003 2004 2005 1937 1915 2624 3090 Figure Number of Evaluation Forms Composition of the Population Students indicate on the evaluation form which type of staff they are 1400 1200 1000 This chart shows that a majority of Students at IANS in 2005 are Operational and Technical Staff These results are consistent with the results of the previous years 1298 977 2004 824 2005 800 693 600 600 345 400 192 223 200 Administrative Managerial Operational Technical Figure Composition of the population (number of students) Printed on 30/11/2004 Produced by the Business and Administration Unit Customer Satisfaction Report Satisfaction 100 This chart shows the satisfaction of the students with the ATM courses The result is obtained by averaging the results obtained on every criterion in the categories Training Content, Instructional Team and Documentation 95 90 85 80 75 70 A result of 87 indicates that, on average, students mark criteria between Good and Excellent 65 60 55 50 2002 2003 2004 2005 83 84,5 86 87,5 Course Results Figure ATM Course Results The increase in satisfaction has been achieved through detailed analysis of the results of each course and introducing corrective action wherever required Segmented View 100 This chart presents the results per measurement Category These are Training Content, Instructional Team and Documentation 95 90 85 80 75 IANS has improved the results on each of these three categories during the last four years 70 65 60 55 50 2002 2003 2004 2005 81,5 83 84,5 86,5 Instructor 88 89 91 92 Documentation 80 80,5 83,5 85 Content Figure ATM Course Results: per Category © Copyright 2006 EUROCONTROL IANS will continue to analyse the results and make improvements where possible and practicable ‘Best’ Results 100 Customers often praise IANS for its staff and excellent infrastructure 90 This informal feedback is confirmed by the course evaluation results The criteria ‘Instructor Subject Knowledge’ and ‘Classroom Equipment’ have consistently received the highest marks from the students during the last three years 80 70 60 50 2003 2004 2005 92,5 93 94,5 95,5 Classroom Equipment IANS will continue to invest in the competence of its instructors by continuation training and competency assessment and it will maintain the building at its high standard 2002 Instructor Subject Knowledge 88,5 89 90,5 91 Figure Criteria with ‘Best’ Results Areas for Improvement 100 IANS continuously looks for opportunities to improve Criteria with the less positive results are the Pre-Course Information and the Course Duration 90 80 70 60 50 Pre Course Information Course Duration Canteen 2002 2003 2004 2005 61,5 64 70,5 78 71 74 76 76 73,5 72,5 75 77,5 Figure Criteria with less positive Results In 2004 IANS has put in place an action plan to improve the Pre-Course information and significant progress has been made in this domain in 2004 and 2005 The optimum duration for each course will be reviewed on a courseby-course basis In 2005 IANS changed the layout of the coffee area and installed new equipment in the kitchen to improve the flow of students Printed on 30/11/2004 Produced by the Business and Administration Unit Customer Satisfaction Report Focus on Safety IANS is heavily involved in the development and delivery of Safety Courses to support Safety Regulation and Safety Management In 2005 the number of courses delivered in the Safety domain increased from 36 to 50 and the number of participants increased from 614 to 844 We are pleased to note that the results in this relatively new area for IANS equal the average obtained in other, more mature, training domains 100 95 90 85 80 75 70 65 60 55 50 2003 2004 2005 Safety 86 88,5 90 Overall 84,5 86 87,5 Figure Safety Domain Course Results It is a clear indication that the rigorous development approach at IANS leads to products that meet customer expectation Measuring Loyalty 100% 90% In 2004 we introduced a new question on the evaluation form: ‘Would you recommend the course to a colleague’ 80% 70% 60% 50% 40% 30% 20% 10% 0% Yes No Blank 2004 91% 1% 8% 2005 92% 1% 7% Figure 10 Would recommend course to a colleague 10 © Copyright 2006 EUROCONTROL In 2005, like in 2004, more than 90% of the students would recommend the course that they attended to a colleague Measuring dissatisfaction 10,00% To measure dissatisfaction IANS uses the percentage of ‘unsatisfactory’ marks on the evaluation forms 9,00% 8,00% 7,00% 6,00% This chart shows the proportion of unsatisfactory marks on the evaluation forms for all criteria 5,00% 4,00% 3,00% The excellent results in 2005 are spread over the entire spectrum of criteria 2,00% 1,00% 0,00% Percentage of 'Unsatisfactory' Results 2002 2003 2004 2005 0,91% 0,90% 0,83% 0,59% Note that that an ‘unsatisfactory’ mark on one criterion does not necessarily mean that the student is not satisfied with the whole product Figure 11 Level of dissatisfaction Printed on 30/11/2004 Produced by the Business and Administration Unit 11 Customer Satisfaction Report ATC Training • Last but not least, the ATC ab-initio students are evaluated at the end of the course At the end of the evaluation they either pass or fail The outcome of the evaluation tends to have a significant impact on the satisfaction of the students IANS believes that the critical success factor for ATC students is their validation rate (i.e the proportion of students that obtain an ATC license) We however present the satisfaction data in this report as it is part of our quality management tools The satisfaction of ATC ab-initio students is measured using the same evaluation as the system used for measuring the satisfaction of the ATM students Satisfaction figures of ATM and ATC students must however not be compared The context of these courses, although delivered in the same building, is completely different: • The duration of ATM courses is maximum of two weeks whereas ATC have a duration of several months The student population of ATM courses are staff members from service providers and member states The student population of ATC courses are abinito ATC students • • ATC Student Satisfaction 100 This chart shows the satisfaction of ATC abinitio students 90 80 70 60 50 2002 2003 Training Content 79,5 79 75 78 Instructors 76,5 73 72,5 77 78 69 76 78 Training Documentation 2004 2005 Figure 12 ATC Student Satisfaction The last few years ATC ab-initio training has been characterised by high rotation of staff and continuous changes to the training content and structure as a result of variations in demand and new regulatory requirements In 2005, IANS significantly improved the documentation and training of the ATC Instructors Compared to 2004 the satisfaction of the students has increased for each segment 12 © Copyright 2006 EUROCONTROL E-Learning IANS introduced satisfaction measurement for its elearning products in the middle of 2004 Students are given an opportunity to evaluate each e-learning module at the end of the module The figures presented in this report cover all the modules on the EUROCONTROL IANS on-line TrainingZone, including the CFMU modules In 2005, students filled in 2774 forms (800 in 2004) evaluation forms were filled in 90% of the students identified to be ‘satisfied’ with the modules and more than 88% of the students indicated that the module met the objectives These figures indicate that the overall satisfaction with the e-learning modules is excellent Why are you studying this module? Personal reasons (e.g to further career, out of personal interest etc) My employer decided I should It was a prerequisite for a class-room based course or programme The module agree not sure-n/a disagree followed a logical sequence, O O O was not too easy or too difficult, O O O was clear and easy to understand O O O The module was agree not sure-n/a disagree easy to use, O O O interactive and interesting O O O In order to successfully access this module, did you have to change your browser (Internet Explorer) settings? install new software (flash plug-in, java plug-in, internet explorer)? disable a popup-killer or anti-intrusion programme (e.g Norton Internet Security)? The module agree not sure-n/a disagree met the objectives set at the beginning,O O O allowed me to check my understanding O O O Did you enjoy studying this module: O yes O no E-Learning Evaluation Form A customised form was developed to capture student satisfaction with the e-learning modules The form collects information related to the ‘reasons for studying’, the complexity of the module, the objectives of the module and the impact of the module on the student’s job Will this module Help you directly in your current job? Help you in a future job? Increase your understanding of the industry without direct impact on your current/future job? Help you personally (e.g out of personal interest)? Structure of ELearning Modules 100% 90% This chart shows the percentage of students that answered ‘yes’ on the following questions: 80% 70% 60% 50% • Structure of the module meets expectations 40% 30% • Level of the module meets the expectations 20% 10% 0% structure level clarity of language 2004 92% 73% 86% 2005 95% 80% 87% Figure 13 Structure of E-Learning Modules Printed on 30/11/2004 Produced by the Business and Administration Unit • Module uses clear language These results indicate that the structure and content of the E-Learning modules meet the needs of the students 13 Customer Satisfaction Report Origin of ELearning Students 45% 40% 35% It is IANS’ strategy to integrate e-learning in well defined training programmes – defined and implemented by the student’s employer - and as pre-requisite modules for classroom courses 30% 25% 20% 15% 10% 5% 0% 2004 2005 Self-driven 43% 28% Employer-driven 30% 39% Prerequisite 27% 33% Figure 14 Why did students study e-learning modules These results indicate that the proportion of students registered through their employer or studying as a prerequisite module has significantly increased compared to the number of students that study out of their own initiative The number of students that see a direct impact on their job from studying e-learning modules has increased from 56% in 2004 to 68% in 2005 14 © Copyright 2006 EUROCONTROL Stakeholder Feedback Training Consultation Group The Stakeholder Feedback Form was distributed to the meeting participants at the end of each Training Consultation Group Meeting The majority of the feedback indicates that the overall perception of the Institute lies between Good and Excellent (on a scale unsatisfactory – satisfactory – good – excellent) Compared to 2004 the results related to communication (content, format and timeliness) have improved We will continue to review and improve our communication with the members of the consultation group Other stakeholders The stakeholder distributed to: evaluation form was also • Participants of the training focus group • The training partners at IATA The results indicate an equally high level of satisfaction of working together with IANS Customer Visits The Customer Visits are a process in which IANS meets stakeholders in member states on a bi-lateral basis It is a mechanism to collect qualitative feedback from various stakeholders The countries visited in 2005 are: Austria, Belgium, BIH, Bulgaria, Greece, Hungary, Spain, Sweden and Turkey During the meeting of the Training Consultation Group in September 2005 it was agreed that IANS would include in its Customer Satisfaction Report information gathered during the Customer Visits During these customer visits, stakeholders have in general expressed a high degree of satisfaction with IANS, its staff and the services it offers The following points have been raised: • OJTI: The delivery of OJTI courses through partnerships is often perceived as an expensive solution (by stakeholders receiving the services) The solution is not perceived as entirely satisfactory • • • • ATSEP Training: Many of the visited stakeholders are currently in the process of defining basic training for Air Traffic Safety Electronics Personnel (ATSEP) Some collaboration with EUROCONTROL and other member states could avoid duplication of effort On-Site Courses: In general the demand for on-site course (i.e courses at stakeholder premises) exceeds the capacity of IANS Although most stakeholders are aware of the E-Learning products of the Institute many of them have not yet been able to assess the feasibility to use the products available ATC Training Notes: IANS should develop a policy to distribute CCC aligned ATC Training Notes Complaints Every training note distributed to students and all pages on the On-Line Trainingzone contain a clearly visible reference to IANS’ complaint procedure The procedure enables stakeholders to submit complaints in an anonymous way or an identified complaint through e-mail The text is as follows: compliment) please tell us If you are not satisfied with the service we provide or you would like to propose an improvement then please fill out the form at http://www.eurocontrol.int/ians/complaint.html, or contact IANS.complaints@eurocontrol.int directly The EUROCONTROL Institute of Air Navigation Services aims to provide the services that you want and to make your stay in the Institute as enjoyable as possible All Institute personnel are there to ensure that your stay at the Institute is successful However, if you have a complaint (or a Although the procedure is clearly marked in all documentation, IANS did not receive formal complaints in 2005 Printed on 30/11/2004 Produced by the Business and Administration Unit 15 Customer Satisfaction Report Conclusion IANS aims to be recognised by its stakeholders a Centre of Excellence in ATM Training We continuously improve our products and processes We measure the satisfaction of our customers with the products that we provide This document presented the results of the satisfaction measurement in 2005 The satisfaction with our ATM Training Products shows an upward trend in all domains for the fourth year in a row The satisfaction of the ATC Ab-initio students with the ATC training started increasing again in 2005 In 2005 we have further exploited satisfaction measures for the E-Learning Products and for the Stakeholders that we work with The results in these domains show a very high degree of stakeholder satisfaction END OF DOCUMENT 16 © Copyright 2006 EUROCONTROL ... Unit Customer Satisfaction Report Introduction This is the 7th report that the Institute devotes to the assessment of the level of satisfaction of its customers The study on which this report. .. include in its Customer Satisfaction Report information gathered during the Customer Visits During these customer visits, stakeholders have in general expressed a high degree of satisfaction with... on 30/11/2004 Produced by the Business and Administration Unit Customer Satisfaction Report Satisfaction 100 This chart shows the satisfaction of the students with the ATM courses The result is