customer satisfaction made easy

A plan for actively listening to customers of Labor Market Information Products and Services A resource handbook from the Customer

Customer Satisfaction Made Easy A plan for actively listening to customers of Labor Market Information Products and Services A resource handbook from the Customer Satisfaction Work Group of the Workforce Information Council Copyright 2003 Table of Contents Topic Page Introduction – Why Measure Customer Satisfaction? How to Measure Customer Satisfaction Step – Pick a Product or Customer Group Step – Set the Stage for Assessment What you want to find out? 7-10 Sample Problem Definition Statement 8-10 How you can best find the answers? 10-31 Qualitative Approach 11-12 Quantitative Approach 13-14 Guide for Using Focus Groups 16-18 Guide for Using A Mail Survey 19-20 Guide for Using A Telephone Survey 21-23 Guide for Using a Personal Interview 24-26 Guide for Using an Internet Survey 27-29 What to Ask 30 Sample Survey 31 2-3 4-6 7-31 Step – Conduct Research and Take Action! Documenting Demand Conclusion 32 33 I n t r o d u c t i o n – Why measure customer satisfaction? Because it’s required! The Workforce Investment Act of 1998 says states need to “consult with customers about the relevance of the information disseminated through the statewide employment statistics system, in order to continuously improve the system.” The two key parts of this statement are: • Consult with customers • Continuously improve the system It’s also a requirement of the One Stop LMI grant from the Employment and Training Administration! Because we have to prove “return on investment!” In times of increased accountability in government, every activity needs to prove that taxpayers’ money is used efficiently and effectively To ensure that LMI products retain their value to customers, we need to continuously ask customers if our products are meeting their needs We also need to create new products that are responsive to customer demands Asking our customers about our products provides proof that LMI products are a good investment Because we can’t afford not to! With limited funding, we have to make sure that every dollar we spend on developing and disseminating information products (whether its for printing or updating a website) is being spent on a product people will use The BEST way to make sure our products are being used is to ask the people who use them Customer satisfaction assessment is a great opportunity for states to make positive changes in the future of the labor market information system! How to measure customer satisfaction? This handbook will help states make customer satisfaction assessment as easy as 1, 2, 3! In order to make it this easy, there are some things you should and should know Form an evaluation team The team will oversee the collection of customer satisfaction feedback and then perform three responsibilities Make changes to products and services based on that feedback; Share satisfaction processes and results with other states; Share satisfaction processes and results with federal representatives to make systemic changes Consider having a publications specialist, an analyst and a projections professional on your team This team may already exist as the group who determines your LMI product content Customer satisfaction assessment does not necessarily have to be a statistically rigorous exercise The process outlined in this handbook is based on solid descriptive research methodology, but it is different from the types of research normally conducted in “LMI shops.” While you may have to meet strict standards for response rate based on a stratified random sample to meet the requirements of the Bureau of Labor Statistics, customer satisfaction assessment is based on a “dialogue” with your customers This dialogue may not be conducted the way you would conduct a BLS survey You should document the demand for your products and services In addition to finding out if customers are satisfied with your products, find out how many are actually using them, how they are using them and where those users are located We have a special section at the end which addresses this issue Create Customer Lists Many of the processes outlined in this handbook will require you to have a list of your customers You may want to consider creating these lists if you not have them already These are lists of customers who receive or use a particular document or LMI product you produce You have many tools in the toolbox of customer satisfaction research You may want to conduct a survey, you may want to have a focus group or conduct interviews with your main customers of a particular product Remember, when you ask just one customer their opinion of a product, you have started to listen to the customer When you change a document based on what a customer said, you are engaged in the process of continuous improvement This leads to the final point: You can’t it all at once! This handbook will walk you through conducting customer satisfaction research At first, it may seem a daunting task to find out what all your customers think about all of your products But, you have to start somewhere and you can’t it all at once Remain calm and dig in! Step Pick a Product or Customer Group State LMI providers should be producing documents for particular customer groups According to the Workforce Investment Act, states should be serving the following customer groups: • Businesses make decisions about product and financial markets, business location, and employee recruitment, compensation, and training • Individuals, including young people and adults, make choices about careers, education, training, and job search • Elected Officials and Policy Makers, including Workforce Investment Boards, make decisions about law, policy, budgets, and regulations • Program Planners determine what workforce and economic development services to provide, and evaluate program performance • Education and Training Providers, including teachers and curriculum specialists, design, deliver, and evaluate programs that develop students’ knowledge and skills • Intermediaries, such as parents, counselors, teachers, mentors, placement workers, and case workers, assist others in choosing education and training opportunities, and in finding employment, and • Researchers study how the labor market works, and conduct policy research The mass media may be included in this category Take a look at the LMI products and services your state produces Which customer groups use each of the products? Create a grid, like the sample that follows, which shows the customer groups and the products you provide for them MATRIX Career Handbook Hot Job Sheet Labor Market Trends Wages 2002 Commuting Patterns Unemployment Rate WIB Conference Website Website Website X X X X X X X X X Media Researchers Intermediaries Education and Training Providers GROUPS Program planners/ Workforce Boards Elected officials and Policy Makers Individuals/ Job Seekers LMI PRODUCTS/ SERVICES Employers CUSTOMER X X X X X X X X X X X X X At each intersection on the grid, you should be assessing satisfaction Once you identify which customers are using what products and services, you can measure customer satisfaction by starting with either of two paths: • start with a product, or • start with a customer As noted on the previous grid, if your state is about to re-publish a Career Handbook, you may want to ask people in the three customer groups that use the document if they are satisfied with it In that case, you would be starting with a product You can be in tune with the changing needs of customers by conducting needs assessment without discussing an existing product For example, each year employment statistics agencies must confer with local workforce investment boards to discuss information needs This is an opportunity to conduct a needs assessment Ask them about how they use data in their core business functions and if workforce information could help them their work This may help you prioritize which information products and services you provide and customize those products for their specific needs Step Set the Stage for Assessment There are two things you need to in step two: determine what you want to find out, and determine how best to find the answers! The rest of this step walks you through this process and gives you examples What Do You Want to Find Out You should develop an Assessment Statement This merely helps you “think through” your descriptive research process It may seem obvious when you complete it, but it will help guide you This is what should be in and should be answered in your statement: • A general description what you want to find out and from whom • A basic question the research needs to answer • What you want to determine in your research? • Management objectives, such as: ✓ Why you are doing the research? ✓ What will be the uses of the information you learn? ✓ Why is it important that you the research? ✓ What is the general setting for the research? ✓ Who are you going to assess? • What data collection technique will you use and why? • When will you it? • How much time/money can you spend doing it? • Who will it? The following pages provide an example of a completed Assessment Statement DETERMINING THE ATTITUDES OF CAREER COUNSELORS CONCERNING CAREER GUIDES AND POSTERS Assessment Statement LE MP SA General Description: To assess satisfaction with career guidance materials delivered by the State Department of Labor (DOL) and determine methods for improving those products for customers Basic Research Question: In what ways can career guidance materials be improved to assist those who advise students on career goals? Research Information Questions: - To determine the level of satisfaction with existing career materials, specifically Guides to Career Choices and accompanying posters; - To determine the most effective attributes of career materials which should be retained in future publications; - To determine the least effective attributes of career materials which should be omitted in future publications; - To determine attributes of career materials from other states which may provide better alternatives for presenting information to career guidance personnel Management’s Objectives: Why is this research being done? We are required to assess customer satisfaction with products as part of its funding grant from the U.S Dept of Labor, Employment and Training Administration (ETA), and needs customer satisfaction input to guide its continuous improvement efforts This assessment is in partial fulfillment of this obligation We also want to create a more useful product A GUIDE FOR USING A MAIL SURVEY General Description and Purpose: You want to survey persons using the US mail and have several questions to ask Normally, there is a large number of persons to be surveyed and you want to project the results to a particular group Appropriateness: The mail survey is very appropriate when the respondents are easily identified but may be located fairly far from each other It is useful when the questionnaire is two to four pages in length You can also enclose a brochure for the recipient to evaluate Recommended LMI Product s or Services: All of your products or services can be evaluated through the mail Types of Questions: The mail survey normally uses closed-ended questions that provide answers You should be able to have about 25 questions per page Too many open-ended questions will be left unanswered Customer Groups: Any group that can be identified such as recipients of a LMI brochure or job seekers, can be surveyed using the mail Selection of Customers: You will want to use an objective, random method of selecting your respondents A frequently used method is systematic sampling Randomly select every “n”th person from the list Number involved: If you have fewer than 1000 persons in a customer group, select everyone If you have more, then select 1000 to be surveyed You will reach about 95% of your selected group 19 Typical Response rate: Depending on the group, the response rate may range from 10% to 50% However, 30% is more typical Having responses from of 300 persons will result in a margin of error of 5.6% when projecting the results to the appropriate customer group Method of Conducting the Mail Survey: You will need a list of names and letterhead stationary Outgoing #10 envelopes and postage-free #9 return envelopes are necessary A followup post card is needed as well Copy the questionnaire onto both sides of the paper Make it look professional Including an incentive such as a bookmark can be effective Typical Costs: Costs must be allowed for copying, envelopes, outgoing postage for both the surveys and a postcard and return postage for the surveys Time Required: Allow at least weeks to obtain the names, design the questionnaire, and obtain the envelopes About days after mailing the questionnaires, send out a follow-up postcard Allow at least 2.5 weeks for responses A second questionnaire mailing might increase response rate, but would take longer Staffing Needs: Different skills are needed Good word processing skills for designing the questionnaire are desirable You will need persons to stuff the questionnaires, enter the data correctly, tabulate and analyze the results Benefits of Using this Approach You have a high certainty of contacting the people you want The design of the questionnaire can be very flexible You can a survey from a centralized location and have control over the process Cautions in Using this Approach The response rate is somewhat low The survey must look professional Include a postage free return envelope If the group being surveyed is not interested in the product or service, the response rate may be extremely low Do not ask too many open-ended questions 20 A GUIDE FOR USING A TELEPHONE SURVEY General Description and Purpose: Calling a group of customers can be a very efficient and effective method of collecting information from them The telephone survey allows for both quantitative and qualitative data collection Appropriateness: Use this method when you have a specific list of persons to be called who will be receptive to your call Many people are too busy to answer, or may not be available to take your call Recommended LMI Products or Services: All products and services can be surveyed using the telephone Types of Questions: Most telephone surveys will ask fewer than 40 questions Yes, you can ask more but the respondent may become irritated These questions will tend to be closed-ended with answers Asking too many openended questions will also irritate the respondent For example, asking “Why” to each question will soon yield very little response Customer Groups: All groups can be difficult to contact Having the specific telephone number and name of a person who is a user of information will greatly assist in the use of the telephone technique Selection of Customers: You will want to use an objective, random method of selecting your respondents A frequently used method is systematic sampling Randomly select every “n”th person from the list 21 Number involved: If you want to have 300 persons included in your data base, then you will need to select about 900 persons This can be less if you have a longer time period to collect the data But if you are not going to use call backs, then selecting times the number desired is a good rule of thumb Typical Response rate: Of those persons actually contacted, you should expect about an 80% response rate Method of Conducting the Telephone Survey: A centralized method of calling persons is best But you may not have a center for telephoning Therefore, employees will need to call from their desks You will need to develop a computer program that will allow the interviewers to ask the questions that are on the computer screen so the information can be directly entered into a data base If persons who are called are not available, then a callback should be made the following day Do not leave a message on a recorder Typical Costs: This could be the least expensive method since persons are already hired and telephones are available Even long distance costs are very reasonable today Interviewers would already have computers If the office does not have a computer program to facilitate the interviewing and data collection, then it would need to be purchased Time Required: This can be a relatively quick method of collecting information Allow about one week to establish the survey, a few days for training and perhaps another week to collect the information If the data are entered directly into the computer, a preliminary report can be generated within a day of the final collection of data 22 Staffing Needs: In addition to people constructing the questionnaire, you will need experienced interviewers The more you have, the less time it takes to collect the data You may need several people to the calling This will take them away from their other assignments Benefits of Using this Approach You can contact many different people very quickly You have control over the survey You can call other people to replace those who not answer or refuse to answer Of those you contact, most will participate This can be the quickest and least costly method Cautions in Using this Approach Be careful about asking too many open ended questions Do not have the survey last more than about minutes Be careful of the questions you ask Do not ask questions that rely upon the customer’s memory too much You will have fewer options in forming your questions and answers It is very helpful to have the questionnaire entered into a computer so the responses can be easily tabulated You will need pleasant people with a strong self confidence to the interviewing 23 A GUIDE FOR USING A PERSONAL INTERVIEW SURVEY General Description and Purpose: A personal interview survey involves directly asking people on a one-toone basis It allows direct interaction with the respondent Appropriateness: If you can easily contact people, you might want to use this technique But unless the respondents really want to be surveyed, this can be a time consuming, frustrating technique This would be most appropriate for persons coming into One-Stop Center or those attending a conference, workshop or training session Recommended LMI Products or Services: Since this technique is the most flexible, one can ask about any LMI product or service using a variety of techniques Types of Questions: Any type of question is appropriate for the personal interview technique Even though many of the questions will be closed-ended, you can use several open-ended ones People are more likely to respond to open-ended question when personally asked You are not as limited to the number of questions as in the other techniques More than 50 questions would be possible 24 Customer Groups: Since many of the customer groups would be difficult to contact in an efficient manner, this technique would be best limited to persons coming to One-Stop Center (if you are asking job seekers, for example) or those attending a conference, a workshop or a training session This can also work for employers who may not have time or want to attend a focus group Selection of Customers: If you limit your choice of respondents to one of the suggested groups, you will be more limited in who you will talk to Therefore, interview everyone who comes into the office during a specified week Also interview everyone who attends a conference, workshop or training session, or every willing employer on a list Number involved: The number involved in this study will be limited to the number who will come to your office, welcome you to their office or attend a conference, workshop or training session Typical Response rate: About 80% of those coming to your office will most likely have time to answer your questions Over 90% of those attending a conference, workshop or training session will respond Some will have left Some are in a hurry Method of Conducting the Personal Interview Survey: Have a person with a pleasant personality conduct the interviews Having a clip board facilitates the interview process and projects a “research atmosphere.” Questions are directly asked and the answers recorded Some questions may require the respondent to look at them in order to answer them Be certain that all open-ended questions are directly asked and that the respondent is encouraged to respond 25 Typical Costs: This can be relatively inexpensive since only one or two persons will be needed to the interviewing (such as an economic planner.) Time Required: This can be relatively quick if the persons are at a session It can be relatively slow if you need to wait for a particular type of person such as an economic planner to call you back for an appointment Staffing Needs: In addition to people constructing the questionnaire, you will need an experienced interviewer Someone to enter the information will be needed Someone will be needed to analyze the open-ended questions Benefits of Using this Approach You have control over the survey and the asking of the questions Of those you contact, most will participate Can be a quick and a low cost method Cautions in Using this Approach You may need extra time to collect information using this approach Some people will not want to be interviewed You are trying to determine customer satisfaction not to create customer dissatisfaction If you ask open ended questions, you will need to have someone analyze them You may not have a very large sample size on which to make decisions You will be limited to those persons who are available to you Trying to personally interview employers at their place of business can be a major problem 26 A GUIDE FOR USING AN INTERNET SURVEY General Description and Purpose: Providing an opportunity for a person to respond to a survey using the Internet This can be accomplished using an e-mail message only or linking with a Web based survey Using the Internet to collect information is very similar to a mail survey Appropriateness: The Internet survey is most appropriate when you have the e-mail addresses of all persons to be surveyed Also, you must be confident that the respondent would not treat a message as “spam” Recommended LMI Products or Services: All products and services, including a Website for your LMI materials, could be surveyed in this manner Types of Questions: Any type of question is appropriate for the Internet based survey Even though many of the questions will be closed-ended, you can use a few open-ended ones You are more limited to the number of questions using this technique since people are accustomed to spending a short time on Websites More than 50 questions would be somewhat impractical Customer Groups: Any customer group with a computer is appropriate for this technique But you need the e-mail addresses 27 Selection of Customers: If you sampling, be certain to take a random sample of the people to be contacted Systematic sampling works very well Number involved: A target number of responses varies depending upon the customer group However, 300 to 400 responses is an adequate number normally But if the total number available to survey is only 100, survey everyone Typical Response rate: The response rate will vary greatly depending upon the knowledge of the LMI products or services and the overall interest of the customer group in the information You can expect 60% from LMI-savvy individuals but only 10% from job seekers Method of Conducting the Internet Survey: The methods used can vary widely The simplest way is to include questions in an e-mail message and have the person hit REPLY and click on the appropriate answer You cannot have very many questions though This is cumbersome If you use an e-mail message with a link to a Web page, you will probably need a computer package to build the survey on a Web page The questionnaire is automatically sent to a collection point ready for analysis A major consideration of both methods is how to load the email addresses for an easy dissemination of the survey 28 Typical Costs: The e-mail only method is very cost efficient It requires only employee time to enter the survey form and then to enter the responses when they return The e-mail with Web page version is more costly A software package may cost $6000 with a $2000 annual service fee However, the data are already entered Time Required: This can be the most efficient method of collecting information It will take a week to build the customer base and develop the survey and enter it Within an hour, you can send out the survey You can then send out a reminder within a day or two Staffing Needs: In addition to people constructing the questionnaire, you will need an experienced person to work with the Internet If you have open-ended questions, someone will need to analyze them Benefits of Using this Approach You can send reminders very easily to the respondents Do not send too many though Data can be collected very quickly This can be a very low cost method Cautions in Using this Approach People may view your survey as another spam related message The response rate will vary greatly Not all customers you want to survey will have email addresses Some of your customers will not be very computer literate and will refuse to the survey If you the email/Web page version, you may need to attach a Word or Word Perfect file containing the survey 29 What to Ask No matter what kind of research method you choose, there are certain questions you should ask The Customer Satisfaction Work Group analyzed many surveys from state and federal LMI agencies and developed nine (9) attributes which should be assessed about LMI products They are: Accuracy: The information is accurate enough for the customer’s use Relevancy: The information is relevant to solving the customer’s problem Accessibility: The customer could easily access the information Understandability: The customer could understand what the content means Comparability: The customer is able to compare the information with other information they use Geographic Detail: The information is in sufficient geographic detail Timely: The information is timely enough for the customer’s needs Completeness: The customer could solve the problem with this information only Importance: The information is important in the customer’s overall problem solving work You should also ask a general satisfaction question (“Overall, how would you rate your safisfaction with this product?”), some demographic questions (“Are you in business or an educator?”), some questions about customer service (“Was our staff helpful?”), and, most important, there should be room for open ended comments The following page is an example of a written survey with all of those characteristics This survey can also be interpolated into questions to ask during a focus group or interview 30 This is a sample survey which includes basic important questions for all survey methods on LMI products Feel free to modify it as needed Notice at the bottom of the form you can indicate from which data source(s) the information originates from You also may want to enter in the top line for the customer Customer Satisfaction Survey What is the name of the information product you are using Describe the product Please indicate all the purposes for which you use this product (Check all that apply) : ❑ Product markets ❑ Compensation ❑ State/local budgeting ❑ Financial market ❑ Training ❑ State/local policies ❑ Business location ❑ Career planning ❑ Develop curriculum ❑ Employee recruitment ❑ State/local policy making ❑ Assist others finding employment ❑ Business planning ❑ Economic development ❑ Inform the pubic ❑ Business budgeting ❑ Find a job What is the one main purpose for which you use this information: Please indicate the broad category which applies to the person or organization using this information: ❑ Business ❑ Researcher ❑ Job Seeker ❑ Student ❑ Elected Officials ❑ State or Federal Policy Maker ❑ Education & Training Provider ❑ Researcher ❑ One-Stop Employee ❑ Media ❑ Workforce Program Planners ❑ Intermediary helping people find employment (such as a career counselor) On a scale of to 5, please circle the appropriate rating of the following attributes of the information product you are using: The information is accurate enough for my use Strongly Disagree Disagree Neutral Agree Strongly Agree N/A The information is relevant to solving my problem Strongly Disagree Disagree Neutral Agree Strongly Agree N/A I could easily access the information Strongly Disagree Disagree Neutral Agree Strongly Agree Strongly Disagree N/A I understand what the content of the information means Strongly Disagree Disagree Neutral Agree Strongly Agree N/A Neutral Agree Strongly Agree N/A The information is in sufficient geographic detail for me Strongly Disagree Disagree Neutral Agree Strongly Agree Disagree Neutral Agree Strongly Agree N/A The information is important3to my decision making process Strongly I am satisfied with this informationStrongly Agree Agree Overall,Disagree Disagree Neutral product Agree Strongly Agree N/A Strongly Disagree Disagree Neutral N/A I was treated courteously by a labor market information Icontact was treated courteously by a labor market information contact Disagree Disagree Neutral Strongly Agree Strongly Agree Agree Strongly Agree N/A Strongly Disagree Disagree Neutral I am able to compare this information with other information I use Strongly Disagree Disagree The Information is timely enough for my needs Strongly Disagree Disagree The information is timely Neutral for my needs Agree enough Agree Strongly N/A Strongly1 Disagree Disagree Neutral Agree 4Strongly Agree N/A I am able to solve my information needs with this information to solve my information needs with this I am able only Strongly Disagreeonly Disagree Neutral Agree Strongly Agree information N/A Strongly1 Disagree Disagree Neutral Agree 4Strongly Agree N/A The information is important3 my decision making process to N/A N/A My questions were adequately answered by a staff person My questions were adequately answered by a staff person Strongly Disagree Disagree Neutral Agree Strongly Agree Agree Strongly Agree N/A Strongly Disagree Disagree Neutral N/A Comments: Thank You for your participation OES ❑ CES ❑ LAUS ❑ ES202 ❑ OFFICE USE ONLY LEHD ❑ Benefits ❑ Job vacancy ❑ Other ❑ _ 31 Step – Conduct Research and Take Action! Now that you have outlined what you will and how you will it, DO IT! The results of your research should be delivered as a report If you use a survey instrument you will need to pretest it to ensure that it can be understood by those you are surveying Ask one of your best customers to try it! You may want to consider two sections to the final report of your findings: comments and recommendations Comments are the general comments or quantitative results from the respondents Recommendations are actionable items the research has uncovered Give the report to the Evaluation Team for action! Documenting Demand In addition to finding out the level of customer satisfaction, the Customer Satisfaction Work Group recommends that states document the demand for products and services This may help in future marketing efforts, creating a customer survey base, and proving return on investment for LMI products There are four types of demand that should be documented: Statistics (counting usage) Web site visits Specific page hits (Web Metrix) Visits – time on each page, data downloaded Customer lists (counting customers) By document By customer group Other Formal Satisfaction Feedback Special Surveys Focus Groups Directed Interviews 32 Anecdotal (counting and quoting comments) Testimonials “I can’t live without it” Proof of Usage “This is how I use it and its value to me” Very Important Notice that part of documenting demand calls for the creation of customer lists Since many states are delivering products over the Internet, it is difficult to attach a name and address to many product users States may want to consider having users register, get onto mailing lists or update lists, and other methods of gathering names, addresses, and phone numbers of users It is also important to record customer contact information if they call, write e-mail or fax requests for information or data Conclusion In an era of continually needing to prove the value of labor market information products and services, the nationwide LMI system needs to build relationships with the customers it serves To that, federal and state agencies need to undertake an aggressive program to actively listen to customers and respond with improved and innovative products This handbook is designed to help agencies measure customer satisfaction 33 ...Table of Contents Topic Page Introduction – Why Measure Customer Satisfaction? How to Measure Customer Satisfaction Step – Pick a Product or Customer Group Step – Set the Stage for Assessment What... level of customer satisfaction, the Customer Satisfaction Work Group recommends that states document the demand for products and services This may help in future marketing efforts, creating a customer. .. who use them Customer satisfaction assessment is a great opportunity for states to make positive changes in the future of the labor market information system! How to measure customer satisfaction?