Chapter 7 collecting data

Chapter COLLECTING DATA Presenter Le Thi Thanh Tam Contents  Sources of Data  Sampling  Interview  Coding Responses  Existing Statistical Data  Ethical Issues in Collecting Data SOURCE OF DATA Informal interviews Observations Formal interviews Written questionnaire Program Record Data from Other Institutions Other Sources of Data Informal interviews  Informal discussions: helpful at the beginning for evaluator to orient her/himself and the study of the situation  Some evaluation studies are mainly qualitative and informal interviews are the major source of information They rely on discussions and observations for their data Observations • • • Embodying a look- around- and- see perspective, watching the program implementation in a variety of settings over time, taking notes & recording what the evaluator can see In some cases, the evaluator will develop a protocol for recording specific behaviors during the observation period Evaluator can tape- record sessions and code the events later If physical properties of the situation are important, the session can be videotaped Does not involve asking anybody anything -> does not usually introduce the biases that obtrusive questioning can bring about Time consuming & remain vulnerable to bias Observer is the instrument and bring her own set of blinders to the task People who know they are being watched may act differently – at least for a while Formal interviews A systematic method for obtaining data Open- ended interview  Starts with list of topics to be covered, a clear sense of what kinds of information are wanted  Interviewer follows up each lead with neutral probes  Where possible, the interviewer will tape record the interviews  When the evaluator want to generalize across them => combining and interpreting the responses 1/ reading – rereading the responses => gathers together narrative answers on the same topic from different respondents  developing a growing awareness of the main themes represented in the responses 2/ coding the answers into a set of categories than count the number of answers that fall into each category Structured interview (survey interview)  Interview form  Ask every respondent the same questions The interviewer starts by explaining the purposes of data collection and works through the questions  Avoid biasing the answers whether by tone of voice, body language or reactions to the respondent’s replies Try not to give any cue, stress that she wants the straight facts, respondent’s candid opinions  Interviews can be conducted in person or by telephone Telephone interview is a good option if: + Respondents are scattered across a wide geographic area + Most respondents have home phones + It is relatively easy to find out phone numbers  Telephone interview is usually less expensive, sometimes have an advantage of impersonality  Inter-person interviews is economical when all the respondents are available in one or a few locations… Written questionnaires Advantages: - Allow the evaluator to ask a broad range of questions - Respondent can think about his responses - More economical than interviews Main disadvantages Low response rate, those who respond to questionnaire maybe atypical of the program population => - Formatting, wording in ways that encourage participation - Ensure that the answer categories represent answers that respondents really want to give - Keep questionnaire relatively short - Consider offering incentives - Definitely send follow – up mailings They require a certain degree of literacy Group-administered questionnaire If the evaluator can collect numbers of participants in one room, she can hand out questionnaires and ask each person to fill out the questionnaire on the spot Program records A natural for evaluation data  Save time and money  Continuity: the agency reporting system provides continuous data over long periods  Avoid bothering clients and staff for additional information, reduces the burden on staff and participants However, - Data on computers is no guarantee that they are up to date or accurate => check the record- keeping system to find out Who? Which? What interval? When and How? How frequently? - Agencies sometimes change their systems: items, procedures for data entry, data checking, etc => Rules have to be worked out in advance that alert the evaluator to any pending changes ,when, how Data from other Institutions Advantages: The authoritative character of official records The savings in time, money and effort However:  Using agency data has a cost  Public concerns about protecting confidentiality and privacy =>legal safeguards, restricted access  There are and should be limits on the use of records because incursions on people’s privacy is a matter of growing concern, and evaluators have a responsibility to limit their requests to essential items and wherever possible, use aggregate data as indicators of outcome   Other Sources of Data Many other sources, methods can provide information for the evaluator’s purpose:  Test of information, interpretation, skills, application of knowledge  Situational tests presenting the respondent with simulated life situations  Simulation games  Psychometric test of attitudes, values, personality, preferences, norms, beliefs  Projective tests  Diaries of events, usually kept explicitly at the evaluator’s request  Focus groups that that bring together 6-16 people to discuss a subject, with the evaluator acting as moderator  Physical evidence  Clinical examinations  Physical tests for nicotine, drugs, alcohol  Review of documents ( minutes of board meetings, transcripts of trials)  Analysis of financial records  Content analysis of media stories  Photograph, videotapes… SAMPLING Two basic ways Purposive sampling Picking particular people for particular reasons When? - To probe the extent to which a program is effective with all of participants or to establish the limits of the program’s effectiveness => select participants at the extremes or select people who represent conditions at the tails of distribution - To address specific policy questions - When lists of participants are not available or when participant’s refusals are very frequent and would undermine the randomness of the sample Convenient but except under special circumstances and for important reasons, it is not generally as justifiable as random sampling Random sampling • Ensure that the units chosen are representative of the population Every unit has to have a known chance of being selected • Allows the evaluator to generalize from the responses of the sampled respondents to the entire population If the sample is drawn from the total population of program participants, study results can be assumed, with the known degree of confidence, to reflect the experiences and views of all the people in the program [...]...Other Sources of Data Many other sources, methods can provide information for the evaluator’s purpose:  Test of information, interpretation, skills, application of knowledge  Situational tests presenting the respondent ...Contents  Sources of Data  Sampling  Interview  Coding Responses  Existing Statistical Data  Ethical Issues in Collecting Data SOURCE OF DATA Informal interviews Observations... systems: items, procedures for data entry, data checking, etc => Rules have to be worked out in advance that alert the evaluator to any pending changes ,when, how Data from other Institutions... the spot Program records A natural for evaluation data  Save time and money  Continuity: the agency reporting system provides continuous data over long periods  Avoid bothering clients and