The Science and Art of Implementing Quantitative Evaluation Surveys

Sample Size Calculations, cont’d Sample Size Calculations, cont’d  Finally, to account for clustering Finally, to account for clustering..  Suppose I intend to survey 100 schools (50 t[r]

The Science and Art of Implementing The Science and Art of Implementing

Quantitative Evaluation Surveys Quantitative Evaluation Surveys

John Hoddinott John Hoddinott

Deputy Director

Deputy Director

Food Consumption and Nutrition Division

Introduction Introduction

 Conducting a quantitative evaluation survey is both a Conducting a quantitative evaluation survey is both a

science and an art.

science and an art.

 The science pertains to the construction of a sample that The science pertains to the construction of a sample that

is representative of the population of interest

is representative of the population of interest

 The art pertains to the implementation of a survey The art pertains to the implementation of a survey

instrument (such as a questionnaire) that generates the

instrument (such as a questionnaire) that generates the

information you need for your evaluation

The science … The science …

 Deciding on your unit of observationDeciding on your unit of observation

 Describing the universe: The sample frameDescribing the universe: The sample frame  Drawing the sampleDrawing the sample

The Unit of Observation

The Unit of Observation

 The unit of observation is simply the unit that is of The unit of observation is simply the unit that is of

interest given the objectives of your study:

interest given the objectives of your study:

• StudentsStudents

• Young womenYoung women

The Sampling Frame

The Sampling Frame

The frame for a sample is a list of the units in the

The frame for a sample is a list of the units in the

population (or universe) from which the units that will be

population (or universe) from which the units that will be

enumerated in the sample area are selected It may be an

enumerated in the sample area are selected It may be an

actual list, a set of index cards, a map, or data stored in a

actual list, a set of index cards, a map, or data stored in a

computer The frame is a set of physical materials

computer The frame is a set of physical materials

(census statistics, maps, lists, directories, records) that

(census statistics, maps, lists, directories, records) that

enables us to take hold of the universe piece by piece

enables us to take hold of the universe piece by piece

(Casley and Lury, 1987, p 52)

The Sampling Frame, cont’d

The Sampling Frame, cont’d

 It is important that you examine carefully any sampling It is important that you examine carefully any sampling

frame that is made available to you for:

frame that is made available to you for:

• DuplicationsDuplications

• Errors (such as inclusion errors)Errors (such as inclusion errors)

• OmissionsOmissions

 The concern is that errors of inclusion or omission are The concern is that errors of inclusion or omission are

non-random

non-random

• Lists of studentsLists of students

Drawing the sample Drawing the sample

 Probabilistic sampling methods use some mechanism Probabilistic sampling methods use some mechanism

involving chance to determine which observations appear

involving chance to determine which observations appear

in the sample These mechanisms include:

in the sample These mechanisms include:

• Systematic samplingSystematic sampling

• Systematic random samplingSystematic random sampling

• Stratified random samplingStratified random sampling

Sampling Sampling

 Systematic methods involve the selection of every nth Systematic methods involve the selection of every nth

observation

observation

 For example, suppose we want a sample of 250 observations For example, suppose we want a sample of 250 observations

from our population of 1000 students

from our population of 1000 students

 We could take the first student on our list, the fifth, the ninth We could take the first student on our list, the fifth, the ninth

and so on This method is relatively straightforward

and so on This method is relatively straightforward

 The drawback is that the ordering of firms from to 1000 The drawback is that the ordering of firms from to 1000

must be random If there is some subtle, difficult-to-observe

must be random If there is some subtle, difficult-to-observe

ordering of the sample (for example, older children tend to

ordering of the sample (for example, older children tend to

be counted as even numbers) the observations drawn will

be counted as even numbers) the observations drawn will

not be a random sampling of the population.

Sampling, cont’d Sampling, cont’d

 Simple random sampling is a better alternative.Simple random sampling is a better alternative.

 The simplest way to this is to use a statistics package The simplest way to this is to use a statistics package

like STATA

like STATA

 Suppose we have a listing of 1000 students and we want Suppose we have a listing of 1000 students and we want

to randomly select 50 of them for interview We use the

to randomly select 50 of them for interview We use the

command:

command:

Sampling, cont’d Sampling, cont’d

 There is a potential weakness with this approach Suppose we There is a potential weakness with this approach Suppose we

are drawing a sample of 100 students from a population of 1000 are drawing a sample of 100 students from a population of 1000 We know that 30% of these have completed grades 5-8 so our We know that 30% of these have completed grades 5-8 so our sample should contain 30 such students However, this is only sample should contain 30 such students However, this is only true on average! Though the likelihood is high that our sample true on average! Though the likelihood is high that our sample will contain 30 such students, it is also possible that it contains will contain 30 such students, it is also possible that it contains 20, 25 or 40.

20, 25 or 40.

 The solution to this problem is random stratified sampling The The solution to this problem is random stratified sampling The

first step is to divide the population into groups or strata Here, first step is to divide the population into groups or strata Here, the division would be between the 300 students in grades 5-8 the division would be between the 300 students in grades 5-8 and 700 other students Using the random number method, and 700 other students Using the random number method,

select 10% of students in each category, so the resultant sample select 10% of students in each category, so the resultant sample contains 30 students in grades 5-8 and 70 others The

contains 30 students in grades 5-8 and 70 others The proportions in the sample are identical to those in the proportions in the sample are identical to those in the underlying population.

Sampling, cont’d Sampling, cont’d

 The final approach is to use cluster based sampling.The final approach is to use cluster based sampling.  Here, you select a unit of observation (eg a school) and Here, you select a unit of observation (eg a school) and

sample within it.

sample within it.

 Cluster based sampling is especially appropriate in the Cluster based sampling is especially appropriate in the

context of randomized designs where randomization

context of randomized designs where randomization

occurs at the cluster level.

Sample Size Calculations Sample Size Calculations

Recall some basic statistical concepts:

Recall some basic statistical concepts:

• Significance levelSignificance level The probability of rejecting a null The probability of rejecting a null

hypothesis that is true – also called Type I errors This is hypothesis that is true – also called Type I errors This is

often expressed as a percentage so that a test of often expressed as a percentage so that a test of

significance level,

significance level, αα, is referred to as a 100, is referred to as a 100αα% level test% level test

• PowerPower The probability, 1- The probability, 1- ββ of correctly rejecting a null of correctly rejecting a null hypothesis that is false

hypothesis that is false

• For a given sample size, there is a trade off increasing power For a given sample size, there is a trade off increasing power and reducing Type I errors

and reducing Type I errors

• We can only increase power and reduce Type I errors We can only increase power and reduce Type I errors simultaneously by increasing sample size

Sample Size Calculations, cont’d Sample Size Calculations, cont’d

 In addition, in order to calculate sample sizes, we need to In addition, in order to calculate sample sizes, we need to

know:

know:

• The size of the impact that we would like to detectThe size of the impact that we would like to detect

• Estimates of standard deviationsEstimates of standard deviations

• (If using a cluster design), estimates of intra-cluster (If using a cluster design), estimates of intra-cluster correlation, also called the design effect

correlation, also called the design effect

Sample Size Calculations, cont’d Sample Size Calculations, cont’d

 (If using a clustered design), first calculate the intra-(If using a clustered design), first calculate the

intra-cluster correlation coefficient:

cluster correlation coefficient:

 Suppose we have the variable score and our clusters are Suppose we have the variable score and our clusters are

defined by the variable, school_id

defined by the variable, school_id

 The command:The command:

• loneway score school_idloneway score school_id

Sample Size Calculations, cont’d Sample Size Calculations, cont’d

 Suppose I want to detect a 20% increase as a result of my Suppose I want to detect a 20% increase as a result of my

intervention (from 100 for the control group to 120 for the

intervention (from 100 for the control group to 120 for the

treatment group)

treatment group)

 I want to have statistical power of 0.80.I want to have statistical power of 0.80.  Standard deviation is 60.Standard deviation is 60.

 I run the Stata command:I run the Stata command:

Sample Size Calculations, cont’d Sample Size Calculations, cont’d

Test Ho: m1 = m2, where m1 is the mean in population 1

Test Ho: m1 = m2, where m1 is the mean in population 1

and m2 is the mean in population 2and m2 is the mean in population 2 Assumptions:

Assumptions:

alpha = 0.0500 (two-sided)alpha = 0.0500 (two-sided)

power = 0.8000power = 0.8000

m1 = 100m1 = 100

m2 = 120m2 = 120

sd1 = 60sd1 = 60

sd2 = 60sd2 = 60

n2/n1 = 1.00n2/n1 = 1.00

Estimated required sample sizes:

Estimated required sample sizes:

n1 = 142n1 = 142

Sample Size Calculations, cont’d Sample Size Calculations, cont’d  Finally, to account for clusteringFinally, to account for clustering

 Suppose I intend to survey 100 schools (50 treatments Suppose I intend to survey 100 schools (50 treatments

and 50 controls) and the intra-cluster correlation is 0.35

and 50 controls) and the intra-cluster correlation is 0.35

 I runI run

• sampclus, obsclus (25) rho (0.30)sampclus, obsclus (25) rho (0.30)

Sample Size Calculations, cont’d Sample Size Calculations, cont’d

 Sample Size Adjusted for Cluster DesignSample Size Adjusted for Cluster Design

 n1 (uncorrected) = 142n1 (uncorrected) = 142  n2 (uncorrected) = 142n2 (uncorrected) = 142

 Intraclass correlation = 3Intraclass correlation = 3

 Average obs per cluster = 25Average obs per cluster = 25  Minimum number of clusters = 94Minimum number of clusters = 94

 Estimated sample size per group:Estimated sample size per group:

Sample Size Calculations: Concluded Sample Size Calculations: Concluded

 The final point to note here is that having insufficient The final point to note here is that having insufficient

sample sizes dooms the evaluation survey even before

sample sizes dooms the evaluation survey even before

you leave the office

you leave the office

 You need to make these calculations on the basis of You need to make these calculations on the basis of

conservative assumptions regarding:

conservative assumptions regarding:

• Effect sizeEffect size

The art The art  The “Hard Arts”The “Hard Arts”

• Questionnaire design and length; pilot testingQuestionnaire design and length; pilot testing

• Use of PDAsUse of PDAs

• Issues in data entry, checking and cleaningIssues in data entry, checking and cleaning

• BudgetsBudgets

 The “Soft Arts”The “Soft Arts”

• Selecting enumeratorsSelecting enumerators

• Selecting SupervisorsSelecting Supervisors

• Interacting with respondentsInteracting with respondents

– Informed consentInformed consent

– PaymentsPayments

The Hard Arts The Hard Arts  TopicsTopics

• Questionnaire design and length; pilot testingQuestionnaire design and length; pilot testing

• Use of PDAsUse of PDAs

• Issues in data entry, checking and cleaningIssues in data entry, checking and cleaning

Questionnaire Design Questionnaire Design

 Getting this right is critical to the success of your project.Getting this right is critical to the success of your project.  The best way to design the questionnaire is to ‘work The best way to design the questionnaire is to ‘work

backwards’ That is, start by thinking about what your

backwards’ That is, start by thinking about what your

report will look like:

report will look like:

• What are the outcomes that you want to measure?What are the outcomes that you want to measure?

• (In the case of non-randomized designs), what variables (In the case of non-randomized designs), what variables determine participation What covariates would you put in determine participation What covariates would you put in

Questionnaire Design, cont’d Questionnaire Design, cont’d

• Do treatment observations actually receive the treatment? Do treatment observations actually receive the treatment? Do they receive only partial treatment? Are there problems Do they receive only partial treatment? Are there problems

with quality? What constraints/problems did they face in with quality? What constraints/problems did they face in

accessing intervention accessing intervention

– Helps explain why you might not find significant impactHelps explain why you might not find significant impact

– Allows you to set up a “treatment on the treated” model as an Allows you to set up a “treatment on the treated” model as an alternative to your “intent to treat”

alternative to your “intent to treat”

– Operational details are of considerable interest to program Operational details are of considerable interest to program managers

managers

• What are the characteristics of your sample? Are there What are the characteristics of your sample? Are there particular sub-groups that you want to identify?

Questionnaire Design, cont’d Questionnaire Design, cont’d  Practical Considerations:Practical Considerations:

• Develop a logical sequence of questions – think of this as a Develop a logical sequence of questions – think of this as a conversation rather than an interview

conversation rather than an interview

• Start with easy/gently questions of the “tell me about Start with easy/gently questions of the “tell me about yourself” type

yourself” type

• Consider which questions should be pre-coded and which Consider which questions should be pre-coded and which should be open-ended For example, a question on marital should be open-ended For example, a question on marital

status could be status could be

– Precoded: Precoded: 1 if single; if married; if widowed; if if single; if married; if widowed; if divorced; if other

divorced; if other

– Open ended: How would you describe your marital Open ended: How would you describe your marital status?

status?

• Use skip patterns:Use skip patterns:

– “If yes, go to Q10”“If yes, go to Q10”

Questionnaire Design, cont’d Questionnaire Design, cont’d

• Think about recall periods:Think about recall periods: – TelescopingTelescoping

– Important but infrequent eventsImportant but infrequent events

• Think carefully about how to approach sensitive topicsThink carefully about how to approach sensitive topics

• Break down complex questions into partsBreak down complex questions into parts – Not How much did you earn from wage work.Not How much did you earn from wage work.

– Better How many hours per day did you work in the last seven Better How many hours per day did you work in the last seven days? How many days per week did you work in the last seven days? How many days per week did you work in the last seven days? How much are you paid for this wage work, allowing days? How much are you paid for this wage work, allowing respondent to answer in terms of hourly, daily or weekly wage respondent to answer in terms of hourly, daily or weekly wage

Questionnaire Design: Pilot Testing Questionnaire Design: Pilot Testing

 Before starting your survey, you need to make sure that Before starting your survey, you need to make sure that

your questionnaire works – this is called pilot testing.

your questionnaire works – this is called pilot testing.

 You should try the questionnaire on 10-20 respondents, You should try the questionnaire on 10-20 respondents,

who represent a variety of respondent ‘types’

who represent a variety of respondent ‘types’

 Pilot testing should reveal the following:Pilot testing should reveal the following:

a)

a) Are definitions used in the questionnaire appropriate This Are definitions used in the questionnaire appropriate This applies both to definitions of units of observation (does the applies both to definitions of units of observation (does the definition of a household correspond with the definition definition of a household correspond with the definition used by the people being studied) and to particular

used by the people being studied) and to particular questions (eg "holdings"; "assets"; "income")

questions (eg "holdings"; "assets"; "income") b)

b) Do respondents understand the questionsDo respondents understand the questions c)

Pilot Testing, cont’d Pilot Testing, cont’d

d)

d) Are questions being asked that cause respondents unease Are questions being asked that cause respondents unease or they refuse to answer

or they refuse to answer e)

e) Are there problems associated with translating particular Are there problems associated with translating particular concepts

concepts f)

f) Is the layout and sequencing of questions sensibleIs the layout and sequencing of questions sensible

g)

g) Can greater use be made of pre-coding Can greater use be made of pre-coding

h)

Pilot Testing, cont’d Pilot Testing, cont’d

 After pilot testing, review results with enumeratorsAfter pilot testing, review results with enumerators  Depending on results, re-do with a smaller number of Depending on results, re-do with a smaller number of

respondents (5-10)

Questionnaire design: A final word Questionnaire design: A final word

 When doing an evaluation survey with a longitudinal When doing an evaluation survey with a longitudinal

design, it is extremely important that at the time of the

design, it is extremely important that at the time of the

baseline, you obtain information on how to contact

baseline, you obtain information on how to contact

respondents in the future:

respondents in the future:

• GPS is idealGPS is ideal

• Cellphone numbersCellphone numbers

• Information on people who could aid in a follow up contactInformation on people who could aid in a follow up contact

 Not only is this information helpful, it can also be used as Not only is this information helpful, it can also be used as

additional regressors in attrition probits

Using PDAs or handheld computers in surveys Using PDAs or handheld computers in surveys  There is increasing interest in using Personal Digital There is increasing interest in using Personal Digital

Assistants (PDAs) such as PalmPilots or handheld

Assistants (PDAs) such as PalmPilots or handheld

computers for data collection.

computers for data collection.

 Using these requires:Using these requires:

• Purchasing the hardware ($200-$500 each)Purchasing the hardware ($200-$500 each)

• Purchasing the software, such as Pendragon Forms or PC Purchasing the software, such as Pendragon Forms or PC Pocket Creations

Pocket Creations

• Some one to write the data entry program using this Some one to write the data entry program using this software

Handhelds: Advantages Handhelds: Advantages

 Speed: Data are available for use immediatelySpeed: Data are available for use immediately

 Filters and skips can ensure that unnecessary questions Filters and skips can ensure that unnecessary questions

are not asked

are not asked

Handhelds: Non-Issues Handhelds: Non-Issues  Its harder to train enumeratorsIts harder to train enumerators

• Not in our experience.Not in our experience.

 Battery life is a problemBattery life is a problem

• Buy extra batteriesBuy extra batteries

 Transferring data to computers is hardTransferring data to computers is hard

Handhelds: Disadvantages Handhelds: Disadvantages  Need access to reliable source of electricityNeed access to reliable source of electricity  Sometimes hard to see in bright daylightSometimes hard to see in bright daylight

 In our experience, data entry errors are more frequent In our experience, data entry errors are more frequent

when using PDAs compared to paper questionnaires and

when using PDAs compared to paper questionnaires and

back office data entry, although these may converge with

back office data entry, although these may converge with

further experience

further experience

 In our experience, surveys take slightly longer when using In our experience, surveys take slightly longer when using

handhelds for numeric information and much longer for

handhelds for numeric information and much longer for

text Consequently, enumerators are much less likely to

Handhelds: Disadvantages Handhelds: Disadvantages

 If a response does not seem quite right, it is relatively If a response does not seem quite right, it is relatively

straightforward to take the questionnaire back into the

straightforward to take the questionnaire back into the

field Note too that checking one response often leads to

field Note too that checking one response often leads to

revisions to other responses.

revisions to other responses.

 If a PDA is lost, then all data are lost In some cases (such If a PDA is lost, then all data are lost In some cases (such

as Pocket PCs, this is also true if the battery dies)

as Pocket PCs, this is also true if the battery dies)

 Safety/security of enumerators – carrying an expensive Safety/security of enumerators – carrying an expensive

electronic device can make them a target for criminals

Data Entry, Cleaning, and Management Data Entry, Cleaning, and Management

 Researchers typically spend a lot of time thinking about Researchers typically spend a lot of time thinking about

critical issues at the outset of their project (sample size

critical issues at the outset of their project (sample size

calculations, questionnaires) and at the end of the project

calculations, questionnaires) and at the end of the project

(data analysis, report writing).

(data analysis, report writing).

 They typically spend less time worrying about the They typically spend less time worrying about the

intermediate phase: data entry, cleaning and

intermediate phase: data entry, cleaning and

management.

management.

 This is a mistake Many evaluation studies fall apart, or This is a mistake Many evaluation studies fall apart, or

fall badly behind schedule, because insufficient attention

fall badly behind schedule, because insufficient attention

is paid to data management

Data Management, cont’d Data Management, cont’d  Avoiding these problems requires:Avoiding these problems requires:

• Paying attention to the design and implementation of the Paying attention to the design and implementation of the data entry software early

data entry software early

Data Management, cont’d Data Management, cont’d  Software Good choices include:Software Good choices include:

• CS PRO (CS PRO (www.cspro.orgwww.cspro.org))

• Microsoft Access (often bundled with Microsoft Office)Microsoft Access (often bundled with Microsoft Office)

• SPSS/Data Entry Module (but this can be expensive)SPSS/Data Entry Module (but this can be expensive)

 Start work on data entry programs as soon as Start work on data entry programs as soon as

questionnaire design (or designs of certain modules) is

questionnaire design (or designs of certain modules) is

finalized

finalized

Data Management, cont’d Data Management, cont’d  Example of data management protocol:Example of data management protocol:

1.

1. Enumerator checks questionnaire for completeness before Enumerator checks questionnaire for completeness before giving it to supervisor

giving it to supervisor 2.

2. Supervisor does quick check to make sure form is complete Supervisor does quick check to make sure form is complete and that critical topics are correct She passes form to:

and that critical topics are correct She passes form to: 3.

3. Data checkers/verifiers who go through form in detail Data checkers/verifiers who go through form in detail Forms are sent back to supervisors/enumerators for Forms are sent back to supervisors/enumerators for

correction Otherwise, they are given to data entry team. correction Otherwise, they are given to data entry team. 4.

4. Data are entered:Data are entered:

– Question as to whether to use single or double data entryQuestion as to whether to use single or double data entry

– Compromise; enter 10-20% of data twice to quality Compromise; enter 10-20% of data twice to quality check on entry

check on entry 5.

5. Violations of range/value are sent back to field for checkingViolations of range/value are sent back to field for checking 6.

6. Additional checks are made when variable aggregates are Additional checks are made when variable aggregates are constructed

Data Management, cont’d Data Management, cont’d  Finally, two major considerations:Finally, two major considerations:  You need system to keep track of You need system to keep track of

questionnaires – these cannot be lost! questionnaires – these cannot be lost!  Ensure data are backed up regularly.Ensure data are backed up regularly.

• Minimum once a weekMinimum once a week

• This means that you have more than one copy held in This means that you have more than one copy held in

different locations

Budgets Budgets

 Budgeting for quantitative surveys has three Budgeting for quantitative surveys has three

components:

components:

• What you need?What you need?

• How many _ you need?How many _ you need?

Budgets: What you need? Budgets: What you need?

 Typical line items in a survey budget:Typical line items in a survey budget:

• Pilot testing of questionnairesPilot testing of questionnaires

• Training of survey staffTraining of survey staff

• Enumerators (salary, payroll taxes and possibly per diem)Enumerators (salary, payroll taxes and possibly per diem)

• Supervisors (salary payroll taxes and possibly per diem)Supervisors (salary payroll taxes and possibly per diem)

• Drivers and TransportDrivers and Transport

• Printing of questionnairesPrinting of questionnaires

• Supplies (clipboards, pens, first aid kits etc)Supplies (clipboards, pens, first aid kits etc)

• Payments to respondents(?)Payments to respondents(?)

• VerifiersVerifiers

• Training of data entry staffTraining of data entry staff

Budget: How many Budget: How many  ““How many” depends on:How many” depends on:

• The size of the sample you are collectingThe size of the sample you are collecting

• Whether it is concentrated in a few places or widely Whether it is concentrated in a few places or widely disbursed

disbursed

• How quickly you want to complete the surveyHow quickly you want to complete the survey

 Example.Example.

• My survey is working in 40 spatially disbursed locations.My survey is working in 40 spatially disbursed locations.

• I am interviewing 30 households in each location I am interviewing 30 households in each location

• Each interview lasts approximately two hours and so I Each interview lasts approximately two hours and so I

assume that an interviewer can complete two interviews per assume that an interviewer can complete two interviews per

day day

• I assume that enumerators work five days, then have one I assume that enumerators work five days, then have one day off

Budget: How many Budget: How many

 If I hire three enumerators, they will complete the survey If I hire three enumerators, they will complete the survey

in one locality in five days

in one locality in five days

• Each day, interviews are completed (3 enumerators x interviews Each day, interviews are completed (3 enumerators x interviews

per day)

per day)

• In five days, 30 interviews are completed (6 per day x days)In five days, 30 interviews are completed (6 per day x days) • One day to move to new site, one day offOne day to move to new site, one day off

• Repeat three timesRepeat three times

 This implies that one team of three enumerators will cover This implies that one team of three enumerators will cover

four localities in one month

four localities in one month

 So, ten teams of enumerators (30 enumerators in total) So, ten teams of enumerators (30 enumerators in total)

will complete survey over a four week period

will complete survey over a four week period

 Suppose you allocate one supervisor to two teams.Suppose you allocate one supervisor to two teams.

Topics Topics  Selecting enumeratorsSelecting enumerators

 Selecting SupervisorsSelecting Supervisors

 Interacting with respondentsInteracting with respondents

• Preparing the ground Preparing the ground

• Interview settingInterview setting

• Interacting with respondents, including:Interacting with respondents, including:

– Informed consentInformed consent – PaymentsPayments

Enumerators Enumerators

 The job description for the ideal enumerator would The job description for the ideal enumerator would

include:

include:

• Communications skillsCommunications skills

• Good knowledge of English (or French, or Portuguese or Good knowledge of English (or French, or Portuguese or Spanish) as well as the local language(s)

Spanish) as well as the local language(s)

• A perceptive intelligence,A perceptive intelligence,

• inexhaustible patienceinexhaustible patience

• unfailing dependabilityunfailing dependability

• Wonderful people skillsWonderful people skills

• Willingness to work long hoursWillingness to work long hours

• An ability to get along with all elements of the local An ability to get along with all elements of the local population

population

Enumerators, cont’d Enumerators, cont’d  Some considerations:Some considerations:

• Level of schoolingLevel of schooling

• GenderGender

Supervisors Supervisors

 Supervisors should be able to the followingSupervisors should be able to the following

a)

a) Maintaining time schedules and co-ordinating workMaintaining time schedules and co-ordinating work b)

b) Checking the enumerator's work by sitting in on interviewsChecking the enumerator's work by sitting in on interviews c)

c) Checking completed questionnaires for misunderstandings Checking completed questionnaires for misunderstandings and recording errors

and recording errors d)

d) Liaising and trouble-shooting with respondents and local Liaising and trouble-shooting with respondents and local community leaders

community leaders e)

e) Acting as courier - transferring equipment and documents Acting as courier - transferring equipment and documents from the office to the field and vice versa

from the office to the field and vice versa f)

f) Covering in emergencies (eg arranging interviews if an Covering in emergencies (eg arranging interviews if an enumerator falls sick)

enumerator falls sick) g)

Interacting … : Preparing the ground Interacting … : Preparing the ground  You can’t just show up and start asking questions.You can’t just show up and start asking questions.

 In both rural and urban areas, even if you have research In both rural and urban areas, even if you have research

clearance, it is a good idea to contact local government

clearance, it is a good idea to contact local government

officials to inform them about the survey work you intend

officials to inform them about the survey work you intend

to carry out.

to carry out.

 It is also helpful to have a standard ‘story’ or explanation It is also helpful to have a standard ‘story’ or explanation

that you and your research team can provide to any one

that you and your research team can provide to any one

curious about your work.

curious about your work.

Interacting with respondents: Informed consent Interacting with respondents: Informed consent  Before an interview can take place, you must obtain Before an interview can take place, you must obtain

“informed consent” from your respondents.

“informed consent” from your respondents.

 Specifically, you need to explain to Specifically, you need to explain to those being those being

interviewed:

interviewed:

• The purpose of their studyThe purpose of their study

• That participation is voluntaryThat participation is voluntary

• Who has funded and supported itWho has funded and supported it • How the data will be collectedHow the data will be collected

• Who they can contact if they have questionsWho they can contact if they have questions

• How the results may be used.How the results may be used.

• How researchers will maintain anonymity and/or confidentiality.How researchers will maintain anonymity and/or confidentiality.

 Consent must be derived from the actual people involved, Consent must be derived from the actual people involved,

not just officials or leaders

Informed consent, continued Informed consent, continued

 The best approach to informed consent is to prepare two copies The best approach to informed consent is to prepare two copies

of a document (or letter) which contains the information of a document (or letter) which contains the information described on the previous slide.

described on the previous slide.

 At the time of first contact, enumerators review the letter with At the time of first contact, enumerators review the letter with

respondents The respondent keeps one copy and signs the respondents The respondent keeps one copy and signs the other which the survey team retains.

other which the survey team retains.

 There should be a log kept indicating who has given voluntary There should be a log kept indicating who has given voluntary

consent. consent.

 In the case of school children (or minors), there is an issue as to In the case of school children (or minors), there is an issue as to

Interacting … : The Interview Setting Interacting … : The Interview Setting

 The decision regarding their setting has two components:The decision regarding their setting has two components:

• Where to conduct an interview, and Where to conduct an interview, and

• When When

 The principal decision with respect to interview location is The principal decision with respect to interview location is

the relative merits of a private versus public encounter

the relative merits of a private versus public encounter

 A public encounter may encourage a more general A public encounter may encourage a more general

discussion of the issues; and where information is in the

discussion of the issues; and where information is in the

public domain, onlookers may assist or correct respondents

public domain, onlookers may assist or correct respondents

with particular questions (although there is no guarantee

with particular questions (although there is no guarantee

that they will)

that they will)

 When the research deals with matters that are personally When the research deals with matters that are personally

sensitive, the presence of outsiders or even other family

sensitive, the presence of outsiders or even other family

members may inhibit respondents, embarrassing them into

members may inhibit respondents, embarrassing them into

evasion or silence.

The Interview Setting, cont’d The Interview Setting, cont’d

 Onlookers may even encourage respondents to answer Onlookers may even encourage respondents to answer

untruthfully For example, working during food crises is

untruthfully For example, working during food crises is

problematic (apart from posing moral dilemmas) because

problematic (apart from posing moral dilemmas) because

respondents have an incentive to understate their stocks

respondents have an incentive to understate their stocks

of grain, and their general wealth, in the expectation that

of grain, and their general wealth, in the expectation that

food aid will be brought into the community This

food aid will be brought into the community This

tendency will be exacerbated in a public interview, since

tendency will be exacerbated in a public interview, since

the respondent who admits to being wealthy may face

the respondent who admits to being wealthy may face

demands for help from poorer neighbours.

demands for help from poorer neighbours.

 In general, therefore, the more sensitive the topic, the In general, therefore, the more sensitive the topic, the

stronger the case for conducting the interview in private.

The Interview Setting, cont’d The Interview Setting, cont’d

 However, often there is no choice It may be difficult, for However, often there is no choice It may be difficult, for

example, to interview workers in firms without their

example, to interview workers in firms without their

supervisors, or fellow employees being present In some

supervisors, or fellow employees being present In some

rural areas, it may be impossible to keep neighbours and

rural areas, it may be impossible to keep neighbours and

visitors away

visitors away

 But ways can be found around this problem.But ways can be found around this problem.

• Changed the subject of the interview to avoid sensitive Changed the subject of the interview to avoid sensitive topics while other people were around

topics while other people were around

• Another solution is to catch people alone - on coffee breaks, Another solution is to catch people alone - on coffee breaks, in the fields working, or on their way to market.

The Interview Setting: When to conduct? The Interview Setting: When to conduct?

 Interviews can be a real imposition on respondents' time Interviews can be a real imposition on respondents' time  It is important to meet on days and at times convenient to It is important to meet on days and at times convenient to

respondents

respondents

 One way of ensuring that interviews are not an imposition One way of ensuring that interviews are not an imposition

is to make appointments to see people.

is to make appointments to see people.

• For example, by making initial contact (and obtaining For example, by making initial contact (and obtaining informed consent) and leaving 'calling cards' and/or informed consent) and leaving 'calling cards' and/or

calenders. calenders.

When to conduct, cont’d When to conduct, cont’d

 A related issue is deciding when in the year to conduct the A related issue is deciding when in the year to conduct the

interview

interview

 This is not usually an issue for traders or manufacturing This is not usually an issue for traders or manufacturing

firms (though food traders are likely to be very busy

firms (though food traders are likely to be very busy

around the harvest period) but can be a matter of concern

around the harvest period) but can be a matter of concern

in rural localities.

in rural localities.

 In semi-arid areas, for instance, it may be easier to work In semi-arid areas, for instance, it may be easier to work

mainly during the dry season when people who are

mainly during the dry season when people who are

actively engaged in farming have time to spare

actively engaged in farming have time to spare

 At a practical level, travelling around a community may be At a practical level, travelling around a community may be

debilitating during the heat of the dry season, but can

debilitating during the heat of the dry season, but can

become virtually impossible during the rains

When to conduct, cont’d When to conduct, cont’d

 On the other hand, farmers in semi-arid areas not work On the other hand, farmers in semi-arid areas not work

on their farms for half the year and sit around under trees

on their farms for half the year and sit around under trees

waiting to be interviewed by researchers for the other

waiting to be interviewed by researchers for the other

half! Seasonal out-migration is common in such

half! Seasonal out-migration is common in such

communities, notably among young men, and it would be

communities, notably among young men, and it would be

myopic to defer interviewing until after the harvest if a

myopic to defer interviewing until after the harvest if a

large number of respondents then disappears from the

large number of respondents then disappears from the

village

village

 Interviewing schedules should be built around the Interviewing schedules should be built around the

seasonal (and daily and weekly) pattern of domestic and

seasonal (and daily and weekly) pattern of domestic and

economic activities.

Paying respondents Paying respondents

 In general, researchers not like to pay respondents in cash for In general, researchers not like to pay respondents in cash for

interviews But interviewing does take time, and can cost

interviews But interviewing does take time, and can cost

respondents money in terms of lost income

respondents money in terms of lost income

 When interviews imply a financial sacrifice, some form of When interviews imply a financial sacrifice, some form of

compensation may be appropriate

compensation may be appropriate

 In the context of longitudinal or evaluation surveys, payments are In the context of longitudinal or evaluation surveys, payments are

a means of reducing attrition.

a means of reducing attrition.

 One approach is to pay respondents the equivalent of a daily One approach is to pay respondents the equivalent of a daily

wage, in recognition of the fact that they were giving up a day's

wage, in recognition of the fact that they were giving up a day's

labour

labour

 An alternative to cash payment is to offer small gifts as a tangible An alternative to cash payment is to offer small gifts as a tangible

expression of thanks (for education related interventions, pens,

expression of thanks (for education related interventions, pens,

notebooks or inexpensive calculators are an option)

Paying respondents, cont’d Paying respondents, cont’d

 But compensation might create friction between But compensation might create friction between

respondents as well

respondents as well

• If significant inequalities exist within the community, equal If significant inequalities exist within the community, equal

payments are inappropriate - a sizeable sum to a landless labourer

payments are inappropriate - a sizeable sum to a landless labourer

will be an insulting pittance to a wealthy merchant

will be an insulting pittance to a wealthy merchant

• Paying wealthy respondents more than the poor only exacerbates Paying wealthy respondents more than the poor only exacerbates existing inequalities, whereas offering payment or gifts only to the

existing inequalities, whereas offering payment or gifts only to the

poor appears to favour some people at the expense of others

poor appears to favour some people at the expense of others

• Further, cash compensation can interfere with the data collection Further, cash compensation can interfere with the data collection

exercise, particularly if it is more than nominal, and if it relates

exercise, particularly if it is more than nominal, and if it relates

(directly or indirectly) to the research issues.

Paying respondents, cont’d Paying respondents, cont’d

 Further, as soon as fieldworkers give gifts, pay informants in Further, as soon as fieldworkers give gifts, pay informants in

cash, or lend money, a precedent is set which inevitably leads to cash, or lend money, a precedent is set which inevitably leads to escalating expectations

escalating expectations

 Failing to meet these may compromise both the current Failing to meet these may compromise both the current

fieldwork and subsequent research projects fieldwork and subsequent research projects

 Conversely, if no payment is offered, non respondents may Conversely, if no payment is offered, non respondents may ‑‑

ridicule participants for being 'used' for nothing ridicule participants for being 'used' for nothing

 Others who are excluded from the sampling frame may want to Others who are excluded from the sampling frame may want to

be interviewed (and may resent being excluded, believing that be interviewed (and may resent being excluded, believing that those selected for interviews will benefit somehow), while some those selected for interviews will benefit somehow), while some who are selected may complain that they have been picked on to who are selected may complain that they have been picked on to answer interminable boring questions.

Paying respondents, cont’d Paying respondents, cont’d

 In some cases, it may be more appropriate to offer a In some cases, it may be more appropriate to offer a

single, sizeable contribution to the wider community as a

single, sizeable contribution to the wider community as a

whole

whole

 This, too, can take a number of forms (depending on what This, too, can take a number of forms (depending on what

seems appropriate), such as a social event, contributions

seems appropriate), such as a social event, contributions

to community fund raising, a health fair

to community fund raising, a health fair

 It is important to donate something to which all members It is important to donate something to which all members

of the community will have equal access and to ensure

of the community will have equal access and to ensure

that the gift does not distort research-related outcomes,

that the gift does not distort research-related outcomes,

unless its presentation is deferred until departure.

Paying respondents, cont’d Paying respondents, cont’d

 There is no simple answer to the question of payment to There is no simple answer to the question of payment to

respondents

respondents

 There are strong reasons for providing some form of There are strong reasons for providing some form of

recompense for the time and trouble taken by

recompense for the time and trouble taken by

respondents

respondents

 But there are many dangers in doing so But there are many dangers in doing so

 Perhaps the best advice is to provide some tangible Perhaps the best advice is to provide some tangible

means of appreciation, but so in a manner sensitive to

means of appreciation, but so in a manner sensitive to

the circumstances of the research.

Interacting … : Post-Research Obligations Interacting … : Post-Research Obligations  Interactions with respondents does not end with the Interactions with respondents does not end with the

completion of the last questionnaire Researchers have

completion of the last questionnaire Researchers have

responsibilities to their respondents

responsibilities to their respondents

 A primary obligation is to tell respondents' story A primary obligation is to tell respondents' story

truthfully

truthfully

 An equally important obligation is to maintain the An equally important obligation is to maintain the

anonymity of respondents This extends beyond not

anonymity of respondents This extends beyond not

using names in the context of writing research reports;

using names in the context of writing research reports;

there is a strong case for not entering individual names

there is a strong case for not entering individual names

onto computer files (and relying on identification numbers

Post-Research Obligations, cont’d Post-Research Obligations, cont’d

 Finally, respondents are often interested in the findings of Finally, respondents are often interested in the findings of

your research Some means of disseminating results

your research Some means of disseminating results

should be built into the project design

( (