Using-and-interpreting-analogue-designs

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/8027167 Using and interpreting analogue designs Article  in  Work · February 2005 Source: PubMed CITATIONS READS 518 authors, including: Bryan G Cook University of Virginia 138 PUBLICATIONS   4,522 CITATIONS    SEE PROFILE Some of the authors of this publication are also working on these related projects: evidence based-practice View project Core concepts in special education research View project All content following this page was uploaded by Bryan G Cook on 26 May 2018 The user has requested enhancement of the downloaded file 93 Work 24 (2005) 93–97 IOS Press Speaking of Research Using and interpreting analogue designs Bryan G Cook∗ and Phillip D Rumrill, Jr Kent State University, Department of Educational Foundations and Special Services, Center for Disability Studies, 405 White Hall, P.O Box 5190, Kent, OH 44242-0001, USA Tel.: +1 330 672 0579; Fax: +1 330 672 2512; E-mail: bcook1@kent.edu Abstract Researchers in rehabilitation counseling and disability studies sometimes use analogue research, which involves materials that approximate or describe reality (e.g., written vignettes, videotaped exemplars) rather than investigating phenomena in real-world settings Analogue research often utilizes experimental designs, and it thereby frequently possesses a high degree of internal validity Analogue research allows investigators to exercise tight control over the implementation of the independent or treatment variable and over potentially confounding variables, which enables them to isolate the effects of those treatment variables on selected outcome measures However, the simulated nature of analogue research presents an important threat to external validity As such, the generalizability of analogue research to real-life settings and situations may be problematic These and other issues germane to analogue research in vocational rehabilitation are discussed in this article, illustrated with examples from the contemporary literature Keywords: Research design, analogue research Introduction Analogue research is defined by the use of materials that “mirror” or approximate reality In the field of rehabilitation counseling, analogue studies investigate individuals’ (e.g., rehabilitation counselors) reactions to stimuli such as written, fictitious case files; videotapes of individuals acting the part of people with disabilities in a specific situations/contexts; role-playing particular counseling techniques; or other simulations of reality [1] The simulated nature of analogue research affords advantages (e.g., allows for utilization of experimental research designs, can be used to enhance the internal validity of research) and disadvantages (e.g., reduced external validity) that must be considered by researchers and research consumers when designing, conducting, interpreting, and applying analogue research Analogue research is often used in the context of experimental designs; as such, the approach typically ∗ Corresponding author features the high degree of internal validity associated with experiments (see [4]) In the simplest form of an experimental design, a researcher randomly assigns participants into two groups, then uses one group as a control – to which no special treatment is given – and the other as the experimental or treatment group – which receives the intervention being examined Because of random assignment to groups (which reduces the likelihood that the groups differ in meaningful ways other than the application of the intervention), differences that exist between the groups on outcome measures can be attributed to the intervention Because experimental designs control for many alternative explanations for significant findings, they are said to possess a high degree of internal validity [5] Research using an experimental design may involve real people in real situations (called “in vivo” research), but it may sometimes be more desirable or practical to conduct an investigation using a simulation of reality; in that case, the study would be classified as analogue research Consider the example of a rehabilitation researcher who wishes to determine the effects of consumers’ 1051-9815/05/$17.00  2005 – IOS Press and the authors All rights reserved 94 B.G Cook and P.D Rumrill, Jr / Analogue designs work histories on assessments of employability by state Vocational Rehabilitation counselors Specifically, she is interested in determining whether counselors evaluate employability differently for consumers who have worked in sheltered workshops than for consumers who have participated in integrated community-based employment settings She might prepare case descriptions of fictitious consumers who are identical in terms of age, gender, years of experience in the world of work, level of education, functional ability, and disability type The only descriptive information that would be varied is type of employment history For half of the case reports, the consumer is described as someone who has worked in sheltered workshops for his or her entire work history For the other half of the reports, the person is described as having worked in various integrated community-based settings At random, the counselors participating in the study would receive these two different descriptions of work history (half of the sample would receive the sheltered workshop profile and half would receive the community-based settings profile) They would then evaluate the consumer’s employability, and the researcher would compare the ratings between the two groups in an effort to determine the impact of employment history on employability assessments Because all case reports are identical except in terms of the type of employment history given for the consumer, any differences between the two groups on employability ratings would be attributed to the consumer’s employment history This example is considered analogue research because the treatment to which participants are exposed (different work histories) is fictitious rather than real (i.e., participants not rate the employability of actual people, but rather a written description of a fictitious person) Also, in this example, an experimental design is utilized because a treatment (i.e., exposure to a description of an individual with work experience in a community-based setting) is randomly applied to participants Because of random assignment, any statistically significant differences in mean ratings between the control (those who rated the employability of the individual with work experience in sheltered workshops) and treatment groups can be attributed to the independent variable (i.e., type of work history) rather than to pre-existing differences between the two groups of participants Conducting analogue research rather than an “in vivo” experimental design might be considered when in vivo experimental research is impossible, impractical, and/or unethical for reasons including: the existence/availability of participants with the characteristics of interest is insufficient to conduct the desired research, it is unethical to apply the intervention to participants or to withhold it from the control group, the cost of the intervention is prohibitive, and the intervention is too time-consuming to implement An analogue study might also be used instead of “in vivo” research if the researcher desires greater control over the treatment or potentially confounding variables in order to increase the internal validity of his or her research Analogue research can also be conducted using quasi-experimental designs in which intact groups constitute the control and treatment groups instead of using random assignment However, analogue research that utilizes quasi-experimental designs lacks the high level of internal validity associated with true experiments [1,5] Using the example introduced previously, in a quasi-experimental design, counselors who work in one state office might receive the description of an individual who has worked only in sheltered workshops, and counselors working in another office in a different county might be given the description of the person with experience in integrated work settings Due to the lack of random assignment to groups, it is possible that a difference in mean ratings between the groups is attributable to the different locations in which participants’ offices are housed – rather than to the differences in the consumers’ work experience as described in the fictitious case reports Strengths of analogue research In regard to internal validity, analogue research provides a potential advantage as compared to most “in vivo” research Because of the simulated nature of analogue research, the researcher can manipulate the two (or more) conditions in experimental (or quasiexperimental) designs in such a way that they are exactly the same except as related to the independent or treatment variable For example, in an “in vivo” study, a researcher might introduce actual rehabilitation counseling clients with different work experiences to rehabilitation counselors for employability assessments However, characteristics other than work history might affect the counselors’ ratings of employability (e.g., age, disability type, level of education) Even if the B.G Cook and P.D Rumrill, Jr / Analogue designs researcher was able to match the clients whom counselors rate on several pertinent variables, the client with experiences only in sheltered workshops might, for example, slur his speech, whereas the client with work experiences in integrated community settings does not It is possible, then, that differences in employability ratings would be due to differences in speech rather than work history In analogue studies, the researcher has tight control over every aspect of the stimuli provided to participants and can ensure that the only difference between the stimulus provided to the control and treatment groups lies in the independent variable being investigated (in this case, work history) Analogue research also gives the researcher complete control over how to present the treatment condition or independent variable to participants For example, let us assume that a researcher wishes to investigate the impact of how a client’s disability occurred on rehabilitation counselors’ attitudes toward the client In an in vivo study involving actual clients, counselors may not know or remember the exact cause of disability for each client, or may have an incomplete understanding of the cause of the disability In this case, an analogue study in which a written vignette describes in detail how the disabilities of two fictitious clients occurred might constitute a preferable method for examining the effect of disability attribution on counselors’ attitudes Two different causes of a disability could be described (e.g., one client was involved in an accident beyond his control, the other client’s disability came about due to reckless driving) and the descriptions randomly distributed to participants Assuming that all other characteristics are held constant in the vignettes, any differences in attitudinal ratings between the two groups can be attributed to the independent variable (i.e., cause of disability) Limitations of analogue research An important limitation of analogue research has to with external validity, or the degree to which findings generalize beyond the conditions present in the investigation If a study has weak external validity, the findings may not pertain to participants or in contexts that are not exactly like those in the investigation Because analogue designs, by definition, involve the approximation of reality, it is highly possible that in reallife conditions results may differ Referring back to our example of an analogue study comparing the effects of different client work histories on counselors’ employ- 95 ability ratings, it is possible that the fictitious case files may be written in such a way that strongly emphasizes work experience Perhaps the ratings regarding employability that counselors give to the vignettes overestimate the impact of work-experience due to the way the vignettes are written In reality, counselors’ feelings regarding the employability of a client might depend more on relevant skills, communication, appearance, and other factors than on the type or location of work experience Threats to external validity can be minimized by making the approximation of reality as close as possible In other words, written vignettes, films, or other simulated materials should be as realistic and detailed as possible However, analogue designs can never truly replicate reality; no matter how realistic a film, written description, or simulation is, it will never truly equate with actually experiencing the situation and the intervention Therefore, one can not be sure that the results of analogue research will generalize to real world environments Other strategies for enhancing external validity in analogue research include using large, representative samples of research participants; replicating previous studies with slight variations in content or administration of the stimulus; and following up analogue studies with in vivo experiments Examples of analogue research in disability literature Gouvier et al [3] designed a study that used analogue research and an experimental design to investigate the effect of disability type, job complexity, and public contact on hiring preferences regarding job applicants with disabilities In the study, 295 undergraduate students taking upper division business courses at a large university (who represented potential future employers) read fictitious job descriptions and four fictitious resumes of individuals who had closed head injuries (CHI), chronic mental illness (CMI), developmental disabilities (DD), and back injuries (BI) Participants were asked to rate the desirability of each applicant for two jobs: janitor (low complexity) and phone operator (high complexity) Finally, participants were randomly assigned to one of two groups One group read job descriptions indicating that the applicant would work during the day (high public contact); the other group believed that the available positions were for night jobs (low public contact) On one of the dependent measures utilized by Gouvier et al., participants completed “paired compar- 96 B.G Cook and P.D Rumrill, Jr / Analogue designs isons between all possible pairs of applicants for each of the two jobs” (p 178) In other words, participants selected whom they would hire from the six possible pairings (CHI/CMI, CHI/DD, CHI/BI, CMI/DD, CMI/BI, DD/BI) for each job (low and high complexity) Results indicated a significant main effect for disability type and a significant two-way interaction between disability type and job complexity The applicant with a back injury was more likely to be hired than applicants with the three other disabling conditions as a phone operator Likelihood of being hired for this position did not differ among the applicants with the other three disabilities However, for the janitor position, the applicant with mental illness was significantly less likely to be hired than the other three applicants– for whom the likelihood of being hired did not significantly differ No significant results were found in relation to amount of contact with the public The authors concluded that bias toward applicants with disabilities varies according to different job requirements The authors also noted that the differences between a real job application process and the simulated conditions of the analogue study “may limit the generalizability of the results” (p 180) Broussard et al [2] also reported the results of analogue research utilizing an experimental design Their study investigated the effects of attribution of responsibility for a disability, race of consumer, and race of rehabilitation counselor on counselors’ predictions regarding consumers’ personal adjustment to disability and vocational outcomes One hundred and fifty (150) rehabilitation counselor trainees (juniors, seniors, and graduate students) from two universities (a historically black Southern university and a predominantly white Midwestern university) participated in the study Each participant read a fabricated vocational evaluation report that contained “vocational interest results, academic aptitude scores, recommendations for next steps, work history, level of education, and a brief job description in which the individual was interested” ([2], p 246) These elements of the vocational evaluation report remained constant, whereas the investigators varied attribution and race The client was described as exhibiting either characterological self-blame attribution (i.e., “I’m the kind of person to whom these things always seem to happen I deserve my fate and have nobody to blame but myself it’s just the way it was meant to be” [p 247]) or behavioral self-blame (i.e., “I think in the past I’ve taken too many chances and not considered the consequences of my actions It seemed like this was going to happen to me eventually I regret that I was not more careful and avoided it” [p 247]); and as either Caucasian or African American Thus, four conditions (behavioral attribution – Caucasian consumer, behavioral attribution – African American consumer, characterological attribution – Caucasian consumer, and characterological attribution – African American consumer) were randomly distributed to participants (who themselves varied by race – being either Caucasian or African American) After reading the fabricated vocational evaluation reports, participants completed a two part instrument on which they predicted the client’s personal adjustment to disability and vocational outcomes A significant main effect was observed regarding consumer attribution for prognosis of personal adjustment Participants predicted better personal adjustment for fabricated clients who made behavioral attributions than they did for those making characterological statements Therefore, the authors conclude that, “counseling trainees should be made aware of how their own perceptions of attribution, and the attribution selfbeliefs of the consumers with whom they work, may influence the decisions they and their consumers make and ultimately affect the success of consumers in the rehabilitation process” (p 253) An additional analysis was conducted to determine the effect of sameness or difference in race between trainee and consumer on participants’ predictions Personal adjustment scores were found to be higher when the consumer and trainee were of the same race Conclusion Analogue research has been and will continue to be utilized in rehabilitation when researchers want to enhance the internal validity of their investigations or conduct research in situations where in vivo experimental research is not feasible Analogue studies are of value to the field because they can investigate the effect of an intervention or stimulus with a high level of internal validity However, decreased external validity is a critical concern The simulated nature of analogue research means that research findings may not generalize or apply to real-life situations As such, we recommend that researchers use analogue research as an initial inquiry to be followed by additional studies involving real people in real situations By understanding the strengths and limitations of analogue research, rehabilitation practitioners are able to appreciate the impact that analogue findings can have on field practice, especially as they apply to the clinical judgments that are made concerning clients B.G Cook and P.D Rumrill, Jr / Analogue designs References [1] [2] J.L Bellini and P.D Rumrill, Research in Rehabilitation Counseling: A Guide to Design, Methodology, and Utilization, Springfield, IL: CC Thomas, 1999 S.L Broussard and W Crimando, Effects of consumer attributions of disability and race on rehabilitation trainees’ predictions of rehabilitation outcomes, Rehabilitation Education 16 (2002), 243–254 97 [3] W.D Gouvier, S Systma-Jordan and S Maryville, Patterns of discrimination in hiring job applicants with disabilities: The role of disability type, job complexity, and public contact, Rehabilitation Psychology 48 (2003), 175–181 [4] P.P Heppner, D.M Kivlighan and B.E Wampold, Research design in counseling, (2nd ed.), Belmont, CA: Brooks/Cole/Wadsworth, 1999 [5] A.E Kazdin, Research Design in Clinical Psychology, Needham Heights, MA: Allyn & Bacon, 1998 View publication stats