Research & Evaluation “The inquiry of truth, which is the love-making, or wooing of it; the knowledge of truth, which is the presence of it; and the belief of truth, which is the enjoying of it; is the sovereign good of human nature.” (Sir Francis Bacon, 1997/1597 , p 6) (from p 430 of book) Purposes: Inquiry of truth Development of new paradigms Without research, knowledge is stagnant Validates what we are doing In a sense, we are all “practitioner—scientists” © 2007 Thomson Brooks/Cole, a division of Thomson Learning Literature Review Examination of prior research Use electronic data bases, such as ▪ ERIC ▪ PsycINFO Statement of the Problem Places research in its historical context Discusses why issue at hand is important Points you in direction of developing research questions, statements, and/or hypotheses See Box 13.1, p 432 for an example © 2007 Thomson Brooks/Cole, a division of Thomson Learning Two types: Quantitative ▪ Assumes objective reality which science can examine ▪ Probability that certain behaviors, values, or bleiefes either cause or are related to other bheaivors values or beliefs Qualitative ▪ Multiple ways of viewing knowledge ▪ Make sense of the world by immersing on in the research situation See Table 13.1, p 435 to describe differences © 2007 Thomson Brooks/Cole, a division of Thomson Learning EXPERIMENTAL Manipulating treatment to show causation Types True Quasi One Shot Case Study and Group Pretest-PostTest Design © 2007 Thomson Brooks/Cole, a division of Thomson Learning NONEXPERIMENTAL Looking at relationships among variables Types Correlation Survey Ex Post Facto (Causal Comparative) True Experimental Research Crème de la crème of research Independent and dependent variables Random assignment Causation ▪ See Figure 13.1, p 435 © 2007 Thomson Brooks/Cole, a division of Thomson Learning Quasi-Experimental Manipulating independent variable Looking for causation No random assignment Use of intact groups Nature of quasi-experimental research leads to less credibility (more threats to internal validity) (see Box 13.2, p 437) See Box 13.3, p 437 © 2007 Thomson Brooks/Cole, a division of Thomson Learning One Shot Case Study and One Group Pretest/Post-Test Design ▪ No random assignment ▪ There is manipulation of independent variable ▪ Example: “ABA” design ▪ Take Baseline measurements then offer treatment and then take baseline measurements again ▪ Do repeatedly ▪ See Figure 13.2, p 438 © 2007 Thomson Brooks/Cole, a division of Thomson Learning Correlational Research Studies relationship between variables using correlation coefficient (positive correlations and negative correlations) Two types: ▪ Bivariate: Studies the relationship between two variables ▪ Simple: Here and now (see Figure 13.3, p 439) ▪ Predictive: Now and the future (e.g.: GRE’s and GPA in college) ▪ Multivariate: The relationship among more than two variables ▪ Multiple regression: (e.g., relationship among empathy, being nonjudgmental, building an alliance, and success in counseling) ▪ Many types: Discriminant analysis, factor analysis, differential analysis, canonical correlational, and path analysis (last two are considered “causal”) © 2007 Thomson Brooks/Cole, a division of Thomson Learning Survey Research Uses questionnaires or interviews Purpose ▪ To gather information from a targeted population ▪ To gain information about the values, behaviors, demographics, and opinions of a population ▪ See Steps: p 441 ▪ Does not tell us underlying reasons why ▪ Not causal ▪ Uses descriptive statistics © 2007 Thomson Brooks/Cole, a division of Thomson Learning 10 Historical Research Purpose: To describe and analyze conditions and events from the past in an effort to answer a research question Relies on systematic collection of information Generally, try to use primary sources (not secondary sources): ▪ Oral histories ▪ Documents ▪ Relics © 2007 Thomson Brooks/Cole, a division of Thomson Learning 17 Descriptive statistics: Measures of central tendency, variability, and relationship Often used with survey research Inferential statistics: Measures whether differences or strengths of the relationships between groups are likely to be the result of chance (e.g., t tests, ANOVA, MANOVA, significance of correlation coefficient, chi square) Generally, experimental and ex post facto research uses inferential statistics while correlational research examines strength of relationships between variables Effect size: Practical significance of one’s findings Example: male counselors more likely than female counselors to try to persuade their clients not to have an abortion, but vast majority of male and female counselors would not this despite a significant difference being found © 2007 Thomson Brooks/Cole, a division of Thomson Learning 18 Relies on inductive analysis: themes and categories emerge from data Often, coding is used: breaks down large amounts of data into smaller parts that hold meaning for the researcher Use multiple sources Find themes See data collection methods and emergence of themes in: Box 13.5, p 449 Box 13.6, p 450 © 2007 Thomson Brooks/Cole, a division of Thomson Learning 19 “The degree to which scientific explanations of phenomena match reality” Two types of validity (Internal and External) Internal: ▪ The degree to which extraneous variables have been accounted for and are not involved in rival hypotheses ▪ Discussed earlier in Box 13.2, p 437 ▪ Quantitative research tends to control for this more than qualitative research © 2007 Thomson Brooks/Cole, a division of Thomson Learning 20 External validity How generalizable are the results? Tight controls of quantitative studies sometimes makes it hard to generalize In qualitative research, external validity is the ability of the researcher to describe the research in ways that will be helpful to other researchers with other populations © 2007 Thomson Brooks/Cole, a division of Thomson Learning 21 Is validity valid in qualitative research? Often, words like “credible and trustworthy” are used instead of validity Based on ability to accurately record information and analyze results Show credibility and trustworthiness by: Prolonged and persistent gathering of info Triangulation: multiple methods Bracketing biases Using an “informant” or “participant observer” Have an “outside auditor” Conduct member checks Other methods? See Box 13.8 © 2007 Thomson Brooks/Cole, a division of Thomson Learning 22 Abstract Review of the Literature Research Hypothesis Methodology Results Discussion, Implications, Conclusions References (APA) © 2007 Thomson Brooks/Cole, a division of Thomson Learning 23 The purpose of evaluation Has program achieved its goals and objectives Has program shown its value Two types of evaluation Formative (Process) Evaluation Summative (Outcome) Evaluation © 2007 Thomson Brooks/Cole, a division of Thomson Learning 24 Formative or Process Evaluation Measures ongoing effectiveness Often Informal Methods: ▪ Ask for verbal feedback ▪ Have participants write down reactions ▪ Complete rating forms © 2007 Thomson Brooks/Cole, a division of Thomson Learning 25 Summative or Outcome Evaluation Used to measure efficacy of a program Used to determine if program should be used in future Used as a method of showing accountability Sometimes experimental research designs are used to assess this type of evaluation See comparison of formative and summative evaluation See Box 13.9, p 455 © 2007 Thomson Brooks/Cole, a division of Thomson Learning 26 See Table 13.2, p 458 © 2007 Thomson Brooks/Cole, a division of Thomson Learning 27 Bias in Research and Evaluation “Every decision a quantitative researcher makes in designing, implementing, analyzing, and interpreting a study is a reflection of his or her personal biases.” Qualitative research allows us to peek inside the world of another culture and understand it from a phenomenological perspective (see Box 13.10, p 457) However, qualitative research certainly is subjective and filled with its own biases © 2007 Thomson Brooks/Cole, a division of Thomson Learning 28 White researchers conducting multicultural research Some appalling research: Tuskegee Experiment To reduce bias of white researchers Chang et al (2010) suggest: ▪ Obtain a skill to explore each cultural perspective ▪ Be aware of their own cultural biases ▪ Do not make cultural differences the major focus, but always be aware of cultural differences ▪ Be considerate of cultural differences of participants © 2007 Thomson Brooks/Cole, a division of Thomson Learning 29 ETHICAL ISSUES Ethical Issue: ACAs Code: Research responsibilities Informed consent Deception Confidentiality © 2007 Thomson Brooks/Cole, a division of Thomson Learning PROFESSIONAL ISSUES ACA and APA both set standards Standards in research ▪ Through statements in their ethical codes ▪ Through curriculum standards set by CACREP 30 Legal Issues Exclusion of females and minorities in research Institutional Review Boards The Counselor in Process Are you willing to risk leaping into the black hole called research? Will you be a conductor, reader, and publisher of research? Will you discover new paradigms? © 2007 Thomson Brooks/Cole, a division of Thomson Learning 31 [...]... experimental and ex post facto research uses inferential statistics while correlational research examines strength of relationships between variables Effect size: Practical significance of one’s findings Example: male counselors more likely than female counselors to try to persuade their clients not to have an abortion, but vast majority of male and female counselors would not do this despite a significant... “credible and trustworthy” are used instead of validity Based on ability to accurately record information and analyze results Show credibility and trustworthiness by: Prolonged and persistent gathering of info Triangulation: multiple methods Bracketing biases Using an “informant” or “participant observer” Have an “outside auditor” Conduct member checks Other methods? See Box 13. 8 ... a division of Thomson Learning 18 Relies on inductive analysis: themes and categories emerge from data Often, coding is used: breaks down large amounts of data into smaller parts that hold meaning for the researcher Use multiple sources Find themes See data collection methods and emergence of themes in: Box 13. 5, p 449 Box 13. 6, p 450 © 2007 Thomson Brooks/Cole, a division of Thomson... The degree to which scientific explanations of phenomena match reality” Two types of validity (Internal and External) Internal: ▪ The degree to which extraneous variables have been accounted for and are not involved in rival hypotheses ▪ Discussed earlier in Box 13. 2, p 437 ▪ Quantitative research tends to control for this more than qualitative research © 2007 Thomson Brooks/Cole, a division of. .. histories ▪ Documents ▪ Relics © 2007 Thomson Brooks/Cole, a division of Thomson Learning 17 Descriptive statistics: Measures of central tendency, variability, and relationship Often used with survey research Inferential statistics: Measures whether differences or strengths of the relationships between groups are likely to be the result of chance (e.g., t tests, ANOVA, MANOVA, significance of. .. includes: Observation (see Box 13. 4, p 446) Ethnographic interviews Collection of Documents and Artifacts © 2007 Thomson Brooks/Cole, a division of Thomson Learning 16 Historical Research Purpose: To describe and analyze conditions and events from the past in an effort to answer a research question Relies on systematic collection of information Generally, try to use primary sources (not secondary... Thomson Brooks/Cole, a division of Thomson Learning PROFESSIONAL ISSUES ACA and APA both set standards Standards in research ▪ Through statements in their ethical codes ▪ Through curriculum standards set by CACREP 30 Legal Issues Exclusion of females and minorities in research Institutional Review Boards The Counselor in Process Are you willing to risk leaping into the black hole called research?... generalizable are the results? Tight controls of quantitative studies sometimes makes it hard to generalize In qualitative research, external validity is the ability of the researcher to describe the research in ways that will be helpful to other researchers with other populations © 2007 Thomson Brooks/Cole, a division of Thomson Learning 21 Is validity valid in qualitative research? Often, words like... Brooks/Cole, a division of Thomson Learning 15 Ethnographic Research [Description (graphy) of human cultures (ethno)] ▪ Margaret Mead was one of the first to do this ▪ Understand events through meanings that people make ▪ Steps: 1 Identify the group to be studied 2 Conduct a literature review 3 Decide on what method to immerse oneself in the culture 4 Make a plan for data collection 5 Plan, usually includes:... designing, implementing, analyzing, and interpreting a study is a reflection of his or her personal biases.” Qualitative research allows us to peek inside the world of another culture and understand it from a phenomenological perspective (see Box 13. 10, p 457) However, qualitative research certainly is subjective and filled with its own biases © 2007 Thomson Brooks/Cole, a division of Thomson Learning 28 ... an alliance, and success in counseling) ▪ Many types: Discriminant analysis, factor analysis, differential analysis, canonical correlational, and path analysis (last two are considered “causal”)... Brooks/Cole, a division of Thomson Learning 16 Historical Research Purpose: To describe and analyze conditions and events from the past in an effort to answer a research question Relies on systematic... “theory” E.g., Want to know about possible PTSD experienced by those who escaped the Twin Towers during 9/11 ▪ Purposeful sample ▪ Meet with these individuals ▪ Questions to give me “rich” and