Using the delphi method as a form of assessment

Using the Delphi method as a form of assessment Delphi Procedure • Complete a survey in which respondents answer a number of open-ended questions • From these responses, create a new survey that lists all the open-ended responses, and the number of respondents that wrote each response • Respondents rate their agreement with each statement on a five-point likert scale, (1) meaning "strongly disagree" and (5) meaning "strongly agree." • Statements with means of 4.0 or higher and standard deviations below 1.0 are designated as areas of agreement • With this information, one can compare like responses over time (pre and post test), conduct factor analyses (given sufficient sample size), and other statistical tests Why Delphi method? • Focus groups can be difficult to manage – Assumption of anonymity could be violated – “Reticent” versus “generous” respondents – Power dynamics Why Delphi method? • Rating consensus based on open-ended responses might be helpful/meaningful • Allows for a natural means of conversation about how results can be used because participants can also be the decision makers • When combined with other findings, can add specific information (e.g., adds meaning to state licensure exams, etc.) • When systematically managed, might include faculty’s review of student work samples, making it more of a “direct” observation of student performance • Might also be a way of getting employers, intern hosts, alumni, and others to participate • Can be managed electronically, without requirement that all participants sit in one room at the same time Case Study: Construction Management Assessment Study • Its program accreditor, the American Council for Construction Education (ACCE), required a submission of a self study; in the self study, faculty had to assess eighteen student learning outcomes • The faculty had only six months in which to conduct alumni, employer, internship studies, in addition to some direct assessment of student work • There was no capstone course or other place in the program where students could be assessed on all accreditation student learning standards • The ACCE was contacted, and they responded to proposal to use a Delphi method affirmatively (“it is not our policy to comment on the assessment method used by programs…” Task One: Selected representative work samples • Faculty were asked to identify which samples of student work they would use to evaluate students on each standard • Used as a source document ACCE Standard Sources of Evidence Program Goal One: The Graduate Will Communicate Effectively Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) MCT213 reports/exams; (2) MCT351 reports/ MCT 306 assignments/ MCT 320 term/ MCT 450 final projects; (3) MCT 320 term papers, MCT 418 presentations, MCT 450 projects, MCT 310 papers, MCT413 article reviews; (4) MCT 320 term papers, 418 presentations, MCT 450 projects, MCT 310 papers; (5) written reports of co-op experiences Program Goal Two: The Graduate Will Be Aware of Important Ethical Considerations in the Construction Industry Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 30 reports/30 exams; (2) no report; (3) MCT 450 projects and exams, MCT 413 exams; (4) MCT 450 projects, MCT 413 exams; (5) exam responses to bids Program Goal Three: The Graduate Will Have Adequate Computer Skills Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 19 reports, 4 exams; (2) 106 assignments, MCT 351 reports, MCT 306 assignments; (3) MCT 450 project and exams; MCT 295 work examples; (4) MCT 450 projects, MCT 295 work examples; (5) MCT450 reports, written reports of co-op experiences Accreditation standards listed Sources of evidence are listed Task Two: Identify Strengths and Weaknesses in Student Performance • Faculty able to comment on each standard were then asked to look at the samples of student work they identified and make a list of strengths and weaknesses • Lists sent to analyst • There was fairly good compliance Task Three: Analyze comments • Statements of strengths and weaknesses listed according to ACCE standard • Some statement combined if sufficiently the same • According to each standard, list is created. Those with like statements (coming from separate faculty) were identified as such; for example, after statement, “Students need to understand how to use accounting to submit bids better” (3 of you made this statement) Task Four: Create survey • Each item on the list was assigned a five-point Liker- scale (5=strongly agree, 1=strongly disagree): Example Program Goal 1: The graduate will communicate effectively 1. Graduating students need improvement in grammar. (3 of you gave this answer) 2. Graduating students need improvement in spelling. (3 of you gave this answer) … [...]... overall ability was 4.27 Satisfactory performance; Construction management graduates are proficient in SoftPlan and in basic computer skills; there are some indications that they are also proficient in SureTrak Senior survey response average for question 1, "communicate effectively" was 3.38 ("agree"); median was also 3.0 Employer (Host) Internship Evaluations Student Performance Report of ACCE Standards... Standards (Delphi Panel) Results of Delphi assessment listed along with findings of other assessments Final Thoughts • The “jury is still out” in respect to the use of the Delphi method as a direct assessment of student learning • Use of this method is a good method of adding specificity to portfolio assessments that identify student performance as “exceeds expectations;” we do not know what that means until...Task Five: Collect and Analyze Information • Information inputted on SPSS and analyzed • Items with higher than 4.0, and standard deviation of less than 1.0 are declared “items of consensus” Task Six: Report So Faculty Can Discuss and Make Recommendations • Results reported to faculty in a way that they can quickly use information • Graphical representations are often the best, where standard is... overall ability was 4.27 Satisfactory performance; Construction management graduates have an awareness of ethical issues in the construction industry Senior survey response average for question 3, "have adequate computer skills" was 3.38 ("agree"); median was also 3.0 Alumni survey response average for question 3, "have adequate computer skills" was 3.64 ("agree"); median was also 4 ("strongly agree")... Co-Op Evaluation and Student Intern Evaluations Alumni Surveys A few interns mentioned that they wanted to continue developing oral and written communication skills Alumni survey response average for question 1, "communicate effectively" was 3.07 ("agree"); median was also 3.0 Interns' overall ability was 4.27 Construction management graduates need improvement in grammar; there are additional indications... Findings of Other Assessments to Discover Themes (Triangulate) • Perhaps more work than usual, but process can be very helpful, if not meaningful and potentially impactful ACCE Standard 1 The graduate will communicate effectively 2 The graduate will be aware if important ethical considerations in the construction industry 3 The graduate will have adequate computer skills Senior Surveys Internship and Co-Op... indications that they also need improvement in personal presentation skills Senior survey response average for question 2, "be aware of important ethical considerations in the construction industry" was 3.38 ("agree"); median was also 3.0 Alumni survey response average for question 2, "be aware of important ethical considerations in the construction industry" was 3.29 ("agree"); median was also 3.0 Interns'... responses to bids Satisfactory performance;Construction management graduates have an awareness of ethical issues in the construction industry No consensus weaknesses Program Goal Three: The Graduate Will Have Adequate Computer Skills Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 19 reports, 4 exams; (2) 106 assignments, MCT... reports, MCT 306 assignments; (3) MCT 450 project and exams; MCT 295 work examples; (4) MCT 450 projects, MCT 295 work examples; (5) MCT450 reports, written reports of co-op experiences Satisfactory performance;Construction management graduates are proficient in SoftPlan and in basic computer skills; there are some indications that they are also proficient in SureTrak No consensus weaknesses Task Seven:... standard is listed next to assessment finding Strengths and weaknesses listed next to standard ACCE Standard Sources of Evidence Strengths Weaknesses Program Goal One: The Graduate Will Communicate Effectively Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) MCT213 reports/exams; (2) MCT351 reports/ MCT 306 assignments/ MCT 320 . industry" was 3.29 ("agree"); median was also 3.0. Interns' overall ability was 4.27 Satisfactory performance; Construction management graduates have an awareness of ethical issues. agree"). Interns' overall ability was 4.27 Satisfactory performance; Construction management graduates are proficient in SoftPlan and in basic computer skills; there are some indications that they. direct assessment of student work • There was no capstone course or other place in the program where students could be assessed on all accreditation student learning standards • The ACCE was contacted,