2006-2369: USE OF RUBRICS FOR ASSESSMENT OF A SENIOR PROJECT DESIGN COURSE Ahmed Khan, DeVry University-Addison Ahmed S Khan, Ph.D is a senior Professor in the EET dept at DeVry University, Addison, Illinois He received his M.Sc (applied physics) from University of Karachi, an MSEE from Michigan Technological University, and an MBA from Keller Graduate School of Management He received his Ph.D from Colorado State University His research interests are in the areas of Fiber Optics Communications, faculty development, and outcomes assessment, and, Internet and distance education He is author of “The Telecommunications Fact Book” and co-author of “Technology and Society: Crossroads to the 21st Century” and “Technology and Society: A Bridge to the 21st Century.” He is a member of IEEE, ASEE, ASQ, and LIA Robert Lawrence, DeVry University Professor Robert Lawrence has been teaching writing and speaking at DeVry for over 20 years He also serves as a General Education advisor for the EET/CET senior project class He received a B.A in English from the State University of New York at Stony Brook and an M.A in English from the University of Iowa Frag: Fragments in Context, educational software he wrote and programmed, was published by the Learning Seed He was recently selected by the Chicago Poetry Society to present some of his poetry at Chicago’s annual Around the Coyote festival Page 11.1372.1 © American Society for Engineering Education, 2006 Use of Rubrics for Assessment of a Senior Project Design Course Abstract Rubrics are becoming an essential link between instruction and assessment This paper describes the application of rubrics to gauge the performance, skills, and competencies of students as they complete their senior projects in the EET and CET programs at DeVry University, Addison, IL ABET’s requirement for accredited programs to implement outcomes-based models has stimulated the growth of formalized assessment programs within the engineering and engineering technology communities The use of rubrics as an assessment tool allows faculty to: (a) Improve student performance by collecting data on student skills and competences, and (b) validate that students are achieving course and program objectives The senior project is a two-semester course sequence in which the students synthesize their previous coursework Students are required to plan, design, implement, document, and present the solution to a software/hardware engineering problem Faculty use rubrics for the assessment of project proposal development in the eight semester and for project implementation in form of prototype development and demonstration in the ninth semester Feedback from the rubrics is used to take corrective action to improve the course sequences, program objectives, and instructional delivery I Introduction A rubric is an assessment tool that allows instructors to enhance the quality of direct instruction by providing focus, emphasis and attention to details as a model for a completed product, project or behavior There are two types of rubrics that are used for assessment: Analytic and Holistic An analytic rubric identities and assesses the components of a completed project, and a Holistic rubric assess student work as a whole (see Table 11) Analytic Several High/more difficult More Holistic Few Moderate/less difficult Less Page 11.1372.2 Table Analytic vs Holistic Characteristic Number of elements Inter-rater reliability Scoring time The use of Rubrics as an assessment tool offers a number of advantages: • • • Student learning can be gauged effectively Student’s areas of strengths and concern can be detected Accomplishments of the various tasks of a project can be evaluated effectively II CQI/Assessment Processes at DeVry University Student success is measured by student performance, satisfaction, retention, and placement EET/CET students learn the specialty technical knowledge necessary to enhance or launch their careers, as well as acquiring the general education competencies, skills, and values that help sustain their learning throughout their careers and add meaning to their lives Like other baccalaureate programs at DeVry, the EET and CET programs include an integrative senior project, where students work in teams to solve a real world problem related to their major Students demonstrate a wide range of competencies during the course of the project, making the direct measurement of student academic achievement via the senior project (EET-410L) and a companion general education capstone course, (HUMN-432) a major part of assessment effort, and overall continuous quality improvement (CQI) process at DeVry Student outcomes assessment at DeVry serves as the "check" function in the "Plan-Do-Check-Act" model for CQI (see Figure 1), which emphasizes the iterative and ongoing nature of the process A number of direct and indirect indicators are also established against which the student learning/performance outcomes are assessed Direct indicators of learning • Pre- and post testing • Oral examination/Research presentations • Electronic portfolios • Evaluation of capstone projects (EET-410L & HUMN-432) Indirect indicators of learning • Alumni survey • Employers survey • Graduation rates • Retention rates • Job placement data Page 11.1372.3 Data collection methods: • Assessment tools for EET-410L • • • • • • Essays and writing samples Portfolio collection of student work Surveys Focused group interviews Industrial advisory committee (IAC) input Use of external evaluators (TAC/ABET and NCA) DeVry’s assessment activities have evolved naturally, in parallel with the changes in DeVry's mission and resulting changes in programs The primary components of DeVry’s assessment processes are listed in Table Table Assessment Processes at DeVry University Assessment Tool Frequency of administration Results Action Taken Student Satisfaction Survey Once per Year Performance gaps revealed Institute Review Team/deans reviews data and recommend corrective actions Student Evaluation of Faculty (SIFF) Every Semester for all faculty for all classes Performance gaps revealed Institute Review Team/deans reviews data and recommend corrective actions Writing Assessment Program Every Semester for HUMN-432 class performance gaps revealed Deans review data and recommend corrective actions Industrial Advisory Board Twice a year Board discusses problems and proposes solution Ongoing process for program/course revisions and improvements Senior project Course (EET410L) assessment using national and local assessment tools Every semester Performance gaps revealed Deans / sequence committees review data and take corrective action [See Figure and Figure Page 11.1372.4 Figure 1: Deming's Plan-Do-Check-Act Cycle PLAN DO ACT CHECK Page 11.1372.5 Figure 2: Assessment Processes for EET/CET program First Term Second Third Fourth Last Term Capstone Courses: EET-410L & HUMN-432 Corrective Actions Assessment Page 11.1372.6 Figure Assessment Process for EET Senior Project (EET-410L) EET senior projects for EET-410L are assessed using national assessment tool Assessment data is entered into the national assessment database (DeVry-OBT) DeVry-OBT generates reports based on EET-410L assessment data and sends them to EET Dean for review / corrective actions EET Dean / Course sequence committees analyze assessment data, and recommend and take corrective actions to close the loop for the CQI process Page 11.1372.7 III Use of Rubrics for the Senior Project Course Sequence The senior project is a two-semester course sequence in which the students synthesize knowledge and skills learnt in the previous courses In the first course (EET-400, Project management) students research, plan and develop a project proposal And in the second course (EET-410L, Senior Project Laboratory) students implement the project plan by building and testing a prototype The project involves a solution to a software/hardwarebased engineering problem In both course rubrics are used to evaluate students’ accomplishments of various tasks of project design, planning and implementation phases The senior project course sequence also presents an excellent opportunity to directly measure the competencies (program objectives) of EET/CET graduating students Two Rubrics, a national and a local, are used to evaluate each student on achieving program objectives (competencies) based on direct observation The national assessment rubric is designed to gauge the student performance in achieving the program objectives, and the assessment data is used to take corrective action in terms of curriculum design and implementation The local assessment tool is designed to identify student strengths and weaknesses at course sequence level; the assessment data obtained is used to take corrective action at local level (campus) by revising the course contents and teaching methodologies at the lecture and laboratory levels.(see Rubric E and Rubric F) Table presents a summary of the use of Rubrics for the assessment of EET/CET senior project and program objectives Page 11.1372.8 Table Use of Rubrics for Assessment of EET/CET Senior Project and Program Objectives Assessment for Project Semester/Time Assessment Results Action Phase for completion Tool of Project Phases Project Phase-I Eight/ 1-4 Rubric A Data reveals the Project Faculty Research: Project viability of approve/disapprove Concept/Idea Generation project concept the project idea/concept Project Phase-II Eight /5-14 Rubric B Data reveals the Project Faculty Project Proposal/Plan viability of approve/disapprove Development and Defense project concept the project proposal Project Phase-III Ninth/1-4 Rubric C Data reveals the Faculty advises Project accomplishment students to take Implementation/Prototype of project tasks corrective action in development (Project with respect to terms of Status Check-I) Gantt chart accomplishment of project with respect to Gantt chart Project Phase-IV Ninth/5-10 Rubric C Data reveals the Faculty advises Project accomplishment students to take Implementation/Prototype of project tasks corrective action in development (Project with respect to terms of Status Check-II) Gantt chart, and accomplishment of weaknesses and project with respect limitations of to Gantt chart Prototype Ninth/11-14 Rubric D Project Phase-V Data and Faculty determines Project if student demonstration completion/Demonstration sussessfully determines the of Prototype/Project accomplishment demonstrated the defense in form of a accomplishment of of project written report and an oral objective project presentation implementation Page 11.1372.9 Rubric E Strengths and weaknesses of students’ competencies revealed Rubric F Performance gaps revealed at course sequence level Program director and curriculum Manager Deans / sequence committees review data and take corrective action Conclusion The paper presented an overview of the use of Rubrics in the assessment of course and programs objectives Rubrics are easy to use and allow faculty to directly assess student knowledge-base, skill levels and competencies in senior project course sequence The faculty and deans find the feedback useful for improving the CET/EET curriculum and student performance References Rogers, Gloria (2005) ABET TEI Assessment Workshop 2.0, Baltimore, September 24, 2005 Khan, Ahmed (2002) Culture of Assessment at DeVry University 2002 Connecting Classrooms, Communities & Careers "Reform with Results" 10th International Conference (June 29 - July 2, 2002),Beaver Creek, Colorado Page 11.1372.10 Outcomes Analyzes test results and draws conclusions Ratings (0 - 4) Evidence Weaknesses/Suggestions for Improvement Analyzes test results and their usefulness toward solving the problem Uses the results toward solving the problem Isolates and identify errors and malfunctions If further tests are necessary, proposes and carries out the same Page 11.1372.16 Objective #2: Create, implement high-level, and Assembly-language programs in support of technical activities Outcomes Ratings (0 - 4) Evidence Analyzes the problem logically Performs feasibility studies Determines the output desired Determines the input needed Identifies the processing required Draws a flow diagram Defines the objective of each module Designs the solution Diagrams a structure chart Selects appropriate data structures Develops the logic for each module in an algorithm Implements the solution Chooses language appropriate for the problem Codes the algorithm into an extendable reusable software Creates good documentation Tests and debugs the software Selects test data Tests and debugs each module Tests and debugs the program as a whole Refines the program Weaknesses/Suggestions for Improvement Page 11.1372.17 Objective #3: For EET: Uses the principles of science, mathematics, and engineering technology to design, implement, and evaluate hardware and software solutions to complex technical problems Outcomes Ratings (0 - 4) Evidence Selects and defines a meaningful problem taking safety, ethical, social, economic, and technical constraints into consideration Proposes a problem for investigation Identifies criteria for the proposal (taking safety, ethical, social, economic, and technical constraints into considerations) Applies criteria in defining and specifying the problem Identifies scope of the problem including a problem statement and solution criteria Devises process to solve problem Identifies subtasks and appropriate actions to solve problem Gathers and organizes resources Plans a realistic schedule for process (e.g., design, construction, report, etc.) Organizes tasks around available technology, personnel and financial resources effectively and efficiently Applies appropriate knowledge of scientific, mathematical, and engineering design tools toward the design and analysis of problem solutions Displays a working knowledge of modern engineering design tools (e.g., HDL, Pspice and Matlab) and applies it to design and analyze problem solutions Integrates knowledge of fundamentals in selecting system components using appropriate technology (including manufacturers' catalogs and network databases) Displays the ability to acquire a working knowledge of new design tools Identifies key issues in designing and building a prototype Identifies materials necessary to build the prototype including power source Prioritizes procurement of design components Define steps to building a prototype Builds, tests and troubleshoots prototype Uses good craftsmanship in building prototype Creates software and documentation Tests operation of the prototype (does it meet original design specs?) Weaknesses/Suggestions for Improvement Page 11.1372.18 Outcomes Optimizes prototype with a commitment to quality, timeliness, and continuous improvement Ratings (0 - 4) Evidence Weaknesses/Suggestions for Improvement Identifies and defines fault identifies and implements remedy Tests solution to determine if design specs are met and if needed, refines it Page 11.1372.19 Objective #3: For CET: Uses the principles of science, mathematics, software engineering, and engineering technology to design, implement, and evaluate software solutions to complex technical problems Outcomes Ratings (0 - 4) Evidence Proposes a problem for investigation Identifies criteria for the proposal (taking safety, ethical, social, economic, and technical constraints into considerations) Applies criteria in defining and specifying the problem Identifies scope of the problem including a problem statement and solution criteria Designs and implements appropriate data structures and algorithms Identifies subtasks and appropriate actions to solve problem Gathers and organizes resources Plans a realistic schedule for process (e.g., design, construction, report, etc.) Organizes tasks around available technology, personnel and financial resources effectively and efficiently Prepares a plan of action to implement the system Displays a working knowledge of modern engineering design tools (e.g., HDL, Pspice and Matlab) and applies it to design and analyze problem solutions Integrates knowledge of fundamentals in selecting system components using appropriate technology (including manufacturers' catalogs and network databases) Displays the ability to acquire a working knowledge of new design tools Applies scientific, mathematical, software, and engineering design tools toward the design and analysis of problem solution Identifies materials necessary to build the prototype including power source Prioritizes procurement of design components Define steps to building a prototype Writes and tests readable and maintainable code Uses good craftsmanship in building prototype Creates software and documentation Tests operation of the prototype (does it meet original design specs?) Page 11.1372.20 Identifies a meaningful problem and defines preliminary solution specifications taking safety, ethical, social, economic, technical constraints and user requirements into consideration Weaknesses/Suggestions for Improvement Outcomes Ratings (0 - 4) Optimizes code with a commitment to quality, timeliness, and continuous improvement Evidence Weaknesses/Suggestions for Improvement Identifies and defines fault identifies and implements remedy Tests solution to determine if design specs are met and if needed, refines it Objective #4: Communicates effectively both orally and in writing Outcomes Ratings (0 - 4) Evidence Communicates effectively in writing Defines goals of the report Synthesizes material into appropriate presentation types (e.g., text, pictures, schematics, graphics, etc.) Selects and focuses on data to support the thesis of the report Present ideas and arguments clearly and logically using appropriate balance of text and visual graphic materials (flow charts, pie and bar charts, block diagrams, etc.) Uses English syntax and technical terms appropriately Communicates effectively orally Chooses structure and order of presentation States problem clearly Uses data, information in audio visual formats to support problem solution Displays a professional demeanor, and uses appropriate body language and word choice Effectively responds to questions and comments Weaknesses/Suggestions for Improvement Page 11.1372.21 Objective #5: Works effectively in a team environment Outcomes Ratings (0 - 4) Evidence Exhibits good dialoguing skills Speaks in turn with clarity using articulation of ideas Listens actively Asks appropriate questions Starts, joins, maintains, and terminates conversation well As part of a small group project, when assigned roles, performs roles effectively Supports other team members in effective performance of their assigned roles Works for constructive conflict resolution Strives for a meaningful group consensus Initiates and participates in group maintenance activities Weaknesses/Suggestions for Improvement Objective #6: Uses applied research and problem-solving skills to support learning at DeVry as well as life-long personal and professional development Outcomes Ratings (0 - 4) Evidence Recognizes the need for knowledge (beyond one's own expertise) that are involved Gains appropriate knowledge through independent or group study Uses appropriate sources (e.g., library or network databases) or experts (team members, teachers, industry engineers, etc.) as sources for knowledge Synthesizes acquired knowledge into the solution Uses engineering problemsolving methodology in solving problems Gathers data from various sources including customers to verify the existence of a problem Defines problem Generates alternative solutions Selects the most appropriate solution based on criteria Applies the solution and collects feedback for continuous improvement Page 11.1372.22 Recognizes the need to know information beyond one's own expertise and has the ability to gather and synthesize the necessary information into the solution of a problem Weaknesses/Suggestions for Improvement Objective #7: Evaluates the broader effects of technology by identifying connections between technology and economics, politics, culture, ethical responsibility, social structure, the environment and other areas Ratings (0 - 4) Outcomes Evidence Weaknesses/Suggestions for Improvement Identifies linkages and causal relationships between technology and social, political, economic, cultural, and environmental conditions Works effectively in diverse environments and adapts technical solution to a diverse audience Pursues technical work within guidelines for professional, ethical, and social responsibility Rating Scale: 4= Outstanding: outcome achieved and demonstrated with no errors (evidence of high quality completed work) 3= Good: outcome achieved and demonstrated with no significant errors (evidence of good quality completed work) 2= Fair: outcome achieved and demonstrated with occasional errors (evidence of mediocre quality completed work) 0= Fail (evidence of poor quality incomplete work) Page 11.1372.23 1= Poor: outcome poorly achieved and with significant number of errors (evidence of poor quality completed work) Rubric F Local Assessment Tool For EET/CET Senior Projects Student’s skill and competency level inventory for EET/CET program at DeVry Dupage Course sequence /Assessment area Degree of Relevan ce to project Competency/skill level Theory 1.DC/AC circuits Low High N/A Strong Satisfactory Weak Comments (Strength/weakness/commen ts) Lab Strong Satisfactory Weak Strength: _ _ Weakness: _ _ Comment: _ Digital Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Strength: _ _ Page 11.1372.24 Weakness: _ _ Devices Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Comment: _ Strength: _ _ Weakness: _ _ Microprocessor Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Comment: _ Strength: _ _ Weakness: _ _ Communications Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Page 11.1372.25 Comment: _ Strength: _ _ Weakness: _ _ DSP/Controls Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Comment: _ Strength: _ _ Weakness: _ _ Computer programming (Use of structured software design, debug and test methodology) Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Comment: _ Strength: Page 11.1372.26 Weakness: Physics Low High N/A Strong Satisfactory Weak Strong Satisfactory Weak Comment: _ Strength: _ _ Weakness: _ _ Technical Writing/public speaking Low High N/A Strong Satisfactory Weak Comment: _ Strength: _ _ Weakness: _ _ Comment: _ Page 11.1372.27 10 Math Low High N/A Strong Satisfactory Weak Strength: _ _ Weakness: _ _ Comment: _ General Comments: _ _ _ _ Page 11.1372.28 For CET projects complete the following additional sections: Course sequence /Assessment area 11 Use of structured design, and programming methods Degree of Relevan ce to project Low High N/A Competency/skill level Comments (Strength/weakness/commen ts) Strong Satisfactory Weak Strength: _ _ Strong Satisfactory Weak Weakness: _ _ 12 Application of structured software test methodology Low High N/A Strong Satisfactory Weak Comment: _ Strength: _ _ Weakness: _ _ Page 11.1372.29 Comment: _ 13 Use of Integrated design and debugging environment methodology Low High N/A Strong Satisfactory Weak Strength: _ _ Weakness: _ _ Comment: _ General Comments: _ _ _ _ _ Page 11.1372.30