The study, as its name reveals, has been conducted with the hope to review and evaluate the validity and reliability of the employed questionnaires in ELT survey research at FELTE, ULIS, VNU. With the design of evaluation research, the study aims to present critical comments on what student practitioners have already applied successfully from their research methodology background, and what they need to improve in order for the research to be more successful. Through the reviews of eight questionnaires employed in six survey graduation papers from 2007 to 2012 at FELTE, ULIS, VNU, the limited validity and reliability of those data collection instruments were revealed. The quality criteria of the source, pilot procedure, main parts and the layouts of the questionnaires, as well as the consideration of the questionnaire items should have received more deliberation for the insurance of the quality of questionnaires. From these findings, some practical implications were proposed with the hope to suggest some solutions to the problem of limited validity and reliability of the questionnaires, thus contributing to the value enhancement of the research projects on ELT Methodology at FELTE, ULIS.
ABSTRACT The study, as its name reveals, has been conducted with the hope to review and evaluate the validity and reliability of the employed questionnaires in ELT survey research at FELTE, ULIS, VNU. With the design of evaluation research, the study aims to present critical comments on what student practitioners have already applied successfully from their research methodology background, and what they need to improve in order for the research to be more successful. Through the reviews of eight questionnaires employed in six survey graduation papers from 2007 to 2012 at FELTE, ULIS, VNU, the limited validity and reliability of those data collection instruments were revealed. The quality criteria of the source, pilot procedure, main parts and the layouts of the questionnaires, as well as the consideration of the questionnaire items should have received more deliberation for the insurance of the quality of questionnaires. From these findings, some practical implications were proposed with the hope to suggest some solutions to the problem of limited validity and reliability of the questionnaires, thus contributing to the value enhancement of the research projects on ELT Methodology at FELTE, ULIS. i TABLE OF CONTENTS Page Acknowledgements i Abstract ii Table of Contents iii List of Abbreviations vi List of Tables and Figures vii CHAPTER 1: INTRODUCTION 1.1. Statement of the Problem and Rationale for the Study 1 1.2. Aims of the Study and Research Question 2 1.3. Scope of the Study 2 1.4. Significance of the Study 3 1.5. Organization of the Study 3 CHAPTER 2: LITERATURE REVIEW 2.1. English Language Teaching Research 4 2.1.1. Introduction to English Language Teaching (ELT) 4 2.1.2. Research in ELT 4 2.2. Survey Research in ELT 6 2.2.1. Definitions of Survey Research 6 2.2.2. Values of Survey Research 7 2.2.3. Survey Research Procedure 10 2.2.4. Data Collection Instruments in Survey Research 13 2.3. Questionnaires in Survey Research 13 2.3.1. Definitions of a Questionnaire 13 2.3.2. Types of Data Collected from Questionnaires 14 2.3.3. Advantages and Disadvantages of Using Questionnaires 15 2.4. Questionnaire Construction 17 2.4.1. Questionnaire Construction Procedure 17 2.4.2. Quality Criteria for Constructing a Questionnaire 19 2.4.2.1. Quality Criteria for the Main Parts of the Questionnaire 19 2.4.2.2. Quality Criteria for the Content of the Questionnaire 20 2.4.3. Criteria for a Well-Constructed Questionnaire 21 2.5. Research Gaps in Questionnaire Construction Evaluation at FELTE, ULIS 23 ii 2.6. Summary 23 CHAPTER 3: METHODOLOGY 3.1. Research Context 24 3.2. Research Question 25 3.3. Research Approach: Evaluation Research 25 3.3.1. Evaluation Research 25 3.3.2. Evaluation Research Procedure 26 3.4. Sampling 28 3.4.1. Evaluands 28 3.4.2. Sampling Strategy: Criterion Sampling 28 3.4.3. Sampling Procedure and Results 30 3.4.4. Description of the Subjects of the Study 31 3.5. Data Collection Instrument 34 3.6. Data Collection Procedure 35 3.7. Data Analysis Procedure 35 3.8. Summary 36 CHAPTER 4: RESULTS AND DISCUSSIONS 4.1. Evaluation of the Reliability 37 4.1.1. Analysis of the Sources and the Pilot Procedure 37 4.1.2. Analysis of the Main Parts 39 4.1.3. Analysis of the Layouts 41 4.2. Evaluation of the Validity 43 4.2.1. Analysis of Questionnaire A 43 4.2.2. Analysis of Questionnaire B 47 4.2.3. Analysis of Questionnaires C1 and C2 49 4.2.4. Analysis of Questionnaires D1 and D2 50 4.2.5. Analysis of Questionnaire E 53 4.2.6. Analysis of Questionnaire F 55 4.3. The Extent to which the Student-Researchers Paid Attention to the Quality Criteria of Questionnaire Construction 61 4.4. Summary 63 CHAPTER 5: CONCLUSION 5.1. Summary of the Major Discussions 64 iii 5.2. Pedagogical Implications 65 5.3. Limitations 66 5.4. Suggestions for Further Studies 67 REFERENCES 68 APPENDICES 70 iv LIST OF ABBREVIATIONS ELT: English Language Teaching FELTE: Faculty of English Language Teacher Education ULIS: University of Languages and International Studies VNU: Vietnam National University v LIST OF TABLES AND FIGURES LIST OF FIGURES Figure 1: Broad Categories of Research 5 Figure 2: Stages in the Planning of a Survey 11 LIST OF TABLES Table 2.1: Overview of Research Styles 7 Table 2.2: Steps of Conducting a Survey Research Study 10 Table 2.3: Equivalence between Survey Research Procedures provided by Cohen et al., (2007) and Mertens (2005) 12 Table 2.4: Advantages and Disadvantages of Using Questionnaires 15 Table 2.5: The Process of Constructing a Good Questionnaire 18 Table 2.6: The Main Parts of a Questionnaire 19 Table 2.7: Types of Validity 21 Table 2.8: Criteria to Evaluate the Reliability of a Questionnaire 22 Table 2.9: Criteria to Evaluate the Validity of a Questionnaire 23 Table 3.1: Evaluation Research Procedure 27 Table 3.2: Criteria for Sampling 29 Table 3.3: The Sampling Procedure and Results 30 Table 3.4: Subjects of the Study 31 Table 3.5: Data Collection Instrument: An Evaluation Framework for Questionnaire Construction 34 Table 4.1: Sample Questionnaire Sources and Pilot Procedures 38 Table 4.2: Main Parts of the Sample Questionnaires 40 Table 4.3: Layouts of the Sample Questionnaires 42 Table 4.4: Analysis of Questionnaire A 44 Table 4.5: Analysis of Questionnaire B 48 Table 4.6: Analysis of Questionnaire D1 51 Table 4.7: Analysis of Questionnaire D2 52 Table 4.8: Analysis of Questionnaire E 54 Table 4.9: Analysis of Questionnaire F 56 vi CHAPTER 1: INTRODUCTION 1.1. Statement of the Problem and Rationale for the Study Research is one major activity in the teaching and learning process at Faculty of English Teacher Education (FELTE), University of Languages and International Studies (ULIS), Vietnam National University, Hanoi (VNU). The student research activity is conducted annually, calling for a large number of projects in various fields, especially in English Language Teaching Methodology (ELT). More importantly, research is used as an activity for assessment of senior students, in the form of graduation paper. Annually, dozens of projects are conducted by students, many of whom choose the survey approach and employ questionnaires as their research instrument. In fact, it is evident that the questionnaire is generally considered one of the most popular research methods utilized by the student researchers. Nonetheless, to which extent do the student practitioners pay attention to questionnaire construction is questionable, especially in terms of validity and reliability. The help of computers and the Internet have led to the ease of questionnaire design, as Dornyei (2003, p.3) says: “it is possible to draw up something that looks respectable in a few hours”, but the quality of data is another issue. As Gillham (2000, p.1) states, “good research cannot be built on poorly collected data”, and well constructed questionnaires will eliminate this risk. In other words, “developing a questionnaire that will yield worthwhile data” (Gillham, 2000, p.1) is the prerequisite for valuable findings in research in general and ELT research in particular. Meanwhile, the scarcity of research projects on the issues of questionnaire construction at FELTE, ULIS is recognized. In the University Library, there has been little or even no evidence of the reviewing and evaluating the current situation of questionnaire construction. Those studies, if conducted may serve as a source of reference for the enhancement in the quality of the questionnaires employed in their researches as well as the quality of the students’ projects as a whole. Therefore, the research “Questionnaire Construction in ELT Survey Research at FELTE, ULIS: An evaluation study” was conducted as regards the graduation 1 theses at FELTE, ULIS, VNU with the close investigation of the questionnaire construction in those survey research projects. 1.2. Aims of the Study and Research Question The study was conducted with the hope to develop an Evaluation Framework for questionnaire construction, to analyze the previously employed questionnaires and to evaluate their validity and reliability. Through the investigation into the utilized questionnaires, which are the products of the questionnaire construction process, this project aimed to present critical evaluation on the questionnaire construction of the previous theses, especially in regards to its relevance to the criteria for a well- constructed questionnaire. To be specific, the research project aimed at answering the following question: To what extent do the questionnaires utilized in the ELT survey research projects at FELTE, ULIS satisfy the quality criteria of questionnaire construction? 1.3. Scope of the Study From the aims of the study and the research questions, the scope of the study is determined. The study concerns with only the graduation papers in the field of ELT Methodology at FELTE, ULIS. As the research topic is questionnaire construction, the graduation papers investigated involves attitudinal issues for the most typical features of the questionnaires to be utilized. Of those research projects on attitudinal issues, the projects which employ questionnaires as their main or only data collection instrument are selected. The researcher limits the investigated period of time to only six years from 2007 to 2012. The reasons for this decision is that 2006 was the year when the Course of Research Methodology was first introduced to the third year students at FELTE, ULIS. These students conducted their graduation papers in 2007; consequently, the starting year for investigation is decided to be 2007 to ensure that the student researchers had been some theoretical backgrounds of research methodology. To sum up, the research Questionnaire Construction in ELT Survey Research at FELTE, ULIS: An evaluation study concerns with only the survey graduation 2 papers which employ questionnaires as their main or only data collection instrument from 2007 to 2012. 1.4. Significance of the Study This research project, focusing on questionnaire construction, aims to provide a clearer view of the questionnaire construction quality in ELT research at FELTE, ULIS. The investigation into the situation of questionnaire construction, especially regarding the validity and reliability of the employed questionnaires, is hoped to present critical evaluation on questionnaire construction – what practitioners have already applied successfully from their research methodology background, and what they need to improve in order for the research to be more successful. Furthermore, through the reviews of related studies on questionnaire, as well as the evaluation tool presented in this study, the author hopes to raise the research practitioners’ awareness of the “standardized questionnaire” as well as provide them a suggestion of questionnaire self-evaluation. The study is expected to serve as a useful source of reference for junior research practitioners who want to employ questionnaire as a tool for their survey research, especially in ELT field. 1.5. Organization of the Study Chapter 1 – Introduction - presents an overview of this evaluation research. Chapter 2 is the review of relevant literature, setting the theoretical foundation for data collection and analysis. Chapter 3 describes the design of this study, including the approach, sampling strategy, data collection instrument and procedure, as well as framework for data analysis. Chapter 4 presents results of the research and discussions from the analysis. Finally, chapter 5 – Conclusion – is the summary of the findings as well as the pedagogical implications and limitations of this evaluation study. 3 CHAPTER 2: LITERATURE REVIEW In Chapter 2, the relevant literature of Survey Research on English Language Teaching (ELT) Methodology is reviewed with the emphasis on the construction procedure of questionnaires as the data collection instrument of the survey research. Particularly, the quality criteria of a well-constructed questionnaire is thoroughly investigated and presented in this chapter. 2.1. English Language Teaching Research 2.1.1. Introduction to English Language Teaching English Language Teaching (ELT) is an applied linguistics branch of study of great attention. This branch started to receive much interest from researchers in the late sixteenth century with the arrival of the first textbooks “designed solely to teach English as a foreign language” (Howatt, 1984, p. 6). Through the appearance and development of various theories in English Language Teaching, this field has become a profession worldwide since 1900. Today, ELT has become the topic of interest of thousands of linguists around the world and also a major in numerous university programs, usually under the name of TESOL – Teaching English to Speakers of Other Languages, or Language Teacher Education. 2.1.2. Research in ELT Research in ELT is generally understood as the systematic investigation in ELT. In Brown and Rogers (2002, p. 12), this term is defined as “any systematic and principled inquiry in language learning and teaching”. It means that English language learning and teaching process is the overall objective of ELT research. The importance of research in ELT is emphasized through the article “Language Teacher Education” by Jourdenais (in The Handbook of Language Teaching, edited by Long and Doughty, 2009, pp. 647-58). In this article, Jourdenais highlights the role of research in ELT: 4 [...]... criteria of layouts Furthermore, the evaluation of administration procedure can be rather complicated especially in regards to its quite simple description in the survey projects of this research context Thus, only the source, the pilot procedure and the layouts are used in the establishment of quality criteria for the reliability of a questionnaire in this research In terms of validity, the quality criteria... at FELTE, ULIS and the data collected from those questionnaires are used as the main source to investigate the current attitudes and situations of the Faculty’s language learning and teaching process Thus, evaluation approach is suitable for the author to develop a framework for quality criteria of questionnaire construction and examine their effectiveness in the context of FELTE, ULIS 3.3.2 Evaluation . to suggest some solutions to the problem of limited validity and reliability of the questionnaires, thus contributing to the value enhancement of the research projects on ELT Methodology at FELTE,