Database Selection Guidelines for Meta-Analysis in | Applied Linguistics |

Trachtenberg, S (1979) Joke telling as a tool in ESL English Teaching Forum, 21(4), 8–12 Vega, G (1989) Humor competence: The fifth component Unpublished master’s thesis, Purdue University, West Lafayette, IN Database Selection Guidelines for Meta-Analysis in Applied Linguistics YO IN’NAMI Toyohashi University of Technology Aichi, Japan RIE KOIZUMI Tokiwa University Ibaraki, Japan doi: 10.5054/tq.2010.215253 & The massive accumulation of knowledge and studies on certain topics within the academic domain heightens the need for more research syntheses, and this is also true for the field of applied linguistics A statistical approach to integrating studies is known as meta-analysis1; researchers attempt to quantitatively summarize a set of empirical data across studies in order to identify consistencies and explain variabilities through this method (e.g., Cooper & Hedges, 1994; Norris & Ortega, 2006, 2007) Although the concept of meta-analysis is a century old, Glass (1976) and Norris and Ortega (2000) initiated a widespread modern interest in this method in education and applied linguistics research According to Cooper (1982), the meta-analysis process can be classified into five stages: (a) the problem formulation stage, in which research questions that are to be addressed in the meta-analysis are specified and formulated; (b) the data collection stage, in which the literature on relevant studies is researched; (c) the data evaluation stage, in which information that will help answer the research questions is coded; (d) the analysis and interpretation stage, in which the coded information is statistically integrated; and (e) the public presentation stage, in which the findings from the synthesis are disseminated among the audience It must be noted that individual studies provide data included in a meta-analysis, which is in contrast to the traditional There are also nonstatistical approaches to synthesizing research (see discussion in Light & Pillemer, 1984; Norris & Ortega, 2006) THE FORUM 169 approach to conducting a study where individual participants provide data Thus, as emphasized by White (1994), discovering previous studies for synthesis is a major concern for meta-analysts because all the synthesized findings depend on these studies How many studies are necessary for meta-analysis? Although as few as two studies can be synthesized, most researchers argue that, in order to represent the domain of research as thoroughly as possible, all existing published and unpublished studies must be comprehensively covered (e.g., Green & Hall, 1984; Rosenthal, 1984; White, 1994).2 Such a comprehensive approach is particularly useful for examining the filedrawer problem, where studies are less likely to be published when they report statistically nonsignificant results, and for examining the stability of findings across published and unpublished studies through moderator variable analyses (e.g., Lipsey & Wilson, 2001) Because a comprehensive inclusion of previous studies is one of the key aspects in meta-analytic research, researchers track down as many relevant studies as possible through such means as using databases, scanning books and journals manually, and soliciting information from researchers who are familiar with the topic Among these search strategies, the current article focuses on using databases We believe database usage is becoming increasingly more important because databases are developing steadily as technology provides enhanced opportunities to create them, and there are certainly many more available today as compared with, say, ten years ago Nevertheless, we reemphasize the point that a meta-analysis must encompass the entire domain of available research, and that databases provide a useful initial step in the analytic process (but they not assure comprehensiveness in encompassing the entire domain of work) For the meta-analysis process in applied linguistics, the Education Resources Information Center (ERIC) has been widely used (e.g., Russell & Spada, 2006; Saito, 2008), because it provides a rich source of published and unpublished materials However, in the fields of education and psychology, Glass, McGaw, and Smith (1981) showed a slight overlap between ERIC and other databases (i.e., Council for Exceptional Children Abstracts, Dissertation Abstracts, and Psychological Abstracts) This finding suggests that ERIC is not a sufficient resource and that a comprehensive literature search requires the use of multiple databases and must not be based on a single database Accordingly, what databases should be used when conducting a metaanalysis in applied linguistics, and in what manner? In order to address these issues, we examine three research questions: (a) Which databases 170 One can also argue that when the number of collected studies is small, such a domain is not perhaps ready for meta-analysis (see Ellis, 2006) TESOL QUARTERLY are used in meta-analyses in applied linguistics? (b) What combination of databases is used in applied linguistics? (c) Which databases provide a comprehensive coverage of journals in applied linguistics? Reed and Baxter (1994) noted the importance of caution in selecting databases that are relevant to one’s needs However, the existence of numerous types of databases makes it difficult to appropriately select which databases should be used and in what combination Because, to our knowledge, there is no summary of database sources tailored for applied linguistics, we compare databases in this article in order to present practical guidelines for selection of databases for meta-analytic purposes We so by addressing each of the three questions posed earlier Research Question 1: Which Databases Are Used in Meta-Analyses in Applied Linguistics? One method for identifying appropriate, useful, and relevant databases is to examine what databases were used in previous metaanalyses For this purpose, previous meta-analyses in applied linguistics were searched manually in March 2009 Norris and Ortega’s (2006) wellknown book and 24 representative journals were reviewed Norris and Ortega’s edited volume was reviewed because it is a collection of metaanalytic studies in applied linguistics The 24 journals were selected because we believe that they are often read by applied linguists and are likely to include meta-analytic articles in applied linguistics.3 The 24 journals that were reviewed include the following: Annual Review of Applied Linguistics (ARAL), Applied Language Learning (ALL), Applied Linguistics (AL), Applied Psycholinguistics (AP), Assessing Writing (AW), Canadian Modern Language Review (CMLR), ELT Journal (ELTJ), Foreign Language Annals (FLA), International Journal of Applied Linguistics (IJAL), International Review of Applied Linguistics in Language Teaching (IRAL), JALT Journal (JALTJ), Language Assessment Quarterly (LAQ), Language Learning (LL), Language Learning & Technology (LLT), Language Teaching (LTea), Language Teaching Research (LTR), Language Testing (LTes), Modern Language Journal (MLJ), Reading Research Quarterly (RRQ), RELC Journal (RELCJ), Second Language Research (SLR), Studies in Second Language Acquisition (SSLA), System, and TESOL Quarterly (TESOLQ) All issues of each journal were reviewed Fifteen meta-analytic studies were identified (marked with asterisks in the references), of which 12 specified the databases that were used When necessary, every effort was made to contact the authors to request further details about the processes they employed Ideally, all applied linguistics journals that report research must be included in order to conduct a more precise examination of the databases that were used in previous metaanalyses THE FORUM 171 TABLE Year of Publishing and Frequency of Meta-Analyses Year 97 98 99 00 01 02 03 04 05 06 07 08 09 Frequency of published articles 1 1 1 0 As shown in Table 1, 2006 had the highest frequency (40%) of metaanalytic studies, and these studies appeared as book chapters in the previously cited Norris & Ortega (2006) collection For the remaining years, a small but consistent number of meta-analytic studies (one or two) was conducted, thereby suggesting continuing interest in metaanalysis among applied linguists Table summarizes the databases used in the 12 studies ERIC was by far the most widely used database, and it was used in all the studies (N 12) This frequency was followed by Linguistics and Language Behavior Abstracts (LLBA; n 7), PsycINFO (n 7), and ProQuest Dissertations and Theses (ProQuest D&T; n 5) The characteristics of these four most frequently used databases and those of the other databases are briefly described on the basis of their homepage information First, ERIC provides access to more than 1.2 million bibliographic records of journal articles and other educationrelated materials free of charge ERIC’s main strength lies in its indexing of a wide range of education-related materials (e.g., conference papers and proceedings, dissertations, and theses) Full texts are often available in the form of ERIC microfiches at local libraries, and full texts for most TABLE Databases Used in Previous Meta-Analyses Database Academic Search Premier Comprehensive Dissertation Abstracts Current Contents ERIC (Educational Resources Information Center) FirstSearch LLBA (Linguistics and Language Behavior Abstracts) MLA (Modern Language Association) International Bibliography OCLC (Online Computer Library Center) Proceedings First ProQuest Dissertations and Theses PsycARTICLES PsycINFO (including PsycLit, which merged with PsycINFO in 2000) ScienceDirect SSCIa (Social Sciences Citation Index) Frequency 1 12 1 1 Note aAvailable only through the Web of Science platform The total frequencies exceed the total number of studies (12) because the use of multiple databases in a single study was coded separately 172 TESOL QUARTERLY materials from the 1990s onward are available in PDF format Identifying and permitting easy access to these fugitive materials is a great advantage of using ERIC for meta-analysis Second, LLBA, as indicated by its name, is a database that encompasses documents in linguistics and related disciplines LLBA provides more than 370,000 abstracts of journal articles, books, book chapters, dissertations, and conference papers, drawn from more than 1,500 publications Documents from 1973 or later are included; however, full texts are not available Third, PsycINFO provides abstracts of scholarly publications in behavioral and social sciences from the 1800s to the present The database encompasses more than 2,150 journals (comprising 78% of the database), English-language books (11%), and dissertations (11%) Full texts are not available in the database Fourth, ProQuest D&T includes more than 2.3 million dissertations and theses, mainly those published in North America One strength of this database is that it provides comprehensive coverage of documents dating from 1637 onward and relates to a wide range of academic disciplines Further, the full texts of the dissertations and theses dating from 1861 onward are available This extremely long-term coverage of documents across numerous fields and the availability of full texts are truly unique features of ProQuest D&T Dissertations and theses can also be searched for in LLBA; however, LLBA appears limited, for example, in terms of fields included (linguistics and related disciplines only), coverage period (from 1973 to the present day), and unavailability of full texts Thus, a meta-analyst who is planning to collect unpublished studies, particularly dissertations and theses, is strongly advised to access ProQuest D&T Brief descriptions of the characteristics of the remaining nine databases are provided as follows First, Academic Search Premier provides citations and abstracts for more than 8,300 journals encompassing numerous subject areas Full-text access to more than 4,500 of those titles is available Second, the Comprehensive Dissertation Abstracts database is a collection of dissertations and theses that are mainly published in North America, including more than 1.6 million records dating back to 1861 To the best of our knowledge, this database is no longer available, and users are referred to ProQuest D&T for dissertation and thesis searches Third, Current Contents or its electronic version, Current Contents Search, provides access to tables of contents, abstracts, bibliographic information, and abstracts from more than 7,600 scholarly journals and 2,000 books Current Contents Search also includes bibliographic information from certain electronic journals before they are published Fourth, FirstSearch provides access to more than 90 million records, with full-text access to more than 10 million articles Unlike other stand-alone databases, FirstSearch permits searching of documents across databases, such as ArticleFirst for all academic areas, Arts & Humanities Search for THE FORUM 173 humanities, and WilsonSelectPlus for business and management Fifth, MLA International Bibliography indexes more than 66,000 books and articles on literature and linguistics, and it has been compiled by the Modern Language Association of America since 1926 The database archives more than million records and provides full-text links to ProQuest D&T Sixth, Online Computer Library Center (OCLC) ProceedingsFirst contains tables of contents of papers presented at conferences worldwide and received by the British Library Document Supply Centre since 1993 Seventh, PsycARTICLES is a database of fulltext articles that are mainly from journals published by the American Psychological Association It encompasses 66 psychology journals published since 1894, totaling more than 140,000 articles Eighth, ScienceDirect contains abstracts of more than 2,500 journals or 8.5 million articles mainly on science, technology, and medicine However, full-text availability varies depending on subscription status Ninth, although the Social Science Citation Index (SSCI) is infrequently used, we believe it is indispensable, becaue it reveals the history of each article by displaying who has cited it and the number of times it has been referred to Thus, articles that cite relevant studies can be easily traced In addition, SSCI enables searches across disciplines for all articles that have common cited references These are invaluable features of SSCI, because metaanalytic findings may be biased if they only include articles that report clear, impressive results that are more likely to be published and cited (e.g., Cooper, 1989; Hunt, 1997).4 Such a bias can be alleviated by using SSCI to locate and include both popular and less popular articles while conducting a meta-analysis Research Question 2: What Combination of Databases Is Used in Applied Linguistics? Thus far, we have examined which databases are widely used for metaanalyses in applied linguistics In order to reveal combination patterns of the databases used in previous studies, the 15 studies that we examined for Research Question were rearranged according to the number of databases used for each study (see Table 3) It was found that the number of databases used had two distribution peaks around and 5, thereby suggesting that previous meta-analytic studies were often conducted with either one or five databases Although using only one database may not allow a comprehensive search of the existing studies, the researchers in those studies might have chosen to trust one database only rather than to use multiple databases Of particular interest was the 174 It is difficult to identify all research reports and publications in a meta-analysis However, statistical procedures can be employed to tackle the file-drawer problem See Hsu (2002), Orwin (1983), and Rosenthal (1991) TESOL QUARTERLY TABLE Combination of Databases Used in Meta-Analyses Number of databases used Unclear Frequency (ERIC) (Comprehensive Dissertation Abstracts, ERIC) (ERIC, LLBA, PsycINFO; ERIC, LLBA, ProQuest D&T) (ERIC, LLBA, ProQuest D&T, PsycINFO; Academic Search Premier, ERIC, LLBA, PsycINFO) (Current Contents, ERIC, MLA International Bibliography, ProQuest D&T, PsycINFO; ERIC, LLBA, OCLC Proceedings First, ProQuest D&T, PsycINFO; ERIC, LLBA, ProQuest D&T, PsycARTICLES, PsycINFO) (ERIC, FirstSearch, LLBA, PsycINFO, ScienceDirect, SSCI) combination of databases The presence of ERIC, LLBA, ProQuest D&T, and PsycINFO was conspicuous: ERIC and LLBA were used together in seven studies; ERIC, LLBA, and PsycINFO in six; ERIC, LLBA, and ProQuest D&T in four; and all four databases in three Thus, based on the frequency of use (Research Question 1) and combinations (Research Question 2) of databases used for meta-analytic studies, we recommend combined use of ERIC, LLBA, ProQuest D&T, and/or PsycINFO Research Question 3: Which Databases Provide a Comprehensive Coverage of Journals in Applied Linguistics? Although the answer to Research Question revealed combination patterns of the databases used in meta-analyses in applied linguistics, the obvious question at this point is related to uniqueness or redundancy features between these databases In order to answer this research question, we conducted another analysis by focusing on two characteristics of databases—journal coverage rates and periods of coverage across databases A corresponding list of databases and journals was constructed Because such databases abound and our focus is on applied linguistics, the homepages of the 24 representative journals were examined in March 2009 in order to determine which databases included the 24 journals The identified databases were also accessed in order to cross-check whether they, in actuality, included the journals that the journal homepages claimed they included Although we found 98 databases, only 10 of them are presented in Table 4: nine databases that provided the most comprehensive coverage of applied linguistics journals, and one that was frequently employed in applied linguistics meta-analyses (i.e., PsycINFO; see Table 2) Two databases that provide journal information (e.g., advertising rates, submissions, peer-review THE FORUM 175 176 TESOL QUARTERLY 70– 82– 68–07 91–03 04–06 [issue 1] 1980 1980 1996 1944 1946 1967 1991 AL AP AW CMLR ELTJ FLA IJAL 70–80; 82–86; 04– 80–03 1963 1979 2004 IRAL JALTJ LAQ 06– 94–95; 97–98 80–03 94–07 1989 ALL 96– 83; 88–92; 94–03 89; 91–97; 99–03; 07– 87– ERIC 1981 Educational Research Abstracts ARAL Journal published since TABLE Databases and Journal Coverage 07– 98–01 91–94; 98– 97–02 01– 00; 02; 08– 86–89; 93– 96; 98– 84– 97–99; 01– 06; 08– Linguistics Abstracts 48; 55–68; 71–94; 98– 99; 04– 56; 93–08 00– 67–68; 71– 74; 86–89; 95; 98– 91– [100%]f 70; 72; 77– 78; 80–84; 86; 88; 90–06 72 [vol 2] – 63–74; 76– 88; 92; 04–05 78; 80–99; [?%]i 02– 89; 96–08 02 [vol 2] – g [51%] 04 [vol 1] 06– [100%]f 72– [51%]g 90– [51%]g 94; 05–06 [49%]h 72– [51%]g 80–06 80– [100%]f 63–00 67–00 46–95 44–95 80–95 80–95 93–97; 00; 02–06 80– 89–07 [100%]f 80– [100%]f MLAa 80–95 f 80–05 84– [100%] LLBA Periodicals Index Onlineb 04– 97–99; 01– 04– 01– 88 [no 2]– 99– ProQuest Education Journalsc 04– 03– 80– 01– PsycINFO 05– 96–01; 05– 05– 80–82; 96– 94–00; 02– 96– 96– 05; 07– Scopus 97– 97– 97– 97– 97– 7 8 9 SSCI Totald THE FORUM 177 95–07 95–96; 99– 07 95; 97; 00– 07 95; 08 1984 1916 1966 1970 1985 1978 1973 1967 LTes MLJ RRQ RELCJ SLR SSLA System TESOLQ b 94; 00–07 88– 96–98; 00– 81– 06 94–08 75–80; 82–03 82; 94–96; 69– 00–04 13 22 70–75; 87; 88–93; 08– 69– 84–90; 92; 94– 70– 20 97– 94–96; 98– 97– 00–07 97–99; 01 97– 97– f 24 c 73– [100%]f 72– [100%]f 85– [100%]f 85– [100%]f 71–95 [?%]i 72– [49%] h 72– [51%]g 84– [100%] f 97– [100%]f 73– [100%] f 04– [51%]g 73– [100%] LLBA 99– 70–75; 81– 97; 01– 22 69; 75–76; 86–91; 95–01 70–71; 73– 79; 83–97; 03– 85–06 78; 80; 85–05 97; 99–07 [vol 3] 84–86; 93–07 [vol 3] 48– 49 [vol 2]– 94 97– MLAa 14 73–95 67–95 78–95 65–95 16–95 61–90 48–95 Periodicals Index Onlineb 12 01– 97– 86– d 02 [issue 1]– 98– 05– ProQuest Education Journalsc 01– 65– 68– 95– 92– PsycINFO 93– 96–01; 05– 18 04– 05– 96; 98– 01; 05– 96– 05– 05– 97–98; 00– 96– Scopus 12 97– 97– 97– 97– 97– 97– 97– 7 10 10 10 7 SSCI Totald Note MLA International Bibliography Formerly, Periodicals Contents Index Formerly, ProQuest Education Complete The number of databases covering a journal, with mean and standard deviation of 6.92 and 2.02, respectively eThe number of journals covered in a database, with mean and standard deviation of 16.60 and 5.23, respectively f100%, gover 50%, or hless than 50% of the articles published in this journal are included for the specified time period iThe percentage of article coverage is not clear for the specified time period a Totale 98– 05 1997 LTR 97–03 02– 1961 LTea 97–06 1997 97– LLT 70–72; 74– 79; 83– 97– ERIC Linguistics Abstracts 1948 Educational Research Abstracts LL Journal published since TABLE Continued status) rather than journal coverage rates and periods (i.e., MLA Directory of Periodicals; European Reference Index for the Humanities) were excluded ProQuest D&T was often used in applied linguistics (see Table 2) but not included because it only covers dissertations and theses, not journals Table presents journal coverage, and a blank indicates that the journal was not included The total number of journals covered by each database ranged from to 24, with a mean of 16.60 and standard deviation of 5.23 (see Table 4) This finding indicates that the journals listed here are, as a whole, wellcovered across the databases However, only LLBA covered all 24 journals ERIC and MLA International Bibliography (MLA) covered 22 journals, Linguistics Abstracts covered 20, and Scopus covered 18 These five databases conspicuously covered a wider range of journals than other databases Because four of the databases (i.e., ERIC, MLA, Linguistics Abstracts, and Scopus) did not contain certain journals, these databases must be used complementarily in terms of journal coverage For example, ERIC and MLA not include Language Assessment Quarterly and Language Teaching In contrast, Linguistics Abstracts includes both journals but lacks Applied Language Learning, Foreign Language Annals, JALT Journal, and TESOL Quarterly, all of which are indexed in ERIC and MLA Thus, when we select from among the three databases (ERIC, MLA, and Linguistics Abstracts), a combination of either ERIC and Linguistics Abstracts or MLA and Linguistics Abstracts must be used, and extra attention must be paid in conducting manual searches of journals that are missing from these databases Along with journal coverage, also noteworthy are the journal coverage periods It was found that ERIC, Linguistics Abstracts, LLBA, MLA, and Scopus were more comprehensive as compared with other databases For example, the above-mentioned five databases cover Applied Linguistics since 1987 (ERIC), 1986 (Linguistic Abstracts, with occasional breaks), 1980 (LLBA), 1980 (MLA), and 1996 (Scopus) In contrast, ProQuest Education Journals, for example, covers Applied Linguistics only since 1999 and PsycINFO only since 2001 However, there are two caveats First, even the five databases that excel in journal coverage rates and periods (ERIC, Linguistics Abstracts, LLBA, MLA, and Scopus) have numerous gaps in the years that are covered, and careful selection is necessary in order to ensure that all periods that researchers intend to include are indeed included in the search An example of this problem is ERIC, which does not cover certain issues of a journal, although it specifies any breaks in journal coverage on its homepage For example, ERIC has been covering Language Learning since 1970 but did not cover it in 1973 and between 1980 and 1982 178 TESOL QUARTERLY Second, even when all the years in which journals have been published are covered, it does not imply that all articles are searchable within a database For example, MLA does not always contain article abstracts or review articles Further, we checked if MLA includes all articles in certain years and found that it tends to exclude older articles For example, it indexes of the 55 articles in TESOL Quarterly in 1975 but 43 of 72 in 2007 Another caution must be heeded for LLBA, which only includes a certain percentage of articles for each journal According to its homepage, more than 50% of the articles in ‘‘priority’’ journals (e.g., ELT Journal) are covered, whereas less than 50% of the articles in ‘‘selective’’ journals (e.g., Assessing Writing) are included All articles in ‘‘core’’ journals (e.g., Annual Review of Applied Linguistics) are covered, although coverage appears to vary across core journals For example, System is one of its core journals, but we found that certain articles (e.g., Carver, Cousin, & Ahrens, 1980; Cunningsworth & Horner, 1985; Oded & Stavans, 1994) from System were not included in LLBA These ratings (i.e., core, priority, and selective) are determined by the staff at LLBA and are based on whether the journals are relevant to linguistics; however, the actual process by which a particular article in priority or selective journals is prioritized over others and included in the database is not publicly known Without this knowledge regarding possible missing articles in databases, meta-analysts are likely to overlook such sources One solution for this problem is to use the search engine available on the journal homepage For example, all issues of Language Learning dating back to 1948 can be searched on the journal homepage in Wiley InterScience Another solution is to obtain a CD-ROM version of the journal that contains the entire corpus of material in its original form; however, this option appears to be limited to ELT Journal at present Further, it is also worth mentioning that the journal coverage periods vary depending on to what extent an institution purchases back files of articles in a database, at least for SSCI When we accessed SSCI at a local university, we found that it covered materials indexed from 1997 to the present (see Table 4); however, it can potentially cover from 1956 or 1975 onward if more back files are purchased Library homepages not always explicitly identify journal coverage periods in a database, and thus these periods must be routinely checked when conducting literature searches The characteristics of the five databases that appeared for the first time in Table are briefly described as follows First, Educational Research Abstracts contains article abstracts from more than 700 education journals, and links to full-text articles are provided where possible Second, Linguistics Abstracts provides access to 55,000 abstracts from almost 600 linguistics journals published since 1985 (full text not THE FORUM 179 available) Third, Periodicals Index Online indexes millions of article abstracts in the arts, humanities, and social sciences, across more than 300 years up to 1995 (full text not available) Unlike MLA, as discussed earlier, we found that Periodicals Index Online contains all article abstracts and review articles from journals, exactly as claimed on its homepage Fourth, ProQuest Education Journals includes over 785 educational journals indexed with abstracts, and most of them are provided in full text Fifth, Scopus includes millions of journal articles and educationrelated materials (e.g., conference proceedings) dating back as far as 1823 Links to full texts are offered when available Citations can be tracked as in SSCI, which was described earlier We recommend Scopus over SSCI because it has a better rate of journal coverage and periods SUMMARY AND DISCUSSION This article examined the combination of databases that have been used in meta-analytic studies and that provide comprehensive coverage of applied linguistics journals in order to generate database selection guidelines for conducting a meta-analysis in applied linguistics A literature search using all or as many databases in Tables and as possible would be ideal for comprehensive coverage of previous studies However, there are cases where database selection is necessary The results for Research Questions and (Which databases are used in meta-analyses in applied linguistics? and What combination of databases is used in applied linguistics?) indicated that ERIC, LLBA, ProQuest D&T, and PsycINFO and combinations of these four databases were frequently used in previous meta-analyses and are recommended for meta-analysts The analysis conducted for Research Question (Which databases provide a comprehensive coverage of journals in applied linguistics?) revealed that LLBA, ERIC, MLA, Linguistics Abstracts, and Scopus covered journals well and are recommended for meta-analysts LLBA includes all of the 24 journals surveyed in this article, and these 24 journals can also be covered using a combination of the other four databases However, because of differences in journal coverage periods and potentially missing volumes and articles, using only LLBA or any single database is quite likely to lead to overlooking relevant articles TABLE Database Recommended for Meta-Analysis in Applied Linguistics RQ1 & RQ2 RQ3 180 ERIC LLBA x x x x ProQuest D&T PsycINFO x x Linguistics Abstracts MLA Scopus x x x TESOL QUARTERLY Based on a summary of our findings in Table 5, we recommend using as many databases as possible or selecting at least two from the seven databases listed according to one’s research purposes and the database characteristics featured, such as journal coverage In particular, we argue that ERIC and LLBA are essential for meta-analyses in applied linguistics Furthermore, because specific issues of a journal are often not included, use of a search engine available on the journal homepage or a CD-ROM version of the journal is recommended Moreover, even if journal coverage overlaps between databases, using multiple databases is strongly advised, because doing so would preclude any failure to identify a study caused by a meta-analyst’s search fatigue or the different frequencies at which databases are updated Unfortunately, database availability depends primarily on the institution a meta-analyst is affiliated with, so not every meta-analyst has access to all these databases If the database one requires is not available at one’s institution, the best alternative would be to work with those who have access to that database If this, too, is difficult, then there are four other available solutions First, it would be helpful to see if a database developer or vendor provides access to individuals For example, PsycINFO is available in the form of 24-hour access to PsycINFO Direct However, an individual subscription to most databases is not possible; most databases offer only site licenses In such cases, a meta-analyst can ask his or her institution to subscribe to the database at his or her expense Free individual or institutional trials are also available for most databases and would be suitable if it is possible to complete the literature search within the trial period Second, certain university libraries enable library members and visitors to access databases whose license agreements permit use by walk-in users (i.e., persons who are not current students, faculty members, or employees of the university) In such cases, anyone present in the library, regardless of whether or not he or she is affiliated with that institution, is permitted access to the collection Third, certain university libraries permit those who are attending the university for a short time to access databases (e.g., students in a continuing education course, visiting scholars.) However, our experience reveals that information via the second and third solutions is not always available through university library homepages or contacting librarians Therefore, we suggest first conducting Internet searches in order to identify which university library or institution has the database one requires and then going there to try accessing the database This trial-and-error approach is time-consuming, but it works Fourth, certain universities (e.g., University of Toronto) permit alumni to gain access to those databases the university has subscribed to, although the number of accessible databases for alumni is usually much THE FORUM 181 more limited as compared to that for current students and faculty members Through these four procedures, most of the databases discussed in this article should be accessible In fact, the authors of this article originally had access only to ERIC, PsycINFO, and SSCI but were able to gain access to all of the remaining databases listed in Table 4: two databases (Educational Research Abstracts and Linguistics Abstracts) through free individual or institutional trial, four databases (MLA International Bibliography, Periodicals Index Online, ProQuest Education Journals, and Scopus) through a walk-in user status, and one database (LLBA) through enrollment in a continuing education course Although a good metaanalysis must include exhaustive literature search processes not only by using databases but also by hand-reviewing books and journals, as well as contacting experts in the field, the availability of an increasing number of databases because of technological advances indicates the increasingly important role of databases in covering the entire domain of work We hope our database selection recommendations will help to enhance the quality of literature searches for meta-analyses in applied linguistics and other fields ACKNOWLEDGEMENTS We thank Atsushi Mizumoto, Takayuki Nakanishi, and the anonymous reviewers for their valuable comments on the earlier versions of this article THE AUTHORS Yo In’nami is an assistant professor of English at Toyohashi University of Technology, Aichi, Japan Rie Koizumi is an assistant professor of English at Tokiwa University, Ibaraki, Japan Their common interests include test format effects and quantitative research methods REFERENCES Note * denotes a meta-analytic study identified for Research Questions and *Blok, H (1999) Reading to young children in educational settings: A meta-analysis of recent research Language Learning, 49, 343–371 Carver, D., Cousin, W D., & Ahrens, P (1980) Self-directedness and exploratory microteaching in an in-service ELT programme System, 8, 205–210 Cooper, H M (1982) Scientific guidelines for conducting integrative research reviews Review of Educational Research, 52, 291–302 Cooper, H M (1989) Integrating research: A guide for literature reviews (2nd ed.) Newbury Park, CA; Sage Cooper, H., & Hedges, L V (Eds.) (1994) The handbook of research synthesis New York: Russell Sage Foundation 182 TESOL QUARTERLY Cunningsworth, A., & Horner, D (1985) The role of simulations in the development of communication strategies System, 13, 211–218 *Dinsmore, T H (2006) Principles, parameters, and SLA: A retrospective metaanalytic investigation into adult L2 learners’ access to Universal Grammar In J M Norris & L Ortega (Eds.), Synthesizing research on language learning and teaching (pp 53–90) Philadelphia, PA: John Benjamins Ellis, N C (2006) Meta-analysis, human cognition, and language learning In J M Norris & L Ortega (Eds.), Synthesizing research on language learning and teaching (pp 301–322) Philadelphia, PA: John Benjamins Glass, G V (1976) Primary, secondary, and meta-analysis of research Educational Researcher, 5(10), 3–8 Glass, G V., McGaw, B., & Smith, M L (1981) Meta-analysis in social research Beverly Hills, CA: Sage *Goldschneider, J., & DeKeyser, R M (2001) Explaining the ‘‘natural order of L2 morpheme acquisition’’ in English: A meta-analysis of multiple determinants Language Learning, 51, 1–50 Green, B F., & Hall, J A (1984) Quantitative methods for literature reviews Annual Review of Psychology, 35, 37–53 Hsu, L M (2002) Fail-safe Ns for one- versus two-tailed tests lead to different conclusions about publication bias Understanding Statistics, 1, 85–100 Hunt, M (1997) How science takes stock: The story of meta-analysis New York, NY: Russell Sage Foundation *Indefrey, P (2006) A meta-analysis of hemodynamic studies on first and second language processing: Which suggested differences can we trust and what they mean? In M Gullberg & P Indefrey (Eds.), The cognitive neuroscience of second language acquisition (Language Learning, 56, sup 1, pp 279–304) Malden, MA: Blackwell *In’nami, Y., & Koizumi, R (2009) A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats Language Testing, 26, 219–244 *Jeon, E.-H., & Kaya, T (2006) Effects of L2 instruction on interlanguage pragmatic development: A meta-analysis In J M Norris & L Ortega (Eds.), Synthesizing research on language learning and teaching (pp 165–211) Philadelphia, PA: John Benjamins *Keck, C M., Iberri-Shea, G., Tracy-Ventura, N., & Wa-Mbaleka, S (2006) Investigating the empirical link between task-based interaction and acquisition: A meta-analysis In J M Norris & L Ortega (Eds.), Synthesizing research on language learning and teaching (pp 91–131) Philadelphia, PA: John Benjamins *Lee, S.-K., & Huang, H.-T (2008) Visual input enhancement and grammar learning: A meta-analytic review Studies in Second Language Acquisition, 30, 307– 331 Light, R., & Pillemer, D (1984) Summing up: The science of reviewing research Cambridge, MA: Harvard University Press Lipsey, M W., & Wilson, D B (2001) Practical meta-analysis Thousand Oaks, CA: Sage *Masgoret, A.-M., & Gardner, R C (2003) Attitudes, motivation, and second language learning: A meta-analysis of studies conducted by Gardner and associates Language Learning, 53, 123–163 *Norris, J M., & Ortega, L (2000) Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis Language Learning, 50, 417–528 Norris, J M., & Ortega, L (Eds.) (2006) Synthesizing research on language learning and teaching Philadelphia, PA: John Benjamins THE FORUM 183 Norris, J M., & Ortega, L (2007) The future of research synthesis in applied linguistics: Beyond art or science TESOL Quarterly, 41, 805–815 Oded, B., & Stavans, A (1994) The effect of ‘‘false’’ schema activation on the construction of meaning System, 22, 497–507 Orwin, R G (1983) A fail-safe N for effect size in meta-analysis Journal of Educational Statistics, 8, 157–159 Reed, J G., & Baxter, P M (1994) Using reference databases In H Cooper & L V Hedges (Eds.), The handbook of research synthesis (pp 57–70) New York, NY: Russell Sage Foundation Rosenthal, R (1984) Meta-analytic procedures for social research Beverly Hills, CA: Sage Rosenthal, R (1991) Meta-analytic procedures for social research (rev ed.) Newbury Park, CA: Sage *Ross, S (1998) Self-assessment in second language testing: A meta-analysis and analysis of experiential factors Language Testing, 15, 1–20 *Russell, J., & Spada, N (2006) The effectiveness of corrective feedback for the acquisition of L2 grammar In J M Norris & L Ortega (Eds.), Synthesizing research on language learning and teaching (pp 133–164) Philadelphia, PA: John Benjamins *Sahari, M (1997) Elaboration as a text-processing strategy: A meta-analytic review RELC Journal, 28, 15–27 *Saito, H (2008) EFL classroom peer assessment: Training effects on rating and commenting Language Testing, 25, 553–581 *Taylor, A., Stevens, J R., & Asher, J W (2006) The effects of explicit reading strategy training on L2 reading comprehension: A meta-analysis In J M Norris & L Ortega (Eds.), Synthesizing research on language learning and teaching (pp 213– 244) Philadelphia, PA: John Benjamins White, H D (1994) Scientific communication and literature retrieval In H Cooper & L V Hedges (Eds.), The handbook of research synthesis (pp 41–55) New York, NY: Russell Sage Foundation 184 TESOL QUARTERLY