jang et al layout

Consequential validity: Test score interpretation and use

The interpretation and use of test scores are mutually dependent and reciprocal While an assessment may accurately reflect what it aims to measure, there remains a risk of inappropriate application of its results (Bachman).

Research conducted on the use of IELTS test scores in higher education reveals differing attitudes among students, administrative staff, and academic faculty concerning the IELTS assessment and the perceived adequacy of institutional entry-level cut scores Studies indicate that these variations in perception can impact the overall evaluation of students' English language proficiency and their readiness for academic challenges (Banerjee, 2003; Deakin, 1997; Coleman, Starfield, 2005).

& Hagan, 2003; McDowell & Merrylees, 1998; O’Loughlin, 2008; Rea-Dickins, Kiely & Yu,

Research indicates that admission staff often lack a clear understanding of IELTS test scores and their implications for assessing students' language abilities Consequently, the responsibility for determining admission cut scores frequently falls to administrators who may not fully grasp what the test measures (Coley, 1999).

Coleman et al (2003) reported similar observations about the inadequate knowledge base of IELTS test score users in different institutional contexts including Australia, the

The disparities in understanding IELTS test scores between the UK and China stem from insufficient systematic training for university administrative and academic staff It is crucial for test score users to gain comprehensive insights into interpreting IELTS scores, particularly regarding the knowledge and abilities students possess at various proficiency levels.

Brooks, 2009; McDowall & Merrylees, 1998) There is a dearth of research on IELTS test score interpretations in the Canadian institutional context, with the exception of Golder,

Reeder and Fleming’s (2009) investigation examining appropriate IELTS test scores at entry level

Furthermore, test score users’ underdeveloped knowledge base can have a profound impact on resource allocation to support students (O’Loughlin, 2008) For example,

Ingram and Bayliss (2007) highlight the potential of IELTS for language support placement, advocating for the increased use of sub-scores in decision-making Meanwhile, Hyatt and Brooks (2009) report that 74% of stakeholders in UK universities believe admitted ELL students need post-entry English language support, yet 64% feel that IELTS results lack sufficient diagnostic information for this purpose This study aims to address concerns about consequential validity, focusing on the adequacy and appropriateness of score interpretations and the actions taken based on these scores.

Intersections between contextual validity and cognitive validity

Previous research clearly indicates a need to support test score users’ assessment literacy for meaningful score interpretations and use (Hyatt & Brooks, 2009)

Analyzing the relationship between IELTS band scores and the academic language requirements in real-life learning contexts is essential This systematic domain analysis offers valuable insights into the conceptual and organizational framework of the target domain.

(McNamara & Roever, 2006, p 21), as well as careful cognitive analysis of test items

Analyzing the domain and cognitive processes that influence test performance can enhance our understanding of both cognitive validity and context validity (Weir & Khalifa, 2008) Research indicates that the IELTS test aligns well with the needs of the target language domain For instance, Weir et al (2012) found that the tasks and cognitive processes used in the IELTS reading subsections closely mirror those required in real-life language contexts However, it has been noted that the strongest correlation with the IELTS reading subtest lies in basic, literal comprehension skills.

(Moore, Morton & Price, 2012) In this latter research, critical evaluation of texts and tasks that required reference to multiple sources demonstrated weak comparability

The investigation revealed significant variations in language demands across different academic disciplines, highlighting a potential gap in the IELTS reading subtest Researchers suggested that further studies at other institutions could enhance these findings Bax (2015) proposed that the IELTS reading test should incorporate more items focused on global reading and expeditious reading, as these skills are indicative of successful readers Addressing the relevance and authenticity of tested skills to the target language use domain is crucial for establishing valid arguments.

This study aims to improve the interpretation and application of test scores by integrating cognitive and contextual validity concerns By developing IELTS reading can-do proficiency descriptors that reflect diverse skill profiles across various band score levels, the research seeks to better support students Additionally, it identifies the specific language and literacy requirements essential for success in real-life academic contexts.

Scoring validity: Blending MIRT CDM with scale anchoring for enhanced test score interpretations

anchoring for enhanced test score interpretations

Scoring validity issues primarily revolve around the consistency and impartiality of test results It is crucial for scoring models to effectively represent and interpret diverse knowledge and skills, ideally articulated through can-do descriptors Additionally, cognitive diagnostic modeling plays a significant role in enhancing the assessment process.

(CDM) is one such model that brings together two advanced modeling approaches: multidimensional item response theory (MIRT) and confirmatory latent class modeling

The assessment process categorizes test-takers into specific skill mastery classes, α 2, based on their proficiency in user-defined skills essential for effective test performance (Jang, 2009; Lee & Sawaki, 2009; Rupp, Templin & Henson, 2010).

A test assessing three skills can categorize test-takers into eight distinct skill mastery profiles, each representing a unique combination of mastered and non-mastered skills Alternatively, posterior probabilities of skill mastery may be utilized for a more nuanced understanding of individual skill levels, rather than relying solely on discrete classifications.

12 www.ielts.org IELTS Research Reports Online Series 2019/2

Diagnostic profiles offer detailed insights into students' abilities and areas for improvement The effectiveness of diagnostic information from Cognitive Diagnostic Models (CDM) relies on well-defined specifications of the linguistic knowledge and cognitive skills assessed by test items Additionally, the relevance of these skills to real-life language demands is crucial for accurately interpreting and utilizing test scores, ultimately impacting students' academic success.

To effectively analyze inter-skill relationships, it is essential to develop item-by-skill specifications known as a Q matrix This approach distinguishes between compensatory (additive) and non-compensatory (multiplicative) models In the non-compensatory model, a lack of mastery in one skill cannot be offset by proficiency in another, highlighting the importance of each individual trait dimension.

Skill mastery profiles from CDM offer in-depth insights into the specific skills students have mastered or not mastered However, MIRT models are rarely utilized for item calibrations Test score reporting often follows a unidimensional continuous trait scale, akin to the existing IELTS band scores.

Scale anchoring methods are essential for establishing proficiency descriptors corresponding to each test score band on a unidimensional ability scale This process involves identifying items that respondents answered correctly at each proficiency level, ensuring a clear understanding of performance across different scoring ranges.

(i.e., conditional item p-values) and creating a set of anchor items for each score level

Salient skills required for a correct response are then derived through content experts’ judgment (Beaton & Allen, 1992; Gomez, Noah, Schedl, Wright, & Yolkut, 2007; Sinharay,

This study explores the integration of unidimensional scale anchoring with CDM’s MIRT-based skill mastery profiling and expert judgment to identify key descriptors for various IELTS score ranges We investigate how these can-do proficiency descriptors can enhance the interpretation of test scores for users and assist prospective students in planning their language support programs and resources.

RQ1: What are test score users’ perceptions about test scores used for admission in terms of how these test scores translate into real-life academic tasks?

RQ2: To what extent do academic language and literacy demands differ across programs?

RQ3: To what extent do IELTS test scores predict academic outcomes as measured by students’ self-reported cumulative GPA and competence/importance regarding their academic language and literacy skills??

RQ4: What are the characteristics of IELTS reading skill profiles?

RQ5: What proficiency descriptors characterise IELTS band score levels based on blended CDM profiling with scale anchoring?

RQ6: How do test score users respond to can-do proficiency descriptors across IELTS band scores and to recommendations regarding university disciplinary language and literacy demands?

Method

Overview of research design

The current project aimed to enhance the meaningful interpretation of IELTS reading scores for users, aiding their decision-making regarding admission to undergraduate programs at the university Employing a mixed methods research design, the study incorporated focus groups, a large-scale survey, and a combined psychometric modeling approach that featured cognitive diagnosis modeling (CDM) and scale anchoring.

Specifically, the project sought empirical evidence to answer the following research questions listed on the following page

The project took place over four developmental phases Table 1 provides the overview of the research design and specific data collection and analysis activities completed to date

Table 1: Overview of the research design

Phase Purpose Method & Participants Analysis

Phase 1 To examine international students’ perceptions about their language proficiency and preparedness for academic language demands

To examine university faculty’s perspectives about international students’ language proficiency and their academic performance

• Focus groups (39 students and 16 faculty members)

• Domain analysis of course materials

• Grounded theory approach of focus group data analysis

• Content analysis of course materials

Phase 2 To develop IELTS reading skill mastery profiles based on CDM application

Form 153 (Form A; N = 5222) and Form 173 (Form B;

• CDM model fit comparison based on goodness-of-fit and parsimony

Phase 3 To examine the perceived importance of, and self-rated competence in, different language skills

To examine the extent to which IELTS test scores predict self-reported academic performance and competence in academic language demands

• Confirmatory factor analysis; Latent class modeling; ANOVA

Phase 4 To generate skill proficiency descriptors by integrating CDM-based profiles with scale anchoring

To examine the extent to which descriptive skill profiles can enhance test score users’ score interpretations and facilitate their discussions about student language support

(6 students and 6 faculty members and administrators)

• Thematic analyses of field notes from focus groups

Participants, data collection and analysis

Ten focus groups were conducted to explore how users interpret admissions test scores and the preparedness of international students These discussions aimed to gather insights into users' beliefs and perceptions regarding the language proficiency of international students, the significance of test scores in the admissions process, and the academic performance of these students.

The study involved focus groups of 2 to 10 participants, totaling 55 individuals, including instructors and students from three targeted programs: Engineering, Commerce, and Economics These programs were chosen due to their significant representation of international students at the University of Toronto, particularly in business, management, and public administration (27%) and architecture, engineering, and technologies (19%) All participants in the focus groups had previously presented their work.

IELTS scores for admission to the university.

The study participants were recruited through collaboration with program administrators, who developed a joint recruitment plan A student recruitment letter was crafted by the research team and sent electronically to students, detailing the research project and providing an email for inquiries Interested students reached out to the research team, receiving further information about the project and focus group, along with a link to an electronic consent form The research team then collected participant names and coordinated communication regarding the focus group logistics, including date, time, and location.

To enhance focus group discussions on shared experiences, homogenous groups were established based on position and year of study (Morgan, 2008) This led to the formation of three focus groups per program: first-year students, upper-year students (second year and above), and instructors Due to a strong response from upper-year Commerce students, two focus groups were conducted for this subgroup, totaling ten focus groups overall The majority of participants were from mainland China, with Commerce students exhibiting the greatest diversity, including four students from India, two from Pakistan, and two from other regions.

Russia Additionally, one economics student was from Taiwan Ages ranged from 19 to

24 years, and the group was made up of 29 females and 16 males See Table 2 for more detailed information about focus group participants.

Table 2: Composition of focus group participants

Program First-year student Upper-year student Instructor Total

Focus group protocols were developed for each group type (see Appendices A and B)

The protocol questions were designed to cover a series of major themes of interest

In the initial two months of their university journey, first-year students were not yet ready to engage in discussions about final exams or the transition between academic years and programs Instead, the focus of inquiries centered around topics such as the admissions process, preparation for language tests, early experiences at the university, language requirements, insights on taking the IELTS exam, and interactions with university language support services.

Upper-year students participated in focus groups where they addressed questions about their personal growth, challenges faced, and evolving experiences throughout their academic journey Each session was video recorded and included an interviewer, a note-taker, and a technical support person, lasting between 60 to 90 minutes.

Focus groups often exceeded their scheduled time due to participants' eagerness to discuss topics in depth, creating an informal and candid atmosphere Students openly shared their thoughts with both the interviewer and each other, leading to rich discussions Note-takers were tasked with capturing key observations and initial insights during the interviews Following each focus group session, the interviewer, note-taker, and camera operator would regularly debrief to compare notes and discuss their observations, identifying similarities and differences.

Focus group data were analysed by applying a grounded theory approach (Charmaz &

In a study conducted by Bryant (2008), constant comparison among three distinct groups—first-year students, upper-year students, and instructors—was utilized Initial categories were developed through open coding until saturation was achieved, indicating that no new concepts were identified (Coleman & O’Connor).

2007) The data was then axially coded, connecting themes to larger categories within a hierarchical structure of categories and sub-categories (Strauss & Corbin, 1998)

Researchers consistently compared findings across groups during the study, leading to the selective coding of key themes (Glaser & Strauss, 2017) This process involved reinterpreting and evaluating the data to explore relationships among the three groups, ultimately resulting in the development of an emergent theoretical framework (Coleman &).

To explore the discipline-specific language demands faced by university students, a content analysis was conducted on course materials from three programs Instructors were asked to submit syllabi and readings from their courses, and the initial sample was expanded through purposive sampling to include eight courses per program, with two courses from each academic year.

We created an analytical framework encompassing overall reading requirements, evaluation types and quantities, the volume of reading necessary for the largest evaluation, and various text types and genres This detailed analysis allowed us to extract crucial information that revealed differences both within and among programs The final comparative matrix was developed, highlighting overall reading demands, text types, total evaluation counts, and evaluation modalities.

Phase 2 aimed to identify the essential knowledge and skills necessary for accurately answering IELTS reading test items and to create reading skill mastery profiles using Cognitive Diagnostic Modeling (CDM) Different CDMs were applied to the response data from various IELTS test forms.

153 (Form A; N = 5222) and 173 (Form B; N = 8251) using the “CDM” package in R

(Robitzsch, Kiefer, George, Uenlue & Robitzsch, 2019) The two basic data sources for

CDM include response data and a weight matrix, called a Q matrix, that specifies the relationship between items and user-identified attributes Constructing a defensible

The Q matrix necessitates a thorough understanding of the target construct and its required attributes It specifies the necessity of skill k for correctly answering item i Utilizing the item-by-attribute specification, Cognitive Diagnostic Models (CDM) assess each test-taker's mastery level for various skills based on their responses to items linked to those attributes.

A rigorous Q matrix development procedure was utilised to ensure that the attributes selected were “theoretically compelling, empirically sound, and relevant to test use”

In 2009, Jang initiated the development of reading skill assessments by reviewing existing taxonomies and strategies in the literature A team of seven content experts performed item content analyses on Forms A and B to clarify the connections between specific attributes and individual test items This initial development identified 11 key attributes that encompass the skills and knowledge deemed essential for successfully completing the assessments.

IELTS reading test In the few cases where inter-rater discrepancies emerged, the seven raters collectively reviewed the item, attribute definitions and coding scheme until a unanimous consensus was reached

Accurate classification of attribute mastery profiles necessitates a sufficient number of items for each attribute Content analyses indicated that most items focused on explicit comprehension, inferencing, and summarising, with limited representation of the other eight attributes Consequently, the initial Q matrix included several attributes assessed by two or fewer items, prompting revisions for more reliable classifications (Hartz, Roussos & Stout, 2002; Jang, 2005) Attributes with inadequate item counts were integrated with the most relevant skills, such as merging the linking of textual information to background knowledge with inferencing, and combining understanding text purpose with summarising Additionally, graphic interpretation was excluded from the final Q matrix as it appeared only in Form A.

CDMs differ from each other in terms of the assumptive nature of inter-skill relationships

Results

RQ1: What are test score users’ perceptions about test scores used for admission in terms of how these translate to real-life academic tasks?

test scores used for admission in terms of how these translate to real-life academic tasks?

Through grounded theory analysis of 10 focus groups with international students and faculty members, we established a framework that illustrates the academic language proficiency and experiences of international students This framework contextualizes phenomena by examining both external and internal factors, detailing the dispositions of teachers and students, and highlighting tensions arising from the disparities in their experiences The findings emphasize key themes related to students' challenges in meeting language demands within the university environment.

To comprehend the experiences of international students, it's crucial to consider them within a larger framework In 2014, Canada initiated its International Education Strategy to enhance its position as a leader in attracting international students By 2017, the country saw a significant increase in the international student population, reaching 494,525, which marked an impressive growth of 88%.

During the 2004–05 school year, international students comprised 11% of the overall Canadian university population At the University of Toronto, this figure is notably higher, with 20.5% of undergraduates identified as international students in the 2016–17 academic year The university attracts students from approximately 165 countries and regions, with the largest groups coming from China (63%) and India (4%).

Korea (3%), the United States (2%), and Hong Kong (2%).

Most international undergraduate students come from countries where English is not the primary language, leading to the widespread requirement of language proficiency tests like TOEFL or IELTS for university admission.

In Canada, where English and French are the official languages, international students face the challenge of attending schools in a foreign language while also adapting to an English-dominant environment that impacts all aspects of their lives.

While some students successfully adapt to their new environment and secure jobs that enhance their experiences, others struggle significantly, such as facing difficulties with basic tasks like ordering food in a different language.

I still remember, I took my IELTS test in early 2017 But, during summer holidays

Before starting university, I visited Canada for the first time to tour the campus I struggled with basic tasks, like ordering pizza at PizzaPizza, not even knowing the term "pepperoni." Although I passed the IELTS, I still found myself facing challenges with English communication As a first-year Commerce student, I realized that language barriers can be difficult to overcome.

Figure 1: International students’ academic and language experience

4.1.2 Language proficiency required for university admission

A key concern among students and faculty is whether international students possess the necessary language proficiency for academic success across various disciplines Many faculty members are unaware of the cut-off scores for international student admissions, leading to worries about their academic performance and social interactions Students often lack a clear understanding of the IELTS cut-off score of 6.5, recognizing it only as an admission requirement However, some students reported that language test preparation positively impacted their studies, with Engineering students noting improvements in writing lab reports and a Commerce student finding value in interpreting graphs Despite these benefits, a significant challenge remains: the disparity between general language requirements at entry and the specific demands of their programs.

This issue appears to be an inevitable consequence of the current admission policy, that adopts external standardised general language proficiency tests for admission requirements A first-year Engineering student explained:

For reading questions in IELTS, we know the specific questions we need to answer after reading it and there are some key words we can search in the paragraphs

University readings often lack the structured format found in IELTS materials, making it challenging for students to locate the necessary information.

Some students noted that the IELTS listening lectures do not accurately reflect their university experiences An Economics student from the University of Toronto highlighted that professors often use idiomatic expressions and culturally specific language, making university lectures significantly more challenging than the IELTS listening section.

Even I got satisfied marking on IELTS test, I still feel the knowledge is not enough for here In the lecture, the professor usually use something really familiar with you because you are local people Use some words or some examples, but I’m so “what’s that? what’s that?” So I search google for that maybe a local team, local brand The name of coffee and I think some, the nouns, they are difficult to remember

Another Economics student pointed out that IELTS listening has speakers with different

British accents, which are not as common in Canada

The language…IELTS is like a British language, right? It’s a British accent and the basically listening is British And here we’re in Canada It’s like kind of

North American And I kinda feel like that’s different, like not connected

Two Economics students pointed out that IELTS writing demands are quite dissimilar from university demands, particularly with regards to writing, by stating that “before

An economics student expressed frustration about the length of the IELTS writing section, noting that while they have experience writing lengthy passages of up to 300 words, the current format feels insufficient When asked for clarification, the student indicated that a longer writing section would be beneficial, suggesting it could better assess their writing abilities and provide a more comprehensive evaluation of their skills.

Most students in the focus groups expressed skepticism about the effectiveness of test scores in reflecting their English language abilities, citing significant language-related challenges they faced upon entering university Despite achieving high IELTS scores, they reported difficulties, particularly in speaking and writing A second-year Engineering student shared insights from his first year, highlighting these struggles.

After I came to U of T, personally I found the most difficult part is writing, so even though I got the highest on writing, that’s umm, that’s the part I find the hardest…

During the university, I participated in some presentation competition and I got some awards, so I think I don’t really have a speaking problem, but the writing part, it’s still really tough part

Another Engineering student had a similar experience with her listening and reading abilities She explained:

I excelled in listening and reading, nearly achieving a perfect score, which boosted my confidence However, upon starting the praxis course, I realized that it demands extensive research and document reading While my partner can complete a research task in two hours, I find myself only able to finish half of it, highlighting the challenges I face in this new academic environment.

I read a website or something I cannot know where is the focus, yeah

Many students reported retaking the IELTS to achieve acceptable scores, suggesting that practice may lead to inflated results For instance, several Engineering and Commerce students attempted the IELTS multiple times, with two Engineering students taking the test four times and two Commerce students doing so three times.

Q2: To what extent do academic language and literacy demands differ across programs?

literacy demands differ across programs?

To examine differences in language and literacy demands across the three programs

In Phase 1 of the study involving participants from Commerce, Engineering, and Economics, two primary data sources were utilized: focus group discussions and content analysis of course materials The objective was to analyze variations both within the programs over the years and between them To facilitate student focus group discussions, a brief self-assessment questionnaire was employed to encourage participants to reflect on various reading skills and strategies, as well as to evaluate the relevance and difficulty levels associated with these skills and strategies (Appendix J).

Students reported a diverse array of course materials, including textbooks utilized across all programs, annual reports and financial statements specific to Commerce, as well as videos and articles relevant to Economics.

Commerce students described finding the volume of text overwhelming and challenging

A recent observation among Engineering students revealed significant disparities in their coursework experiences, with one student struggling with a challenging textbook, while two others noted a lack of extensive reading assignments This highlights the variability in academic demands across different courses within the same Engineering program.

Economics students face challenges with discipline-specific vocabulary and writing formats, while Engineering students struggle with math content and assignment instructions, though opinions on the clarity of instructions vary Additionally, several Economics students report difficulties due to insufficient background knowledge, which hinders their ability to understand texts and complete assignments effectively.

Key skills identified as essential include explicit and implicit comprehension, graph interpretation, discipline-specific vocabulary, and the ability to integrate texts from various fields such as financial statements and economic theories Understanding context, particularly in case studies, performing critical evaluations, and recognizing the audience for different readings are also crucial Additionally, summarizing effectively is vital for writing reports and persuading customers Notably, there were variances in perceptions among students, even within the same discipline, highlighting that while some students prioritize implicit comprehension, others may view it as less important Nonetheless, a general consensus on these skills emerged across different fields of study.

Economics students generally consider understanding the purpose of a text to be of minimal importance, with a consensus among them on this viewpoint In contrast, one Commerce student emphasized the significance of recognizing the audience for various readings, while another deemed the purpose of the text to be the least important aspect Additionally, one Commerce student also viewed text organization as having low importance.

Content analysis of course materials from three programs revealed significant differences in reading types and volumes In Commerce, first-year students primarily engage with textbooks, while second-year students begin incorporating journals, and by third year, case studies are introduced, necessitating a shift toward critical and analytical reading skills Economics shows a similar pattern, transitioning from textbook-heavy reading in the first year to a greater emphasis on journals in the later years Conversely, Engineering students predominantly rely on textbooks throughout all four years, supplemented by online manuals, maintaining a focus on expository reading throughout their studies.

While all programs utilized textbook readings extensively, the content of these texts differed considerably A closer examination of these differences highlighted intriguing disciplinary-specific variations among the programs In the field of Commerce, the texts were primarily descriptive, offering detailed information and instructions, and were mainly text-centric.

In addition to text, these readings included some formulas, figures, graphs and tables

Economics texts often feature descriptive and cause-and-effect formats, utilizing fewer words than Commerce texts while incorporating numerous formulas, figures, graphs, tables, and maps Similarly, Engineering texts are predominantly descriptive, focusing on problem-solving and presenting information through flowcharts, formulas, and various visual aids like figures, tables, and diagrams, resulting in a less text-heavy presentation.

Distinct differences regarding how students are evaluated were evident both as students progress through the years within a program, as well as between programs

First-year Commerce students primarily rely on traditional exams and quizzes for their grades, while second-year introduces group assignments to enhance communication skills In the third year, assessments diversify to include both individual and group tasks, promoting analytical thinking and presentation skills By fourth year, some courses shift away from traditional exams to more flexible formats like take-home reflections Similarly, Economics courses follow this pattern, with first-year assessments dominated by exams, evolving in the second year to include diverse tasks such as homework and tutorials, and in the third year to problem sets and quizzes, with group work only appearing in the fourth year.

Similar to Commerce, not all fourth year Economics courses have exams, for example, one course evaluated students largely based on writing assignments and reflections

The focus in Engineering courses is shifting from memorization and multiple-choice assessments to enhancing practical language skills through writing and presentations Throughout the four years of study, evaluations primarily consisted of midterms and exams, often accounting for 100% of the grade, with quizzes, assignments, tutorials, and projects contributing a smaller portion These assignments emphasized hands-on tasks, indicating a blend of knowledge retention and practical application Most coursework in Engineering programs was independent, with only one course in the fourth year incorporating a group project.

Q3: To what extent do IELTS test scores predict academic outcomes as measured by students’ self-reported cumulative GPA and competence/importance regarding their academic language and literacy skills?

academic outcomes as measured by students’ self-reported cumulative GPA and competence/ importance regarding their academic language and literacy skills?

4.3.1 Self-rated language competence and importance

A comparison of self-reported language importance and competence was conducted among students with varying language backgrounds and academic programs, utilizing factor scores derived from CFA models Due to the complexity of the disciplinary literacy factor, which encompasses a range of program-specific skills, its scores were deemed challenging to interpret and were therefore excluded from further analyses.

When assessing students' language backgrounds, relying solely on legal residency status—such as being an international student, permanent resident, or citizen—may not effectively identify those needing language support This is because numerous international students hail from English-speaking nations, while many permanent residents and citizens may have acquired English as an additional language.

Students were classified into three distinct language status groups based on their country of birth and participation in English language assessments during the admissions process The first group includes students born outside of Canada who completed a language test or program, referred to as "Born outside with test."

Students born outside of Canada who did not need to take a test or program, as well as those born in Canada under the same conditions, represent distinct categories within the educational system Additionally, there is a specific group of students born outside Canada who were required to complete a test or program.

The study categorizes international visa students into three main groups: the International group (n = 241, 26%), primarily from non-English speaking countries, excluding those who completed a Canadian high school curriculum abroad; the Born Outside Without Test group (n = 290, 32%), which includes immigrants who graduated from Canadian high schools, those who studied the Canadian curriculum outside Canada, and students from English-speaking countries; and the Domestic group (n = 365, 40%), mainly consisting of individuals born and raised in Canada, along with some second-generation immigrants who learned English as an additional language Additionally, a small subset of Canadian-born students (n = 21, 2%) had to demonstrate their English proficiency through a language test or program.

Students who left Canada at a young age and returned during high school or university were excluded from the analysis in this section for clarity.

A series of one-way ANOVAs were performed to assess group differences in self-reported competence and the perceived importance of various academic language skills The analysis revealed significant effects of language background status on self-reported competence in general reading and productive skills, with F(2, 576) = 7.48, p = 001 and F(2, 576) = 6.95, p = 001, respectively.

The strength of these relationships was relatively small, as indexed by η² of 025 and

Post-hoc Bonferroni tests revealed significant differences in self-rated competence in general and higher-order reading skills between the Born Outside With Test group and the Domestic group, with the former rating their abilities lower However, the effects on competence in higher-order reading skills were not statistically significant, as indicated by F(2, 576) = 2.73, p = 07.

The ANOVA results presented in the lower part of Figure 2 reveal significant differences in the perceived importance of higher-order reading skills among three language background status groups, with an F-value of 7.96 and a p-value of less than 000 The effect size is small, indicated by η² = 025.

A post-hoc Bonferroni test showed that the Domestic group rated the importance of higher-order reading skills significantly higher than the other two groups.

Figure 2: Comparison of self-rated language competence and importance by language group

General reading skills Higher-order reading skills Productive skills

The study examined the variations in self-assessed language skills and their perceived significance among different academic programs The participants, primarily majoring in Engineering, represented a diverse range of disciplines, highlighting the multifaceted nature of language competence across various fields of study.

(n = 278, 30%) and Life Sciences (n = 243, 27%), followed by Commerce and Social

Sciences (n = 122, 13%), and Computer, Physical, and Mathematical Sciences (n = 97,

11%) Students who reported majoring exclusively in the Humanities (n = 6, 0.7%), in

Architecture (n = 4, 0.4%), and in two or more distinctive disciplines (n = 132, 14%) were excluded from the analyses in this section, along with those whose program response was missing (n = 35, 4%).

A series of one-way ANOVAs were conducted to evaluate self-reported competence and the importance of language skills in university settings Results indicate that students in Commerce and Social Science demonstrated the highest levels of competence in all three language skills, followed by those in Life Science and Engineering, while students in Computer Science reported lower competence levels.

Science and Physical and Mathematical Science reported the lowest levels of self-competence among students Notably, general reading skills showed a significant variance across different academic programs, albeit with a small effect size.

F(3, 478) = 3.96, p = 008, η² = 024 Students in Computer Science and Physical and

Mathematical Science indicated significantly lower competence than those in

Students exhibited varied perceptions regarding the importance of different language skills across programs, with notable differences observed in the significance attributed to general reading skills, listening skills, and writing skills This variation highlights the diverse educational priorities and focuses present in different language programs.

F(3, 522) = 8.40, p < 000, η² = 046; higher-order reading skills, F(3, 522) = 10.44, p < 000, η² = 057; and productive skills, F(3, 522) = 2.84, p = 037, η² = 016

Post-hoc tests indicated that Life Science students consider general reading skills significantly more crucial for academic success than their peers in Engineering, Computer Science, and Physical and Mathematical Sciences Additionally, Commerce students placed a higher importance on advanced reading skills in relation to their academic work compared to students in Engineering programs.

Figure 3: Comparison of self-rated language competence and importance by academic program

General reading skills Higher-order reading skills Productive skills

4.3.2 Predictive validity of IELTS test scores

The predictive validity of IELTS test scores was assessed by examining the differences in performance categories based on IELTS reading band scores and average band scores across three key criteria, including academic outcomes such as cumulative grade point average (CGPA).

Q4: What are the characteristics of IELTS reading skill profiles?

CDM was utilised in order to generate multidimensional reading skill profiles based on IELTS reading scores The attributes included in the first Q matrices consisted of:

(1) explicit textual comprehension; (2) making inferences beyond the text;

(3) summarising the text; and (4) processing vocabulary knowledge Explicit textual comprehension was defined as having basic comprehension of information in the text

Effective reading comprehension involves both explicit and inferential processing of information Inferential processing requires readers to utilize background knowledge and text clues to form hypotheses, predict future events, and understand the author's intent This often entails grasping implicit information to determine causal relationships Summarising key ideas at a global level necessitates recognizing the text's organizational structure and distinguishing main ideas from supporting details Additionally, processing vocabulary involves deducing meanings of infrequent words through various knowledge types Understanding these vocabulary terms is crucial for overall comprehension The attributes utilized in this analysis are summarized in Table 5.

Table 5: Process of attribute identification

Initial attributes set (11) Granularity adjustment (7) Final attribute set (4)

Explicit comprehension at the local level

Explicit comprehension at the global level

Summarising main ideas Summarising main ideas Summarising main ideas

Understanding text purpose Understanding text purpose

Inferencing at the local level Inferential reasoning Inferential reasoning

Inferencing at the global level

Linking to background knowledge Linking to background knowledge

Processing vocabulary knowledge Processing vocabulary knowledge

Table 6 illustrates the attribute distributions for Form A and Form B, revealing that a significant majority of items (65.0% for Form A and 57.5% for Form B) necessitated explicit textual comprehension Inferencing was the second most common attribute, accounting for 30.0% and 25.0% of items, while summarising followed closely with 25.0% and 27.5% Additionally, vocabulary knowledge was required for 17.5% of Form A items and 12.5% for Form B Notably, over 50% of the total items for both forms focused on explicit textual comprehension, raising concerns about construct representation at the reading attribute level, a finding echoed by Li (2011), who noted that 50% of MELAB items also required extracting textually explicit information.

According to Jang (2009), retrofitting Cognitive Diagnostic Models (CDM) to existing tests with a specific Q matrix can lead to significant challenges These issues often arise from an unbalanced distribution of items across various attributes and the under-representation of key skills that are essential to the target construct.

Table 6: Item distribution by attributes in the Q matrix

After selecting the GDINA model to estimate skill mastery profiles from IELTS response data, we assessed the diagnostic discrimination power of items at the item level This evaluation utilized three key values: phat (m), phat (nm), and pdiff (m-nm) to gauge the items' effectiveness in distinguishing between different levels of skill mastery.

Phat (m) indicates the probability of correctly answering a question when all necessary attributes are mastered, whereas Phat (nm) signifies the probability of answering correctly without complete mastery of those attributes The difference between these two probabilities is represented by pdiff (m-nm) (Kim, 2015).

Appendix J presents Forms A and B, which show average probabilities of correct answers at 79.4% and 70.5%, respectively, indicating a moderately high mastery of the required attributes In contrast, the average phat (nm) scores were notably lower, at 40.5% for Form A and 37.4% for Form B.

Form A and Form B respectively, indicating that test-takers without mastery of required attributes had an average chance of 40.5% and 37.4% of providing the correct answer

On average, masters outperformed non-masters by an average of 38.8% on Form A and

33.1% on Form B Overall, skill masters were well differentiated from non-skill masters

The model's estimated item statistics closely aligned with the observed item statistics, supported by satisfactory RMSE values However, certain items struggled to distinguish between skill masters and non-masters, notably items 18 (.041) and 30 (.051) on Form A, as well as items 5 (-.006), 8 (-.006), and 30 (.040) on Form B, which exhibited poor pdiff (m-nm) scores These results are further illustrated in Figures 4 and 5, indicating that these items may need further analysis regarding their specified attributes and characteristics.

Figure 4: Item p-values between masters and non-masters for Form A

Figure 5: Item p-values between masters and non-masters for Form B

From the final application of the G-DINA model to the response data, skill mastery probabilities were estimated for all individual test-takers and for all skills As shown in

Table 7, the summarising attribute showed the highest proportion of mastery, with 56.5% and 50.3% of test-takers having likely mastered the given skill for Form A and Form

B respectively The vocabulary skill showed 49% and 34% of test-takers having likely mastered this skill for Form A and Form B, while the explicit comprehension skill showed

35% and 40% mastery respectively Overall, the inferential reasoning skill showed the lowest proportion of mastery among the four skills, with 28% and 31% respectively

The vocabulary skill exhibited varying levels of mastery across different forms, highlighting the need for further examination of Q matrix specifications and the comparability of forms regarding difficulty and discrimination In our final analysis, we excluded the vocabulary skill from the Q matrix while developing skill proficiency descriptors.

G-DINA model to update the skill profiles

Table 7: Proportions of mastery across attributes

Reading attribute Proportions of mastery

In developing individual test-takers’ skill mastery profiles, we applied a cut-off point of

.5 (e.g., Kim, 2015; Lee & Sawaki, 2009b; Li, 2011; Ravand, 2016; Ravand & Robitzsch,

2015; Yi, 2017) As shown in Table 8 and Figure 6, out of 16 total possible mastery classes, 27% of test-takers who took Form A did not master any of the four skills

The next most frequent profiles included 0011 (masters of summarising and vocabulary) and 1111 (masters of all skills), representing 22% and 15% respectively

Form B showed a slightly different distribution of mastery classes from Form A

About 41% of Form B test-takers did not master any skill, whereas 23% mastered all

About 10% showed mastery of summarising Form B’s class distribution is common in

CDM applications typically exhibit two predominant flat classes, 0000 and 1111 However, Form A presents a notable deviation from this trend, highlighting the need for further examination of the varying mastery patterns across different forms.

Table 8: Frequency of skill mastery classes/patterns

Figure 6: Comparison of attribute pattern distribution between Forms A and B

Q5: What proficiency descriptors characterise IELTS band score levels based on blended CDM profiling with scale anchoring?

IELTS band score levels based on blended CDM profiling with scale anchoring?

4.5.1 Step 1: Determining the proficiency levels

The IELTS test features nine band score levels; however, not all levels clearly differentiate between distinct skills and knowledge To assess the appropriate number of proficiency levels, we analyzed the correlation between IELTS reading band scores and CDM-based skill mastery profiles, focusing particularly on the 6.5 score, which is the most frequently used cut-off at the University.

Toronto Figure 7 shows the overall score distribution based on Form A response data

Given discrepancies between the two reading test forms, we used only Form A for proficiency descriptor development

Figure 7: Band score distribution for Form A

The earlier CDM results indicated that the vocabulary attribute did not have a consistent relationship with IELTS band scores, primarily due to the limited number of related items and their insufficient diagnostic power Consequently, during the proficiency descriptor development phase, we revised the Q matrix for Form A by removing the vocabulary attribute We selected the final model (G-DINA) after comparing the fit of five CDMs and updated the skill mastery profiles based on three key attributes Table 9 illustrates the comparison of mastery proportions by reading attribute, excluding the vocabulary attribute, against the results from our previous analysis that included it.

Frequency and proportion of each skill mastery class were also presented in Table 10.

Table 9: Comparisons of proportions of mastery: three- vs four-attribute models

Reading attribute Proportions of mastery

Table 10: Frequency and proportion of skill mastery class patterns (three-attribute model)

(Exp-Inf-Sum) Frequency Percentage

Figures 8–10 illustrate the distribution of model-estimated skill mastery levels for three attributes across various IELTS band scores, with the red horizontal line indicating a mastery status level of 5.

Figure 8: Skill mastery probability estimates for basic comprehension across the IELTS band scores

Figure 9: Skill mastery probability estimates for summarising main idea across IELTS band scores

Figure 10 : Skill mastery probability estimates for inferential reasoning across the IELTS band scores

Students achieving an IELTS band score of 6.5, the minimum requirement for most University programs, exhibit strong skills in explicit comprehension and summarizing main ideas However, their inferential reasoning skills fall slightly above 5, suggesting that even at this cut-off score, there is a notable deficiency in their mastery of this critical comprehension skill.

Table 11: Average skill mastery estimates across the IELTS band scores

Conditional p-values were calculated for each of the IELTS band score levels

To identify anchor items, we utilized two primary criteria: we focused on items with conditional p-values of 65 or 7 after rounding, ensuring they are differentiated from the adjacent lower level by at least 2 Table 12 presents the anchor items, highlighted in shaded cells, alongside their overall p-values and estimates for item difficulty and discrimination parameters derived from the application of the 2PL IRT model.

Table 12: Conditional p-values across the IELTS band scores

Item IELTS band score levels Overall p -value

We analyzed the diagnostic discrimination index values at both the attribute and item levels as estimated from the Cognitive Diagnosis Model (CDM) According to Table 13, a higher discrimination index value for an item-attribute pair indicates greater informativeness.

Table 13: Diagnostic discrimination index at the item and attribute level

4.5.4 Step 4: Creating proficiency descriptors from the anchor item pool

When establishing proficiency descriptors, we focused on key band score levels, particularly noting the cut-off score of 6.5 required for university admission Additionally, we observed that there were minimal anchor items below 5 and above 7.5 Consequently, the finalized band score levels were set to include scores of 5.5 and higher.

Table 14 shows a set of proficiency descriptors for each of these selected IELTS band score levels

Table 14: Proficiency descriptors for IELTS reading band score levels 5.5 and above

Band score At each IELTS reading band score level, students in general can:

5.5+ • locate a keyword or a topic sentence by scanning and skimming a text

• comprehend the literal meaning of short phrases or simple sentences

• figure out the meaning of high-frequency vocabulary

6.0+ • understand the main idea from a paragraph

• distinguish the main idea from supporting details

• figure out the meaning of moderately difficult vocabulary

6.5+ • comprehend implicit meaning in a text

• summarise the main idea from a long, grammatically complex text

• figure out the meaning of low-frequency vocabulary

• start to infer implicit meaning from text

7.0+ • synthesise the main idea with supporting details from the text

• make inferences about implicit information from the text

• understand logical connections between ideas across sentences

7.5+ • infer meaning in the text that is specific to a certain culture

• figure out colloquial expressions in the text

• comprehend the text with abstract vocabulary and grammatically complex sentence structures (e.g., if-then, although-)

As noted above, we paid close attention to the observation that students at 6.5 demonstrate insufficient mastery of inferential reasoning This is noted in bolded font above

4.5.5 Recommendations for preparing students for discipline-specific academic language and literacy demands

A major finding from our project revealed a notable gap between students' perceptions of their readiness for academic success, as indicated by their IELTS scores, and their actual preparedness In response, we formulated a series of recommendations aimed at both test-takers and those utilizing test scores (see Table 15) These guidelines can assist university admissions offices and international student affairs units in delivering more thorough information to incoming students about the expectations they will face before commencing their programs.

Table 15: Recommendations for incoming students

Many undergraduate students find that their academic pursuits necessitate more advanced skills and knowledge than anticipated, with varying requirements across different fields such as engineering and economics Notably, traditional language tests like IELTS or TOEFL may not adequately evaluate these specific skills To excel in your academic journey, it is essential to cultivate competencies in the key demands identified by international students at the University of Toronto.

1 Be prepared that the length and number of texts/materials that students are assigned to read in courses are much longer than typical passages used in IELTS or TOEFL

Most courses require sustained stamina and attention while reading long texts or multiple sources.

2 Basic comprehension of main ideas is insufficient for successful academic work

Most coursework requires students to critically appraise or compare different viewpoints presented in a long text or across multiple texts.

Technical and abstract vocabulary presents a significant challenge for international students, as these terms often have varied meanings across different fields of study This complexity can hinder their ability to effectively read academic materials and engage in discussions, whether in speech or writing.

4 Challenges with reading academic materials are not just due to complex grammar or vocabulary Texts convey meanings and views that are specific to certain cultures

When international students come from a different cultural background, they often struggle due to a lack of cultural or background knowledge.

When studying English, many reading passages tend to be general in nature However, academic reading materials vary significantly across different disciplines, often necessitating the interpretation of statistical tables, figures, mathematical formulas, or computational codes.

They vary widely depending on the intended audience and publication type (e.g text book vs business journal vs technical manual).

6 International students often have difficulty understanding collocations (e.g., land a deal, make progress) and idiomatic expressions (e.g., under the weather, a piece of cake)

Although these are not frequently used in academic reading materials, they are often used in oral conversations and in lectures.

Language is integral to academic success, as it is rarely used in isolation Engaging with various reading materials, attending lectures, participating in discussions, and crafting reports or essays are essential components of academic work Often, course assignments necessitate the integration of multiple language tasks, extending beyond simple reading or listening activities.

Q6: How do test score users respond to can-do proficiency descriptors across IELTS band scores and to recommendations regarding university disciplinary language and literacy demands?

proficiency descriptors across IELTS band scores and to recommendations regarding university disciplinary language and literacy demands?

Students often perceive their IELTS report and university acceptance letters as indicators of their readiness for university-level English However, many feel significantly unprepared upon arrival, realizing that the communications they received did not adequately convey the actual reading demands of their programs This disconnect highlights a lack of understanding regarding the specific academic challenges they would face in their studies.

Students reported that the course readings they faced were markedly different from those encountered during admissions assessments, highlighting challenges related to difficulty, volume, length, stamina, specialized vocabulary, and text structure They expressed a need for strategies to effectively navigate complex, discipline-specific texts, such as business journals and scientific articles, particularly in identifying main points and interpreting results.

Students expressed a desire for clearer communication about language mastery and university expectations, alongside targeted advice for developing essential skills They sought actionable steps, text samples, and real-world examples to better understand these skills in practice Feedback specifically focused on proficiency descriptors and recommendations.

• some terms are too general

• simplify some wording to be less academic

• clarity required around what cultural references/culturally-specific language means

• clarity required for some statements, perhaps provide examples/context, e.g “evaluating consequences to infer meaning”

• make skills actionable, provide context or example of specific tasks to connect descriptors to real-world requirements

• provide resources that can be used to improve skills

• overall, skill descriptors seem accurate based on student’s memory

Students highlighted the following areas where they felt that further preparation was lacking and would have been highly beneficial:

• importance of knowing how to skim/scan, speed read, manage high volume of reading

• summarising and extracting argument from lengthy readings

• understanding text organisation, extracting logical structure and understanding relationship between paragraphs and where to find information in an article

• understanding vocabulary that is complex/technical/colloquial/includes metaphors.

• importance of practising these skills

• writing/speaking and productive language skills, listening in lectures

• professional communication (social pragmatics): asking professional questions, communicating via email

Faculty/admissions focus group participants confirmed findings from the Phase

Participants in focus groups and surveys expressed concerns about students' preparedness for course content upon admission, highlighting a growing trend of same-language peer groups that contributes to a sense of isolation They noted the tension between attracting students through comparable admissions standards with other Canadian universities and the risk of setting them up for failure due to discrepancies in language abilities versus program requirements Emphasizing the need for improved communication, they called for clearer guidance on the evolving language demands students can expect at the university.

The group engaged in extensive discussion regarding recommendations that included four main areas: 1) sharing findings with relevant collaborators at the university (e.g

To enhance the experience of incoming students, the Dean’s office and committees focus on four key areas: 1) improving pre-arrival communication and education through the website, online videos, and expanded orientation modules; 2) providing ongoing university support by increasing awareness of available resources, offering targeted assistance for upper-year students, and developing online modules to track academic progress; 3) fostering professional development by expanding instructor orientation to include insights into the experiences of international students; and 4) ensuring a comprehensive support system that addresses the diverse needs of the student body.

2 The Skill Proficiency Descriptors across the IELTS Reading Band Score Levels (Table

12) and set of recommendations (Table 13) reflect feedback from the student focus group, discussed

Discussion

This study examines the validity arguments surrounding the interpretation and application of IELTS reading scores, focusing on consequential validity, the interplay between contextual and cognitive validity, and scoring validity.

A study conducted in 2011 by O'Sullivan investigated the interpretation and use of IELTS test scores at a higher education institution in Canada While standardized tests like TOEFL and IELTS are utilized for admissions at the University of Toronto, there has been no local research examining how test score users interpret and apply these external test scores.

Deville and Turner (2000) emphasized the importance of test users ensuring that admission requirements align with the academic language demands of specific programs Test developers are tasked with creating assessments that adhere to professional standards, delivering accurate insights into students' language abilities, and offering user guides to aid in local decision-making processes (AERA, NCME & APA, 1999).

This study investigates how local IELTS test score users at the University of Toronto interpret scores and utilize them for undergraduate admissions decisions Through a comprehensive four-phase approach, we aimed to understand international students' perceptions of their language proficiency and academic readiness Additionally, we developed reading skill mastery profiles to assess the potential for improving test score interpretations by employing descriptors created through scale anchoring.

Many international visa students reported feeling unprepared to meet the academic language and literacy demands upon entry, despite having fulfilled the language proficiency requirements This sentiment aligns with prior research on the academic performance of international students in the context of English for academic purposes (Bruce & Hamp-Lyons, 2015; Hamp-Lyons, 2011; Sawir, Marginson).

Forbes-Mewett, Nyland & Ramia, 2012) and discipline-specific literacy practice

Our study aligns with previous research on IELTS reading, highlighting the need for further investigation into the reading skills and strategies assessed by the IELTS Reading Module Weir et al (2009) suggested that the module may require modifications to better reflect the academic reading abilities of university students, emphasizing the importance of incorporating texts and tasks that extensively evaluate students' expeditious reading skills.

Furthermore, our results raise concerns about the lack of items at the text level Our

CDM analyses reveal that many assessment items focus on sentence-level comprehension, aligning with Weir et al (2009), where summary completion tasks emphasize word matching and sentence comprehension strategies over efficient text-level reading This limited focus raises concerns, as students often face real-life academic reading challenges that require extensive reading within tight time limits.

Our analysis of the collected course materials reveals a small yet diverse selection of programs, emphasizing the specialized reading skills necessary for effectively navigating various text types Students across these programs encounter common challenges, highlighting the importance of tailored reading strategies.

(Commerce, Economics and Engineering) were related to the amount of reading, abstract and technical academic vocabulary, and cultural background knowledge

Many students perceive the reading volume as 'overwhelming' and struggle with discipline-specific vocabulary, while implicit cultural assumptions hinder their text comprehension This creates a significant gap between students' experiences with IELTS texts and undergraduate materials Research by Weir et al (2009) indicates that IELTS texts are estimated to be at a much lower grade level than undergraduate texts, which are further complicated by cultural specificity and specialized vocabulary Our analysis of course materials supports the conclusion that IELTS texts do not adequately reflect the challenges students face when transitioning to undergraduate programs.

The debate over whether high-stakes tests should mirror real-life reading scenarios is essential, given that complete authenticity is impractical due to time constraints and the need to minimize cultural biases Nevertheless, the validity of test score interpretations hinges on how well the test's characteristics and the cognitive strategies they invoke align with the actual reading activities and processes students experience in their academic work.

International students encounter distinct challenges when entering university programs, particularly regarding their language proficiency Many quickly discover that their foundational language skills are inadequate for understanding complex, discipline-specific materials, including expository textbooks and analytical journals.

Program-specific support is crucial throughout the academic years to address the diverse language needs of students This tailored assistance helps scaffold students as their language requirements evolve and become more focused on specific disciplines.

International students often face challenges with social speaking skills and academic language demands, leading to a decline in their confidence regarding language abilities as they engage in schoolwork Survey data indicates that international visa students assess their language and literacy skills significantly lower than their domestic counterparts, despite placing a higher value on these skills, particularly higher-order inferential reasoning Notably, many upper-year students expressed a strong desire to enhance their language proficiency.

Over time, English proficiency among international students has declined, as many believe they can achieve satisfactory grades in courses that do not involve active discussion participation This mindset of diminished confidence and avoidance is counterproductive, particularly for those who aspire to build careers in Canada post-graduation.

Research indicates that IELTS test-takers who excelled in their overall scores, particularly in reading, reported feeling more prepared for academic language challenges compared to those who merely achieved the minimum required score Additionally, there is a positive correlation between IELTS test scores and CGPA, although this relationship is relatively weak for both overall and reading band scores.

CGPA across the IELTS band score levels were not statistically significant Note that we grouped the IELTS test-taker sample into three groups based on the cut-off score (≤6.5,

7.0-7.5, >7.5); therefore, the results indicate that the cut-off score may not be sensitive enough to predict different academic achievement levels

The study findings are somewhat consistent with previous research that reports a weak or no predictive relationship between IELTS and academic performance (Dooey &

Conclusions

In brief, we recommend that the IELTS partners should consider the following aspects of the test

Provide more information about what the IELTS test scores mean to users

The IELTS test is a key requirement for international students applying to the University of Toronto, but while university staff are familiar with its band scores, their understanding of its effectiveness as a measure of language proficiency is limited Despite its recognized "symbolic value," the test's role in accurately assessing language skills requires further validation and enhancement.

IELTS needs to provide decision-makers, program staff, faculty and students with more information about what test scores mean and what the test does and does not measure

Consider ways to facilitate students' test score interpretations and use, to prepare them for academic study

The disparity between IELTS reading texts and those encountered in first-year undergraduate studies greatly affects students' understanding of their test scores, their perceived readiness, and their overall attitudes towards the exam Although achieving complete authenticity in test conditions and materials may not be feasible (Green et al., 2010), it is essential to acknowledge these differences.

IELTS can support students’ desire to have more information about what their scores mean and what they should expect and be prepared for during undergraduate study

Test developers need to consider ways to facilitate students’ test score interpretations and use, to further prepare them for academic study

Investigate if the reading test has sufficient items that measure higher-order thinking skills

The development of reader profiles and proficiency-level skill descriptors relies on three key reading attributes: basic comprehension, summarizing main ideas, and inferential reasoning Our analysis indicates that a limited number of items encourage global reading that goes beyond local literal understanding, aligning with findings from previous research (Moore).

Morton & Price, 2012; Weir et al., 2009) Our finding that students at the band score

The University of Toronto's faculty and staff have expressed concern over the common local cut-off score of 6.5, noting that students' lack of inferential reasoning could negatively impact their educational experiences Further investigation is necessary to determine whether this issue arises from an insufficient number of assessment items that effectively measure higher-order thinking skills, particularly inferential reasoning at the text level.

Investigate IELTS' predictive relationship with academic success across different programs and year of program

Higher IELTS scores, particularly in reading, are linked to students feeling better prepared for academic language requirements Additionally, there is a positive correlation between IELTS reading test scores and academic success, as indicated by CGPA To fully understand IELTS' predictive validity, further research is needed to explore its relationship with academic success across various programs and academic years.

Facilitate test score users’ score interpretations and use by providing descriptive information about what the test measures and what it doesn’t

Test score users have positively received the IELTS reading proficiency skill descriptors, finding them valuable for interpreting test band scores and identifying areas for improvement Students particularly appreciated the recommendations that emphasize the characteristics of academic language and literacy requirements Faculty and staff also found the skill descriptors informative and expressed interest in exploring various ways to support students' academic success To enhance the understanding of test scores, test developers should provide clear descriptive information about what the test measures and what it does not.

Bachman, L F (2005) Building and supporting a case for test use Language

Assessment Quarterly: An International Journal, 2(1), 1–34.

Banerjee, J V (2003) Interpreting and using proficiency test scores Unpublished doctoral thesis, Department of Linguistics and Modern English Language, Lancaster

Bax, S (2015) Using eye-tracking to research the cognitive processes of multinational readers during an IELTS reading test IELTS Research Reports Online Series, 2015/2.

Beaton, A., & Allen, N (1992) Interpreting scales through scale anchoring Journal of

Bruce, E., & Hamp-Lyons, L (2015) Opposing tensions of local and international standards for EAP writing programmes: Who are we assessing for? Journal of English for Academic Purposes, 18, 64–77.

Chalhoub-Deville, M., & Turner, C E (2000) What to look for in ESL admission tests:

Cambridge certificate exams, IELTS, and TOEFL System, 28(4), 523–539.

Chapelle, C A., Enright, M K., & Jamieson, J M (Eds.) (2008) Test score interpretation and use (p 1–25) Building a validity argument for the Test of English as a Foreign

Charmaz, K & Bryant, A (2008) Grounded theory In L M Given (Ed.), The SAGE encyclopedia of qualitative research methods (pp 375–376) Thousand Oaks, CA: SAGE

Coleman, G., & O’Connor, R (2007) Using grounded theory to understand software process improvement: A study of Irish software product companies Information and

Coleman, D S., Starfield, S., Hagan, A (2003) The attitudes of IELTS stakeholders:

Student and staff perceptions of IELTS in Australia, UK and Chinese tertiary institutions

IELTS Research Reports Vol 5 Canberra: IDP Education Australia

Coley, M (1999) The English language entry requirements of Australian universities for students of non-English speaking background Higher Education Research &

Costello, A B., & Osborne, J W (2005) Best practices in exploratory factor analysis:

Four recommendations for getting the most from your analysis Practical Assessment,

Cotton, F., & Conrow, F (1998) An investigation of the predictive validity of IELTS amongst a group of international students studying at the University of Tasmania

Deakin, G (1997) IELTS in context: Issues in EAP for overseas students EA Journal,

DiBello, L., Roussos, L., & Stout, W (2007) Cognitive diagnosis part I In C R Rao,

& S Sinharay (Eds.), Handbook of statistics (Vol 26): Psychometrics (pp 979–1030)

Dooey, P., & Oliver, R (2002) An investigation into the predictive validity of the IELTS Test

Gerbing, D W., & Hamilton, J G (1996) Viability of exploratory factor analysis as a precursor to confirmatory factor analysis Structural Equation Modeling:

Glaser, B G., & Strauss, A L (2017) Discovery of grounded theory: Strategies for qualitative research Routledge.

Golder, K., Reeder, K., & Fleming, S (2009) Determination of appropriate IELTS band score for admission into a program at a Canadian post-secondary polytechnic institution.

IELTS Research Reports Vol 10, 1 Canberra: IELTS Australia and London: British

Gomez, P G., Noah, A., Schedl, M., Wright, C., & Yolkut, A (2007) Proficiency descriptors based on a scale-anchoring study of the new TOEFL iBT reading test

Government of Canada (2014) Canada’s international education strategy Retrieved from: http://international.gc.ca/global-markets-marches-mondiaux/assets/pdfs/overview- apercu-eng.pdf

Green, A., ĩnaldi, A., & Weir, C (2010) Empiricism versus connoisseurship: Establishing the appropriacy of texts in tests of academic reading Language Testing, 27(2),

Hamp-Lyons, L (2011) What Is EAP? Handbook of Research in Second Language

Hartz, S., Roussos, L., & Stout, W (2002) Skills diagnosis: Theory and practice

User Manual for Arpeggio software Educational Testing Services.

Henson, R., & Templin, J (2007) Large-scale language assessment using cognitive diagnosis models Paper presented at the annual meeting of the National Council for

Measurement in Education, April 2007, Chicago, IL.

Hinkin, T R (1998) A brief tutorial on the development of measures for use in survey questionnaires Organizational Research Methods, 1(1), 104-121.

Hurley, A E., Scandura, T A., Schriesheim, C A., Brannick, M T., Seers, A., Vandenberg,

R J., & Williams, L J (1997) Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives The International Journal of Industrial, Occupational and

Hyatt, D., & Brooks, G (2009) Investigating stakeholders' perceptions of IELTS as an entry requirement for higher education in the UK IELTS Research Reports Vol 10, 1

Canberra: IELTS Australia and London: British Council.

Ingram, D., & Bayliss, A (2007) IELTS as a predictor of academic language performance, Part 1 IELTS Research Reports Vol 7, 1 Canberra: IELTS Australia and

Jang, E E (2005) A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG-TOEFL Unpublished doctoral dissertation, University of Illinois at Urbana–Champaign, Urbana, IL

Jang, E E (2009) Demystifying a Q-matrix for making diagnostic inferences about

L2 reading skills Language Assessment Quarterly, 6(3), 210–238.

Junker, B W., & Sijtsma, K (2001) Cognitive assessment models with few assumptions, and connections with nonparametric item response theory Applied Psychological

Khalifa, H., & Weir, C J (2009) Examining reading (Vol 29) Ernst Klett Sprachen.

Kim, A Y (2015) Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability Language Testing, 32(2),

Krueger, R A., & Casey, M A (2000) Focus groups: a practical guide for applied research, 3-19.

Lee, Y W., & Sawaki, Y (2009) Application of three cognitive diagnosis models to ESL reading and listening assessments Language Assessment Quarterly, 6(3), 239–263.

Li, H., Hunter, C V., & Lei, P W (2016) The selection of cognitive diagnostic models for a reading comprehension test Language Testing, 33(3), 391–409.

McDowall, C., & Merrylees, B (1998) Survey of receiving institutions' use and attitude to

IELTS IELTS Research Reports Vol 1, 116.

McNamara, T., & Roever, C (2006) Language testing: The social dimension (Vol 1)

Milanovic and Weir (2010) provide a critical examination of language assessments in relation to the Common European Framework of Reference for Languages Their work includes case studies and reflections on the implementation of the Council of Europe’s Draft Manual, highlighting its significance in standardizing language proficiency evaluations.

Moore, T., Morton, J., & Price, S (2012) Construct validity in the IELTS Academic

Reading test: A comparison of reading requirements in IELTS test items and in university study IELTS Research Reports Vol 11, 1–89 Canberra: IDP: IELTS Australia and

Morgan, D (2008) Focus groups In L M Given (Ed.), The SAGE encyclopedia of qualitative research methods (pp 353–354) Thousand Oaks, CA: SAGE Publications

O'Loughlin, K (2008) The use of IELTS for university selection in Australia: A case study

IELTS Research Reports, Vol 8, 1 Canberra: IELTS Australia.

Onwuegbuzie, A J., Dickinson, W B., Leech, N L., & Zoran, A G (2009) A qualitative framework for collecting and analyzing data in focus group research International

Ravand, H (2016) Application of a cognitive diagnostic model to a high-stakes reading comprehension test Journal of Psychoeducational Assessment, 34(8), 782–799.

Ravand, H., & Robitzsch, A (2015) Cognitive diagnostic modeling using R Practical

Rea-Dickins, P., Kiely, R., & Yu, G (2007) Student identity, learning and progression:

The affective and academic impact of IELTS on 'successful 'candidates IELTS Research

Reports, Volume 7, 1 Canberra: IELTS Australia and London: British Council.

Robitzsch, A., Kiefer, T., George, A C., Uenlue, A., & Robitzsch, M A (2019) Package

‘CDM’ Handbook of diagnostic classification models New York: Springer.

Rupp, A A., Templin, J., & Henson, R A (2010) Diagnostic assessment: Theory, methods, and applications New York: Guilford.

Sawir, E., Marginson, S., Forbes-Mewett, H., Nyland, C., & Ramia, G (2012) International student security and English language proficiency Journal of Studies in International

Schmitt, T A (2011) Current methodological considerations in exploratory and confirmatory factor analysis Journal of Psychoeducational Assessment, 29(4), 304–321.

Shanahan, T., & Shanahan, C (2012) What is disciplinary literacy and why does it matter? Topics in Language Disorders, 32(1), 7–18.

Sinharay, S., Haberman, S J., & Lee, Y H (2011) When does scale anchoring work?

A case study Journal of Educational Measurement, 48(1), 61–80.

In their 2007 report, Smith and Haslett examine the attitudes of key decision-makers in tertiary education towards English language testing in Aotearoa New Zealand Their findings, based on a national provider survey, are detailed in IELTS Research Reports, Volume 7, pages 1-44, highlighting the perspectives and implications of English language assessments in the educational landscape.

IELTS Australia and London: British Council.

Strauss, A., & Corbin, J (1998) Basics of qualitative research (2nd ed.) Thousand

Taylor, L., & Weir, C J (Eds.) (2012) IELTS collected papers 2: Research in reading and listening assessment (Vol 2) Cambridge University Press.

Teddlie, C., & Tashakkori, A (2006) A general typology of research designs featuring mixed methods Research in the Schools, 13(1), 12–28

University of Toronto (2016–17) Enrolment report Retrieved at: UT: http://www.provost. utoronto.ca/Assets/Provost+Digital+Assets/Enrolment+Report+2016-17+-+Final.pdf

Wang, C., & Gierl, M J (2011) Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading Journal of Educational

Weir, C., Hawkey, R., Green, A., & Devi, S (2012) The cognitive processes underlying the academic reading construct as measured by IELTS IELTS Collected Papers 2:

Research in Reading and Listening Assessment, 2, 212.

The study by Weir et al (2009) explores the connection between the academic reading skills assessed by the IELTS exam and the reading experiences of first-year students at a British university The findings highlight how IELTS scores reflect students' reading proficiency and their ability to engage with academic texts This research provides valuable insights into the effectiveness of IELTS as a predictor of reading success in higher education contexts.

97 Canberra: IELTS Australia and London: British Council.

Weir, C., & Khalifa, H (2008) A cognitive processing approach towards defining reading comprehension Cambridge ESOL: Research Notes, 31, 2–10.

Weir, C J., & O'Sullivan, B (2011) Test development and validation Palgrave Macmillan.

Yi, Y S (2017) Probing the relative importance of different attributes in L2 reading and listening comprehension items: An application of cognitive diagnostic models Language

Appendix A: Phase 1 focus group protocol – student

1 [To be filled out by the team]

2 Participants are given the following documents to be collected before the focus group starts:

• Explain the purpose of the study and focus group

1 REASON FOR APPLYING TO UOFT

1.1.1 When did you decide to apply to U of T and why?

1.1.2 How did you prepare your application for admission?

1.1.3 Let’s talk more about language requirements Which test did you take for admission? How was your experience with the test?

1.1.4 What did the test score say about your English language proficiency?

Probing: Was it a fair representation of your ability?

3 LANGUAGE DEMANDS IN THE CLASSROOM

1.1.5 Let’s go back to the first week of the school semester Can you describe your experience in the first week in terms of language demands?

1.1.6 How about language demands outside of the classroom?

1.1.7 Since the start of the semester, you may have read a number of course materials, including textbooks, online resources, manuals, lecture notes etc

What reading materials do you usually read?

Probing: Are there any challenges with these materials?

Probing: How do cope with them? Do you use any particular reading strategies?

1.1.8 Now, we’d like to think about specific reading processes and strategies

Assess your reading skills by reviewing this list of strategies, rating your proficiency in each, and reflecting on whether you feel you need improvement or have made progress in recent months.

Distribute the self-assessment form

• Relevance (is it an important skill for my field of study)?

Upon finishing the form, students keep the form for discussion and return it later

1.1.9 Is there any strategy that is not relevant to your field of study?

1.1.10 Is there any strategy that is not included in the list but critical for your field of study?

1.1.11 What is the most difficult strategy to master?

1.1.12 Have you sought any help to deal with English language demands?

Probing: How useful was it?

Probing: Are there any supports you wish the school provided for international students’ language needs?

Appendix B: Phase 1 focus group protocol – instructor

Instructors are requested to provide online or printed materials from a specific course that presents challenges for international students, drawing from their experiences They will share their syllabi to give insight into the language requirements of the course The focus should be on a course where international students typically face significant difficulties.

1 [To be filled out by the team]

2 Participants are given the following documents to be collected before the focus group starts:

• Consent form (Collect before starting the focus group)

• Explain the purpose of the study and focus group

Hello and thank you so much for taking the time to join today’s focus group I am

Our research project, funded by Cambridge IELTS, aims to improve the interpretation of test scores and inform decision-making regarding student admissions and support A key concern is the discrepancy between test measurements and the skills required for success in disciplines like engineering To address this, we are gathering insights from students and instructors to gain a clearer understanding of the specific reading demands within these fields.

The aim is to enhance discussions on effective support strategies throughout the campus, and I am here to facilitate these conversations I encourage you to engage openly with fellow members of the focus group.

Do you have any question? Great Let’s get started

1.1.1 How do international students do in your courses? Let’s start with first year students?

Probing: What is your general observation of their language proficiency?

How well are they prepared when they start the program?

Where do they struggle most? Why?

1.1.2 How about upper year students? What changes do you see as students progress to upper years?

Probing: Do students improve their language proficiency?

Where do you see the most improvement?

Where do they struggle even after spending time here?

1.1.3 Now let’s delve into specific academic reading demands in your courses

To enhance our discussions, we requested that you bring specific course materials for review Please examine the materials you provided and briefly share which class they pertain to, whether it is field general or specific, and identify the intended audience.

1.1.4 Can you identify some key reading activities students are expected to do?

Probing: Are there any unique skills your program students should have for your course using the material?

Where do students struggle most?

1.1.5 Did you have any successful experiences with supporting students struggling with specific reading demands?

Probing: Do you have any suggestions for other faculty teaching international students for the first time?

What other areas do you think students are in need of?

Are you aware of any initiatives (e.g professional development workshops) or supports provided to faculty to improve how they support/teach international students?

Appendix C: Domain analysis of academic language and literacy demands

Year Course Text type Evaluation

Final Exam (50%) Research Requirement (1%) Online homework/Quizzes (10%)

Final Exam (40%) Online homework/Quizzes (10%) Essays (10%)

Midterm Exam (30%) Research Requirement (3%) Group Assignment (Paper + Presentation) (35%) Class Participation/Contribution (17%)

Final Exam (32%) Research Requirement (3%) Group Assignment (Presentation) (20%) Class Participation/Contribution (10%) News Briefings (5%)

Take-home Exam/Reflection (25%) Group Assignment (35%) Class Participation/Contribution (10%) Individual Case Assignment (x2) (30%)

Midterm Exam (20%) Final Exam (30%) Research Requirement (3%) Group Assignment (Presentation) (35%) Class Participation/Contribution (12%)

Textbook Chapters (46%) Final Exam (40%) Tutorial/Class Participation (4%) Portal Quizzes/Online Tests (9%) Warm-up Exercise (1%)

Final Exam (40%) Tutorial/Class Participation (4%) Portal Quizzes/Online Tests (9%) Warm-up Exercise (1%)

Final Exam (25%) Tutorial/Class Participation (5%) Homework (8%)

Midterm Exam (49%) Final Exam (25%) Tutorial/Class Participation (5%) Portal Quizzes/Online Tests (21%)

Midterm Exam (30%) Final Exam (50%) Problem Sets (20%)

Midterm Exam (30%) Final Exam (45%) Portal Quizzes/Online Tests (10%) Assignment (Writing) (15%)

Course Overview (12%) Group Assignment (report/presentation) (48%)

Final Paper (30%) Tutorial/Class Participation (15%) Assignment (Writing) (30%) Group Assignment (Presentation) (25%)

Final Exam (45%) Lab (15%) Projects/Assignment (10%)

Final Exam (40%) Quizzes (2%) Practical (15%) Tutorial (8%)

Final Exam (50%) Lab (10%) Project/Assignment (10%)

Final Exam (45%) Project/Assignment (10%) Quizzes (20%)

Appendix D: Characteristics of five CDMs

DINA (Junker & Sijtsma, 2001) Non-compensatory Identity

DINO (Templin & Henson, 2006) Compensatory Identity

G-DINA (de la Torre, 2011) General Identity

NC-RUM (DiBello et al., 1995; Hartz, 2002) Non-compensatory Log

Appendix E: Undergraduate language demand survey

A1 I have read the above information.

A2 I agree to participate in the survey.

A3 I agree to participate in a focus group

A7 Current Year of Study (Year 1, Year 2, etc.):

The following section will ask about your demographic background

B1 What is your year of birth?

B3 Which language do you use most fluently?

B4 Do you use another language(s)?

B5 Indicate each and your proficiency level:

B6 Indicate each and your proficiency level:

B7 Were you born in Canada?

B9 What year did you arrive in Canada?

B10 What is your current status?

B11 What high school curriculum did you complete?

□ Canadian curriculum outside of Canada

□ Non-Canadian curriculum in Canada

□ Non-Canadian curriculum outside of Canada

B12 What was your high school Grade Point Average (GPA) in English?

B13 What was your high school Grade Point Average (GPA) in Math?

B14 What was your overall high school Grade Point Average (GPA)?

B15 How much time did you take off?

Section C: Demographic Information, Part II

C1 What year did you start studying at the University of Toronto?

C2 Did you take any time off from University of Toronto?

C3 Select the language test or program you took as part of your admission to the University of Toronto?

□ Canadian Academic English Language (CAEL) Assessment

□ International Foundation Program (IFP) course

□ UT School of Continuing Studies, Academic English (Level 60)

□ Other preparatory course at U of T

C4 On the Cambridge English Language Assessment what was your score for

C5 On the CAEL what was your score for

C6 On the CanTEST what was your score for

C7 On the ELDA/COPE what was your score for…

C8 On the ELDA/TOP what was your band score?

C9 On the IELTS what was your score for

C10 In the International Foundation Program (IFP) course what was your grade?

C11 On the MELAB what was your score on

C12 On the TOEFL cBT what was your score on

C13 On the TOEFL iBT what was your score for

C14 On the TOEFL PBT what was your score for

C15 In the UT School of Continuing Studies Academic English (Level 60) course what was your grade?

C16 In the other preparatory course at U of T what was your grade?

C17 Did you think your language test score reflected your language proficiency at that time?

C18 Please explain why you feel this way:

C19 After you met the language requirements for university admission

(e.g., achieved IELTS 6.5 or high school completion) and started your university program, did you feel you were prepared for the language demands in your courses?

C20 In what faculty are you registered?

C21 For your current program, what year are you in?

C22 What is your current major or specialist?

C23 Have you changed your major?

C24 Are you considering changing your major?

C25 What is your career goal after graduation?

C26 Why are you interested in this career?

C27 What kind of skills do you think are necessary for the career in which you are interested? (You can choose more than one.)

C28 Select all skills in which you feel prepared:

C29 Please explain why you feel this way.

C30 What is your Cumulative Grade Point Average (CGPA)?

C31 Does your Cumulative Grade Point Average (CGPA) reflect your academic ability?

C32 Please explain why you feel this way:

The following section will ask about your academic language demands

D1 Reflecting on your school-work since the start of the current school year, how often do you do the following

(Never, Rarely, Sometimes, Often, Most of the time, Always)

• Communicate with course instructors/tutors/TAs to discuss coursework

• Write a short assignment (e.g., lab report, short paper)

• Write a long paper (e.g., final term paper)

• Read lecture notes or PowerPoint slides

• Read long academic materials (e.g., articles, book chapters)

• Read text with statistics tables and figures

• Read entrepreneurial, legal and government documents

• Read text in a language other than English

• Read literature and/or fiction (e.g., novels, short stories, poetry)

• Read mathematical equations and computational programming codes

• Understand written instructions for assignments and tests

• Scan and skim for keywords in course readings

• Understand main ideas in course readings

• Summarise main ideas in course readings

• Distinguish the main idea from minor details in course readings

• Distinguish facts from opinions in course readings

• Understand technical vocabulary in course readings

• Understand cultural and idiomatic expressions in course readings

• Understand grammatically complex text in course readings

• Make prediction based on course readings

• Evaluate a writer's viewpoint using additional sources

• Generate questions based on course readings

• Understand implied ("hidden") meanings in course readings

• Solve problems by applying the information from a course reading to real life

• Read a large amount of materials in a limited time

• Have sufficient background knowledge about other cultures in order to understand course readings

Section E: Language Demands, Part II

E1 Reflecting on your school-work since the start of the current school year, how important is it to do the following

(Not well at all, Not well, Somewhat not well, Somewhat well, Well, Very well)

• Read professional magazines (e.g., Accounting Today, American Banker,

• Read entrepreneurial, legal and government documents

• Summarise main ideas in course readings

Section F: Language Demands, Part III

F1 Reflecting on your school-work since the start of the current school year, how well can you do the following

(Not well at all, Not well, Somewhat not well, Somewhat well, Well, Very well)

• Read professional magazines (e.g., Accounting Today, American Banker,

• Read entrepreneurial, legal, and government documents

Thank you for completing the survey Your answers to the questions in each section of the survey are important to the research study

Appendix F: Language competence CFA mode

Appendix G: Language importance CFA mode

Appendix H: Phase 3 focus group protocol – student

Participants are given the following documents to be collected before the focus group starts:

• Can-do descriptor document “IELTS Reading Skill Report” (with draft letter)

• Introduction of researchers and participants

• Purpose of focus group and general rules

• Once you met the required IELTS reading band score, do you think you were well-prepared for the reading demands at UofT?

Probing: In terms of reading skills, how well prepared were you when you started at UofT?

What reading skills do you feel you need to develop to succeed in your program?

Think about your IELTS reading band score What did the test score say about your reading skills? What do you think students can do with a _ score?

3 FEEDBACK ON THE “IELTS Reading Skill Report”

• [Introducing the Can-Do Descriptor Form – IELTS Reading Skill Report]:

We recommend including the "IELTS Reading Skill Report" with the university admission letters for prospective international students.

Phase 3 focus group protocol – instructor

Participants are given the following documents to be collected before the focus group starts:

Phase 1 project findings (Student and Instructor Focus groups and thematic analyses)

• Open for discussion/feedback/response

Findings and analyses from the Undergraduate Language Demands Survey

Description of scale anchoring findings and sharing of proposed enriched communication to incoming students

• Open discussion regarding recommendations, applications within their own programs/faculties

Appendix J: Self-assessment questionnaire to stimulate participant thinking during Phase 1 focus groups

A I can do the following in my courses

(Strongly Agree, Agree, Somewhat Agree, Somewhat Disagree, Disagree,

• Understanding the main points and supporting details in course readings.

• Understanding hidden meanings ("reading between the lines") in course readings.

• Understanding information that is directly stated in a course reading.

• Understanding how parts of a text are organised.

• Connecting what is written in the text to my own knowledge.

• Understanding and identifying different types of texts (e.g reports, government documents, etc.) on a topic.

• Reading a text and understanding why it was written and who it was written for.

• Using my vocabulary and sentence knowledge to understand technical words in a text.

• Using my vocabulary and sentence knowledge to understand non-technical words in a text.

• Critically evaluating the claims, evidence or data presented in a printed text.

• Evaluating the validity and accuracy of digital texts.

• Drawing on ideas from a range of texts to support my own argument.

• Connecting ideas from a variety of text types and media.

• Interpreting graphs, tables, and other types of data visualisation.

• Using information from texts to make predictions.

B This is a very important skill for students in my program