Research aims and questions
This study aims to investigate the cognitive processes of IELTS test-takers during AWT1 tasks, similar to the work of Yu et al (2011) However, while Yu et al relied on think-aloud protocols for data collection, this research employs the Tobii X2-60 eye-tracking technology to gain insights into the participants' cognitive processes during the test The specific research questions guiding this study are outlined as follows.
RQ1: What are the cognitive processes involved in taking IELTS AWT1 tasks?
RQ2: To what extent are there differences in test-takers’ cognitive processes due to different features of AWT1 graph prompts?
RQ3: To what extent are test-takers’ cognitive processes affected by their graphicacy?
RQ4: To what extent are test-takers’ cognitive processes related to their English writing abilities?
Research site and participants
In order to make meaningful comparisons between this and our previous research
(Yu et al 2011), we collected data from the same institution – Zhejiang University
(www.zju.edu.cn) It is one of the largest and most prestigious universities in China; a large number of its undergraduate and postgraduate students take the IELTS
The call for participation (Appendix 2) was circulated on the university’s websites
Nearly 800 students expressed interest in participating in a study by registering at www.survey.bris.ac.uk, providing personal details such as their name, contact information, and IELTS test experience From this group, 5% were selected as potential participants based on criteria including their intention to take the official IELTS test within six to nine months or prior test experience, ensuring familiarity with IELTS AWT1 To create a balanced sample, factors such as gender, academic discipline (science, social sciences, or arts), and academic status (undergraduate or postgraduate) were also taken into account A waiting list was maintained to replace any participants who withdrew or failed eye calibrations Those who completed all tasks received a £20 honorarium as a token of appreciation for their involvement.
In total, 34 students participated at various points (i.e., completed at least one of the tasks in the project, they were identified as Participant #1, Participant #2, …
Participant #34) To further ensure anonymity, every participant was reported as “he”
Our study analyzed data from 27 students whose eye movements were effectively recorded These participants completed all writing tasks, interviews, and focus-group discussions, providing a comprehensive dataset for our analyses.
27 students Nine of them took an official IELTS test recently Some basic information about these participants is reported in Table 1
To minimize the impact of handwriting quality on raters' evaluations during the eye-tracking tasks, we opted to word process two written responses that participants initially composed on paper Nevertheless, it is essential to acknowledge that IELTS raters generally assess handwritten submissions.
Table 1: Characteristics of 27 participants who completed every data collection
Arts and Humanities Engineering Medicine Science Social science
PhD or other doctoral study
The study sample comprised 15 female and 12 male students, with 12 enrolled in undergraduate programs, 10 in master's programs, and 5 pursuing doctoral degrees Among these students, one-third (9) were focused on social sciences, while 7 were studying engineering.
6 from science, 4 from arts and humanities, and 1 from medicine.
Data collection procedure
The research collected qualitative and quantitative data at three stages using different instruments, as summarised below
In the initial phase of our project, we gathered baseline data during a single session Prior to data collection, we provided students with a detailed explanation of the project's purpose and procedures, ensuring they understood before signing the consent form (Appendix 3) To assess the students’ writing skills, we administered IELTS Academic Writing Tasks 1 and 2 (Appendices 4 and 5) in a standard testing environment.
IELTS test condition The students’ hand-written scripts were word processed 5 (in
Calibri, font size 11) as they were, i.e., no grammatical errors or typos were corrected, before being marked by IELTS certificated raters A few scripts were double-marked
Second, we administered the graphicacy questionnaire to understand the participants’ knowledge, familiarity and experience of using different types of graphs (Appendix 6)
We conducted a questionnaire to assess students' familiarity and experience with computers and word processing software, as detailed in Appendix 7 This measure aims to evaluate their knowledge and proficiency in using such technology effectively.
The Tobii eye-tracking system utilized is an interactive, screen-based tool that requires students to type their written responses Given the high level of computer literacy among participants, this approach was taken as a precautionary measure.
The results of the questionnaire confirmed our prediction (see Section 4)
At the second stage, each participant was randomly assigned to three out of four
AWT1 tasks of different graph prompts (Appendices 8 to 11); and the order that the participants completed their three tasks was also randomised
• Eye-tracking Task 1 (E1 hereafter) has two graphs (one line, the other horizontal bar) about credit card debt
• Eye-tracking Task 2 (E2 hereafter) has one vertical bar and the other pie chart about the carbon dioxide emissions (1990–2008) and the sources for producing electricity (2008) in China
• Eye-tracking Task 3 (E3 hereafter) has one line graph about the global fossil carbon emissions from 1880 to 2000
• Eye-tracking Task 4 (E4 hereafter) has two statistical tables about IELTS and
TOEFL iBT test-taker performance by geographic regions in Asia in 2011 and
The study involved a consistent number of participants across four tasks, adhering to the methodology established by Yu et al (2011) Each task was allotted 20 minutes, with a five-minute break in between to maintain participant focus Time reminders were provided verbally at intervals of 10, 5, and 3 minutes.
The tasks were presented on the screen (15 inch diagonal, 10 inch height,
1920x1080 resolution) of Hewlett-Packard Elitebook 8570 laptop (Windows 7
Professional, Core i7, 8GB RAM), as a “screen recording” element in Tobii Studio
In the Enterprise version 3.2.1 study, tasks were displayed on the left half of the screen while participants typed their responses on the right, with spelling and grammar checks disabled Originally designed as two separate A4-size pages in an Adobe fillable form, the tasks were presented in a non-scrollable two-page view to ensure uniformity among participants, thereby minimizing errors in defining Areas of Interest (AOI) for eye-movement data analysis This layout simulates a real-life paper-based testing environment, allowing test-takers to read prompts on the left and write simultaneously on the right Despite the smaller appearance of text and graphs, a pilot study confirmed that they remained sufficiently large and readable for students with normal eyesight.
This research prioritizes broader themes over individual words, making the font size of specific lexical items less significant compared to Spinner, Gass, and Behney’s (2013) analysis of "articles" within reading passages.
However, we do agree with them that screen layout is critical in any eye-tracking research
The Tobii X2-60 eye-tracker, with a 60 Hz sampling rate, captured participants' eye movements without the need for chin rests It operates at a distance of 45–90 cm and allows for head movement tolerance up to 70 cm, covering a 50x36 cm area To reduce distractions, the eye-tracker was mounted centrally at the bottom of the laptop screen Each participant's eye fixations and saccades were meticulously calibrated for accuracy, and the eye-tracking procedure was thoroughly explained both verbally and through on-screen instructions in Tobii Studio before the experiment commenced.
The calibration type was set as “regular” with “red” as foreground colour and
The calibration process utilized a medium speed with nine calibration points, although some participants were withdrawn due to calibration failures, often caused by unusual colored or shaped glasses or contact lenses The screen capture was configured to a 10 frame rate, with the integrated camera and audio of HP laptops activated, allowing simultaneous recording of participants' head movements and background sounds, such as typing This recorded data serves as valuable contextual information for analyzing each participant's eye movement The I-VT filter was employed as the fixation filter in Tobii Studio, following specific settings for accurate data interpretation.
Gap fill-in (interpolation) Enabled, with max gap length 75 ms
Eye selection Average of left and right
I-VT classifier Velocity threshold: 30 degrees/second
Merge adjacent fixations Enabled, with max time between fixations 75 ms, and max angle between fixations 0.5 degrees Discard short fixations Enabled, with minimum fixation duration of 60 ms
Following the completion of the three eye-tracking tasks, participants were individually interviewed, using their recorded eye-movement videos as stimuli to facilitate discussions on the cognitive processes involved This exploration aimed to understand how various graph prompts, along with participants' graphicacy and writing skills, influenced their cognitive processes The interviews were conducted in Chinese, and although the initial plan was to replay the full-length eye-movement videos for each participant, this approach was only implemented with the first few individuals.
We realized that requiring participants to engage for extended periods became unrealistic and burdensome, particularly as many were fatigued from intensive work and prolonged screen time To enhance participant engagement and contributions, we opted to replay only randomly selected episodes as discussion stimuli (refer to Appendix 14) Each interview varied in length, lasting from a few minutes to half an hour per task, resulting in a total of approximately [insert total number] interviews conducted.
11 hours of retrospective stimulated interviews; with just less than half an hour, on average, with each participant
Following the completion of the stimulated recall interviews, we held six focus group discussions conducted in Chinese, utilizing the same guiding questions as those used in the individual retrospective stimulated interviews.
In contrast to individual interviews, the group discussions were primarily student-led, with the researcher stepping in only when necessary to reduce their influence on student responses and interactions While the stimulated recall interviews were not completed as initially intended, they, along with the focus-group discussions, offer valuable supplementary insights that enhance the interpretation of participants' eye-movement data.
This research involved three distinct stages, as detailed in Table 2 The data collected encompassed participants' performance in two writing tasks—graph-based and topic-based—conducted under normal examination conditions, along with assessments of their graphicacy and familiarity with computers.
In Stage 2, data was gathered on participants' cognitive processes while completing three distinct AWT1 tasks using eye-tracking technology and retrospective stimulated interviews Stage 3 involved conducting six student-led focus group discussions Additionally, field notes were recorded throughout the data collection, offering valuable insights into the participants and their test-taking experiences.
Table 2: Summary of data collection stages, sources and size
Data collection stage Instrument/data Data size
Stage 1 (normal test condition) IELTS AWT1
IELTS AWT2 (topic-based) Graphicacy questionnaire Familiarity: computer and word processing
Stage 2 (eye-tracking experiments) Eye-movements videos
Stimulated retrospective interviews videos IELTS AWT1 (three tasks)
Stage 3 (focus group) Student-led focus-group discussions 6 hours
Methods of data analysis
A mixed approach in data analysis was adopted to explore the multiple sources of data in order to understand the complexity of test-takers’ cognitive processes
In Stage 1, participants' written scripts were collected under normal examination conditions and were processed without any corrections for grammatical errors or typos Additionally, their writings during the eye-tracking experiments were obtained using Tobii technology.
In a studio setting, anonymised scripts were evaluated by certified IELTS raters based on established IELTS criteria, with some scripts undergoing double-marking to ensure rating consistency This assessment, combined with data on participants' computer familiarity and graphicacy, was utilized to analyze the relationship between these factors and writing performance in the test.
Qualitative interviews and focus-group discussions were transcribed, coded, and categorized using Nvivo 10 to gain insights into test-taking cognitive processes from the participants' perspectives Additionally, eye-tracking data served as a primary source of both quantitative and qualitative evidence, shedding light on how participants engaged in these cognitive processes during testing.
To begin analyzing qualitative data, we examined the visualizations of recorded eye movements to gain a comprehensive overview Using Tobii Studio, we individually replayed each recording and accessed the eye movement data through the "visualisations" function This process involved viewing the data in two modes—“sliding window” and “accumulate”—and analyzing both “gazeplot” and “heatmap” outputs sequentially.
After reviewing individual recordings, we compared the accumulated gazeplots and heatmaps for each participant across different AWT1 tasks to visually analyze variations in eye fixations and saccades This comparison highlighted differences among participants and tasks (refer to Appendices 15 and 16 for examples) Utilizing the same eye-movement data from three participants in Task 1, we generated the gazeplots and heatmaps Following the individual-level analysis, we examined the visualizations of all recordings at the task and group levels across various time segments.
We analyzed the distinctions and commonalities in visualizations based on various graph types, including tables, line graphs, pie charts, horizontal bar graphs, and vertical bar graphs (refer to Appendices 8 to 11) Our examination also considered participants' English writing proficiency, graphicacy skills, and familiarity with computers.
For our statistical analysis of eye movement data, we established Areas of Interest (AOIs) within the recorded media In the left half of the screen, we identified two to three AOIs based on the number of graphs presented in the task prompt, with the standard task instructions designated as a single AOI and each graph treated as an individual AOI.
In the study, we identified three Areas of Interest (AOIs) in the left half of the screen when two graphs were utilized in a task Conversely, the right half featured a single AOI, covering roughly the top 50% of the main textbox where participants input their responses The analysis focused on eye fixations across various AOIs and the saccades occurring between them.
AOIs, especially the saccades between the AOIs in the left half and those in the right half of the screen provide the essential evidence into test-takers’ cognitive processes
Before analyzing the eye-movement data to answer the four research questions, it is essential to outline the participants' characteristics, including their familiarity with computers, graphicacy skills, and proficiency in English writing.
Participants’ familiarity with using computers and word processing
To ensure a thorough understanding of participants' computer and word processing skills, we assessed their familiarity with these tools The responses to Question 2 (refer to Appendix 7) indicate that usage patterns can be mutually exclusive; for instance, students who frequently use computers in their dormitories may not utilize university lab computers as often, and vice versa.
We selected the highest score from Questions 2a, 2b, and 2c as the representative score for Question 2 The questionnaire utilized a 1 to 4 scale, where a higher score reflects greater familiarity with computers The maximum achievable score for the questionnaire is 72, calculated from 18 questions, each worth 4 points.
The mean score of nearly 63, representing 87.5% of the maximum possible score, confirms our prediction that participants possess a high level of familiarity with computer usage and word processing, with a standard deviation of 4.85.
The difference between male and female students, with female students about
3 points higher, was not statistically significant (note the small sample)
Figure 1: Participants’ familiarity with using computers and word processing
The students demonstrated a higher proficiency in Chinese word processing compared to English, with average familiarity scores of 4.00 for Chinese and 3.26 for English Additionally, their ability to send emails reflected a similar trend, with mean scores of 3.89 for Chinese emails and 2.81 for English emails, indicating a greater comfort level with Chinese communication tools.
Table 3: Participants’ familiarity with word processing in English and Chinese
Paired Samples Test Paired Differences t df
95% Confidence Interval of the difference Lower Upper
Pair 2 Sending English emails – Sending
Participants’ graphicacy
The graphicacy questionnaire used a scale of 1 to 6, with a larger number indicating a higher graphicacy level for all the questions but No 12, 13, 33 and 34–37
For Questions 12, 13, and 33, a higher score indicated a lower graphicacy level due to the negative phrasing of the statements; thus, participants' responses were recoded for consistency with other questions Questions 34–37 sought participants' opinions on the correlation between their graphicacy and IELTS AWT1 performance, rather than directly measuring their graphicacy levels The data from these four questions were analyzed separately.
31 items in the questionnaire to measure the participants’ graphicacy level, with the maximum of 186 (31x6) points and minimum of 31 (31 x1) and Cronbach’s
As shown in Figure 2, the mean of the participants’ graphicacy was 138.2
The study revealed that participants achieved an average score of approximately 74.2% of the maximum possible, with a standard deviation of 3.0 Their graphicacy profile closely resembled that of the participants in Yu et al (2011) Notably, there was no statistically significant difference between male and female students, although male students scored, on average, 7.5 points higher.
Table 4: Participants’ familiarity with different types of graphs (N') graphQ14 (bar graph) graphQ15 (line graph) graphQ16 (pie) graphQ17 (diagram) graphQ18 (statistical table)
Questions 14, 15, 16, 17 and 18 measured the participants’ familiarity with particular types of graphs (bar, line, pie, diagram and statistical table respectively) The data
The participants showed a greater familiarity with pie charts compared to other graph types, while their familiarity with statistical tables was the lowest Despite these observations, none of the differences in familiarity levels were statistically significant.
The study explored participants' perspectives on the connection between their graphicacy skills and performance in IELTS Academic Writing Task 1 Responses to Question 34 revealed diverse opinions, indicating that concerns about effectively demonstrating writing ability due to difficulties in graph description were prevalent across six categories However, the chi-square test results (chi-square=3.89, df=5, n.s.) showed no significant differences among these categories.
Many participants underestimated the impact of their graph description skills on their IELTS Academic Writing Task 1 performance Despite this, a significant majority (24 out of 27, chi-square 0.59, df=4, p