Part combined 14 06 2018 – ETRA Tutorials Department of Psychology Tutorial 3 Gaze Analytics Pipeline Andrew Duchowski (Clemson University, USA) Nina Gehrer (University of Tübingen, Germany) Gaze Anal[.]
Department of Psychology Tutorial 3: Gaze Analytics Pipeline Andrew Duchowski (Clemson University, USA) Nina Gehrer (University of Tübingen, Germany) 14.06.2018 – ETRA Tutorials Gaze Analytics Pipeline: Origins • What is it? - series of Python scripts followed by analysis in R - goal: automation • How did it start, evolve? - ETH Winter School 2016 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze Analytics Pipeline: Ontology • Where does it fit? - Note quite PyGaze (www.pygaze.org) | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze Analytics Pipeline: Ontology • Where does it fit? - Note quite eyetrackingR (www.eyetracking-r.com) | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze Analytics Pipeline: Objectives • How does it work? - key goals: visualization and analysis see Gehrer et al., [2018], Gaze Analytic Methods in Clinical Psychology, this ETRA, Sunday 14:35 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze Analytics Pipeline: Objectives • How does it work? - key goals: visualization and analysis - like R’s tidyverse, sort of - idea is the same: import data, tidy, visualize, collate, analyze - each step a different Python script | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze Analytics Pipeline: Objectives • How does it work? - key goals: visualization and analysis | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Time Schedule 13:30 – 15:30: • Experiment Overview & Introduction to file system Structure - Starting point, objectives and preparations • Gaze analytics pipeline overview - Running the python scripts • Traditional gaze analytics (part 1) - Working with R scripts 15:30 – 16:00: Coffee break 16:00 – 17:00: • Traditional gaze analytics (part 2) & advanced gaze analytics | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Emotion categorization paradigm | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Emotion categorization paradigm The trial starts when the fixation cross is fixated for 300ms | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: microsaccades • Histograms 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: microsaccades • Without microsaccades within 20ms after fixation start | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: transition count • Transition matrix overall and number of transitions per condition § see Krzys’ talk this ETRA, Sunday 15:10 Number of transitions between AOIs Emotion categorization angry disgusted fearful happy neutral sad surprised Emotion category | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: transition entropy • Want to compare transition matrices between conditions angry disgusted fearful happy | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A neutral sad surprised © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: transition entropy • Normalized transition entropy • Higher entropy means “surprise!” 0.6 0.4 0.0 0.2 Transition Entropy 0.8 1.0 Emotion categorization task angry disgusted fearful happy neutral sad surprised Emotion Category | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: transition entropy • Normalized transition entropy • Higher entropy means “surprise!” • Stationary entropy: long run 0.6 0.4 0.0 0.2 Transition Entropy 0.8 1.0 Emotion categorization task angry disgusted fearful happy neutral sad surprised Emotion Category | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Transition entropy: example • Predicting the weather: - if rainy, probability it remains rainy is 51 - if sunny, 21% chance it become cloudy | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Transition entropy: example • Predicting the weather: - if rainy, probability it remains rainy is 51 - if sunny, 21% chance it become cloudy | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Transition entropy: example • Predicting the weather: - if rainy, probability it remains rainy is 51 - if sunny, 21% chance it become cloudy | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Transition entropy: example • Predicting the weather: - if rainy, probability it remains rainy is 51 - if sunny, 21% chance it become cloudy • Transition matrix gives likelihood of next period • What about next two periods? 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Stationary entropy: example • What about next two periods? - if it’s rainy today, chance it will be sunny in days is 25 • Similarly, probability in the long is 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Stationary entropy: example • In the long run - steady-state (stationary) transition probabilities converge - steady-state vector is eigenvector of with eigenvalue 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Stationary entropy: transition vs stationary entropy? • Ultimately, not super certain of stationary entropy’s utility • Because: Ht = AAACR3icbVC7TsMwFHXKq5RXgJHFokJioUoqJLogVWLpWAR9SE2JHNdp3TpOZDtIlZW/Y2Fl4xdYGECIEbfpAC1XsnR8zrnX1ydIGJXKcV6twtr6xuZWcbu0s7u3f2AfHrVlnApMWjhmsegGSBJGOWkpqhjpJoKgKGCkE0xuZnrnkQhJY36vpgnpR2jIaUgxUoby7YeGr+A1vPBCgbB2M83ioV/VMss8mUa+ph7lXoTUCCOm7wybUJ/m0nhJSox7nHn5gPyS+XbZqTjzgqvAXYAyWFTTt1+8QYzTiHCFGZKy5zqJ6mskFMWMZCUvlSRBeIKGpGcgRxGRfT3PIYNnhhnAMBbmcAXn7O8OjSIpp1FgnLO95bI2I//TeqkKa31NeZIqwnH+UJgyqGI4CxUOqCBYsakBCAtqdoV4hEyiykRfMiG4y19eBe1qxXUq7u1luV5bxFEEJ+AUnAMXXIE6aIAmaAEMnsAb+ACf1rP1bn1Z37m1YC16jsGfKlg/EIm1DQ== X X ⇡i pij log2 pij log2 s i2S Hs = j2S AAACFnicbZDLSsNAFIYn9VbrLerSzWAR3LQkItiNUHDTZUV7gSaEyXTSDp1MwsxEKCFP4cZXceNCEbfizrdxkmahrT8MfPznHOac348Zlcqyvo3K2vrG5lZ1u7azu7d/YB4e9WWUCEx6OGKRGPpIEkY56SmqGBnGgqDQZ2Tgz27y+uCBCEkjfq/mMXFDNOE0oBgpbXlmo+NJeA0bjkxCL6UO5U6I1BQjlt5lmRNTjzosmqQFZZ5Zt5pWIbgKdgl1UKrrmV/OOMJJSLjCDEk5sq1YuSkSimJGspqTSBIjPEMTMtLIUUikmxZnZfBMO2MYREI/rmDh/p5IUSjlPPR1Z76zXK7l5n+1UaKClptSHieKcLz4KEgYVBHMM4JjKghWbK4BYUH1rhBPkUBY6SRrOgR7+eRV6F80batp317W260yjio4AafgHNjgCrRBB3RBD2DwCJ7BK3gznowX4934WLRWjHLmGPyR8fkD78if3A== X ⇡i log ⇡i i2S Ht H s AAAB83icbVBNS8NAEJ34WetX1aOXxSJ4KokI9ljw0mMF+wFNCJvtpl262cTdiVBK/4YXD4p49c9489+4bXPQ1gcDj/dmmJkXZVIYdN1vZ2Nza3tnt7RX3j84PDqunJx2TJprxtsslanuRdRwKRRvo0DJe5nmNIkk70bju7nffeLaiFQ94CTjQUKHSsSCUbSS3wyR+JI/kmZowkrVrbkLkHXiFaQKBVph5csfpCxPuEImqTF9z80wmFKNgkk+K/u54RllYzrkfUsVTbgJpoubZ+TSKgMSp9qWQrJQf09MaWLMJIlsZ0JxZFa9ufif188xrgdTobIcuWLLRXEuCaZkHgAZCM0ZyokllGlhbyVsRDVlaGMq2xC81ZfXSee65rk17/6m2qgXcZTgHC7gCjy4hQY0oQVtYJDBM7zCm5M7L86787Fs3XCKmTP4A+fzBwbPkPs= transition entropy is always smaller • Long-term distribution of transitions is expected to be more uniform 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: where to go from here? • Important to remember what the pipeline offers: metrics • Which metrics to use will depend on study hypothesis • General strategy “recipe” for controlled experiments: - formulate hypothesis § don’t start with “I wonder what would happen if ” § start with “I bet this would happen if ” - design experiment (e.g., within-, between-subjects) choose metrics § gaze metrics (process metrics) often supplement performance metrics - choose analytical tools (stats, e.g., ANOVA, something else) • Can exploratory research or pilot studies beforehand 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A © U n iv e rsity o f T u e b in g e n Gaze analytics pipeline: write paper • Remember analytics pipeline is meant to help automate analysis Implementing Innovative Gaze Analytic Methods in Clinical Psychology • Once that’s done, write the paper § see Nina’s talk this ETRA, Sunday 14:35 • This too has a basic “recipe”: - abstract, intro, background - hypothesis § recent trend is to register this a priori - methodology § design, stimulus, apparatus, procedure, participants - results discussion conclusions 1 | D u ch o w ski & G e h re r, G a z e A n a lytic s P ip e lin e – T u to ria l, E T R A A Study on Eye Movements in Antisocial Violent O�enders Nina A Gehrer Michael Schönenberg∗ Clinical Psychology & Psychotherapy University of Tübingen [nina.gehrer,michael.schoenenberg] @uni-tuebingen.de Andrew T Duchowski† Visual Computing Clemson University duchowski@clemson.edu Krzysztof Krejtz Psychology Department SWPS University of Social Sciences & Humanities kkrejtz@swps.edu.pl ABSTRACT A variety of psychological disorders like antisocial personality disorder have been linked to impairments in facial emotion recognition Exploring eye movements during categorization of emotional faces is a promising approach with the potential to reveal possible di�erences in cognitive processes underlying these de�cits Based on this premise we investigated whether antisocial violent o�enders exhibit di�erent scan patterns compared to a matched healthy control group while categorizing emotional faces Group di�erences were analyzed in terms of attention to the eyes, extent of exploration behavior and structure of switching patterns between Areas of Interest While we were not able to show clear group di�erences, the present study is one of the �rst that demonstrates the feasibility and utility of incorporating recently developed eye movement metrics such as gaze transition entropy into clinical psychology The ability to decode nonverbal social information in order to infer the emotional state of an interaction partner is crucial for e�ective social interaction Accordingly, individuals are able to quickly and e�ciently identify emotional expressions from speci�c facial cues [Smith et al 2005; Tracy and Robins 2008] These cues are similar across cultures, at least for the six basic emotions, i.e., anger, disgust, fear, happiness, sadness, and surprise [Ekman 1999; Ekman and Friesen 1971] The accurate interpretation of emotional expressions is based on the processing of relevant regions of the face and directing visual attention to them (e.g., wide-open fearful eyes or smiling happy mouth) [Eisenbarth and Alpers 2011; Schurgin et al 2014] Thus, tracking eye movements while viewing emotional faces is a promising approach to gain insight into the processes underlying categorization of emotions In clinical research, eye tracking can be a useful tool to explore deviations in scanning patterns that could account for emotion recognition impairments associated with psychological disorders Impairments in facial a�ect recognition have been linked to the development and maintenance of various psychological disorders including autism [Uljarevic and Hamilton 2013], depression [Dalili et al 2015], anxiety disorders [Demenescu et al 2010], schizophrenia [Kohler et al 2009], attention-de�cit hyperactivity disorder [Bora and Pantelis 2016], and antisocial personality disorder (ASPD) and psychopathy [Dawel et al 2012; Marsh and Blair 2008] The majority of clinical studies exploring eye movements while viewing faces does not tap the potential of the myriad analytical methods available Although analysis of dwell time or number of �xations to certain Areas of Interest (AOIs) can yield interesting �ndings, an inclusion of more innovative and complex analytical methods (e.g., sequential analysis of eye movements) may add valuable information Here, we present an analysis of scan patterns while viewing faces including widely-used standard eye movement parameters (e.g., total dwell time) as well as more recently developed metrics such as gaze transition entropy [Krejtz et al 2015] Based on these measures, we investigate group di�erences in attention orienting to the eyes, extent of exploration behavior and structure of switching patterns between AOIs in antisocial violent o�enders (AVOs) and a matched healthy control group CCS CONCEPTS • Applied computing → Psychology; KEYWORDS eye tracking, antisocial o�enders, facial emotion recognition ACM Reference format: Nina A Gehrer, Michael Schönenberg, Andrew T Duchowski, and Krzysztof Krejtz 2018 Implementing Innovative Gaze Analytic Methods in Clinical Psychology In Proceedings of ETRA ’18: 2018 Symposium on Eye Tracking Research & Applications, Warsaw, Poland, June 14–17, 2018 (ETRA’18), pages https://doi.org/10.1145/3204493.3204543 ∗ This study was funded by the German Research Foundation (Scho 1448/3-1) † This work was supported by the U.S National Science Foundation (grant IIS-1748380) Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro�t or commercial advantage and that copies bear this notice and the full citation on the �rst page Copyrights for components of this work owned by others than ACM must be honored Abstracting with credit is permitted To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci�c permission and/or a fee Request permissions from permissions@acm.org ETRA’18, June 14–17, 2018, Warsaw, Poland © 2018 Association for Computing Machinery ACM ISBN 978-1-4503-5706-7/18/06 $15.00 https://doi.org/10.1145/3204493.3204543 INTRODUCTION © U n iv e rsity o f T u e b in g e n