Reliability of cortical activity during natural stimulation Supplementary Material Uri Hasson , Rafael Malach2, and David J Heeger3 Department of Psychology and the Neuroscience Institute, Princeton University Department of Neurobiology, Weizmann Institute for Science Department of Psychology and Center for Neural Science, New York University Supplementary Figure Computing response reliability to naturalistic stimulation Functional MRI data are acquired continuously while a subject watches (freeviewing) a movie, a segment from a movie, or an episode of a TV series The same procedure might be used to assess different types of media, including audiobooks, political speeches, pop songs, and other types of music The subject’s engagement monitoring galvanic can their skin be eye assessed by movements and responses (GSR) The analysis procedures comprise the following steps I Talairach alignment preprocessing images of and Anatomical the brain are acquired in addition to the fMRI scans The anatomical images of each subject’s brain are computationally registered to what is known as the Talairach coordinate system, so that corresponding regions of each brain are roughly aligned with one another There are several possible Page | alternatives to the Talairach coordinate space, including MNI space and surfacebased alignment, but the goal in each case is to ensure that the data from each subject are aligned to one another The fMRI data are aligned to the anatomical scans, and processed by temporal high-pass filtering to remove slow (low frequency) drift over time, 3D-motion correction to compensate for any residual head motion, and (optionally) spatial smoothing to overcome any residual misregistration between brains II Correlation analysis Intra-SC is computed, separately for each subject, as the correlation between responses to repeated presentations of the same stimulus InterSC is computed as the correlation between responses across subjects The correlations can be computed on a voxel-by-voxel basis for all pairs of subjects/repeated presentations, yielding a set of correlation coefficients for each voxel Another variation of the analysis is to first average across subjects within each of two or more groups before computing the correlations Supplementary Figure shows that both analyses yield similar results The inter- and intra-SC can also be computed between the response time courses from predefined regions of interest, instead of doing so separately for each voxel Finally, the correlations can also be computed within a sliding temporal window, yielding information about the temporal evolution of response reliability [1] The use of Pearson correlation is simple, but can be generalized to any measure of similarity (e.g., based on coherence [2], mutual information, principal components analysis [3], canonical correlation analysis, independent components analysis, etc.) The method can also be generalized to different types of neurophysiological (e.g., fMRI, EEG, PET, IRS, etc.), physiological (e.g., galvanic skin response), or behavioral (e.g., eye movements) signals III Statistical parametric mapping For each voxel, the average correlation coefficient (r) is calculated, after applying Fisher transformation Fisher transformation normalizes the distribution of the r values, which allows for testing whether the distribution of the pair-wise values is significantly different from zero (i.e., given that r values, without normalization, are constrained to the interval [-1,1] and hence are not normally distributed) Alternatively, a statistical threshold can be selected based on spurious correlations measured with different movie segments (as was done in Figure and Supplementary Figure 2) Statistical significance of the correlations can also be determined by utlizing a phase-randomization procedure in which the phase of the Page | Fourier transform of the time series is randomized without changing the Fourier power This is done separately for each voxel and each subject (or repeated presentation) and then the correlations are recomputed from the phase-randomized time series Repeating this many times (e.g., 1000 phase-randomizations) produces an empirical distribution of the correlation coefficients under the null hypothesis that there is no inter- nor intra-SC These inter- and intra-SC analysis methods differ from conventional fMRI data analysis methods in several ways In a conventional fMRI experiment, a time-series of brain activity images is collected while a stimulus or cognitive task is systematically varied Based on the experimental design, the experimenter specifies a model consisting of a set of predictors or explanatory variables The model formalizes hypotheses about expected changes of the fMRI signal to be compared with the measured response time course at each voxel (or ROI) during the experiment The inter- and intra-SC analyses circumvent the need to specify a model; there is no need for prior knowledge of the expected activity in any given brain area during a given stimulus or task Rather, these methods can be used to identify brain regions with reliable response time courses, within or across subjects, without the need for any prior assumptions or hypotheses about their functional properties Consequently, these methods are well-suited to naturalistic, continuous stimuli such as movies, music, etc Supplementary Figure Inter-subject correlation in brain activity Inter-SC maps are shown for 23 subjects who watched an episode of the Alfred Hitchcock Presents TV series Each panel corresponds to either the medial and lateral views of an inflated right (RH) hemisphere The inter-SC was computed in two distinct ways: The inter-SC was computed on a voxel-by-voxel basis for all pairs of subjects (253 pairs), yielding a set of correlation coefficients for each voxel We then calculated the average correlation coefficient (r) per voxel, after Page | applying Fisher transformation to normalize the individual correlation coefficients To ensure that these mean correlation values were not biased by outliers, we performed a second order t-test on the inter-SC values to confirm that the mean value was significantly different from zero These mean correlation values reflect the degree of similarity across subjects To set a statistical threshold, we also computed the inter-SC between the Hitchcock episode and the Washington Square park video (207 pair-wise comparisons) The inter-SC between the two independent movies was extremely low, as expected, and we chose a cutoff that was above the highest value exhibited by any voxel in this analysis A variation of the analysis was to first average across subjects within each of two groups before computing the inter-SC To produce the statistical maps, we split the 23 subjects (arbitrarily) into two independent groups (n=11 and n=12) We then averaged the time courses on a voxel-by-voxel basis across all subjects within each group after z-normalizing each individual time course This procedure averaged out individual differences, while enhancing common shared signals, independently for each group Finally, we computed the inter-SC between the average time courses To set a statistical threshold, we computed the inter-SC between the average response time courses for the Hitchcock episode (n=11, n=12) and the average response time courses for the Washington Square park video (n=9) Again the inter-SC between the two independent movies was extremely low, and we chose a cutoff that was above the highest value exhibited by any voxel in this analysis Both analyses yielded similar results Note, however, that averaging the signal across subjects attenuated the noise and individual variability in the responses This is why the correlation threshold was lower when computed for each pair of subjects than when computed after averaging across subjects Page | Supplementary Figure Selectivity of responses in different brain regions after removing a spatially global reference signal Inter-region correlations and within-region intersubject correlations for 15 example brain areas (same format as Figure 3B) were The correlations computed removing a after spatially global ‘reference’ signal from the response time courses in each brain region The global reference was defined as the mean time course averaged across all brain areas exhibiting high inter-SC, and was removed by orthogonal projection This was done to eliminate any response components that were shared by all regions (such as might be expected to be evoked by arousal) The removal of the common signal, as expected, enhanced the response selectivity (upper panel, compare this inter-region correlation matrix and the interregion correlation matrix in Figure 3B) The high within-region inter-SC (bottom panel, compare with the with-region inter-subject correlations in Figure 3B), even after removing the global reference signal, demonstrates that response reliability within each brain area is not entirely driven by non-specific, spatially global effects (e.g., arousal) Previous fMRI studies with conventional experimental protocols have likewise removed reference response time courses from each cortical sub-region of interest [4-7] Like simple subtraction of response time courses from different cortical subregions, the orthogonal projection applied here is based on the assumption that regionally specific response components and global (i.e., spatially nonspecific) components of cortical activity superimpose linearly It has been shown that apparently spontaneous fluctuations of cortical activity [8] superimpose linearly with event-related responses [9], and that the removal of such global or nonspecific components improves the signal-to-noise ratio in fMRI measurements [5, 7] Page | Supplementary Figure Individual differences Idiosyncratic responses in posterior superior temporal sulcus (pSTS), evoked by the Sergio Leone movie, in each of four subjects Same format as Figure Page | References Hasson, U., et al., Neurocinematics: the Neuroscience of Film Projections, 2008 2(1): p 1-26 Hasson, U., et al., A hierarchy of temporal receptive windows in human cortex J Neurosci, 2008 28(10): p 2539-50 Hanson, S.J., A.D Gagliardi, and C Hanson, Solving the brain synchrony eigenvalue problem: conservation of temporal dynamics (fMRI) over subjects doing the same task J Comput Neurosci, 2008 Donner, T.H., et al., Opposite neural signatures of motion-induced blindness in human dorsal and ventral visual cortex J Neurosci, 2008 28(41): p 10298-310 Fox, M.D., et al., The human brain is intrinsically organized into dynamic, anticorrelated functional networks Proc Natl Acad Sci U S A, 2005 102(27): p 9673-8 Meng, M., D.A Remus, and F Tong, Filling-in of visual phantoms in the human brain Nat Neurosci, 2005 8(9): p 1248-54 Sylvester, R., et al., Visual FMRI responses in human superior colliculus show a temporal-nasal asymmetry that is absent in lateral geniculate and visual cortex J Neurophysiol, 2007 97(2): p 1495-502 Leopold, D.A and N.K Logothetis, Spatial patterns of spontaneous local field activity in the monkey visual cortex Rev Neurosci, 2003 14(1-2): p 195-205 Arieli, A., et al., Dynamics of ongoing activity: explanation of the large variability in evoked cortical responses Science, 1996 273(5283): p 1868-71 Page | ... (i.e., spatially nonspecific) components of cortical activity superimpose linearly It has been shown that apparently spontaneous fluctuations of cortical activity [8] superimpose linearly with... Spatial patterns of spontaneous local field activity in the monkey visual cortex Rev Neurosci, 2003 14(1-2): p 195-205 Arieli, A., et al., Dynamics of ongoing activity: explanation of the large variability... time-series of brain activity images is collected while a stimulus or cognitive task is systematically varied Based on the experimental design, the experimenter specifies a model consisting of a set of