Studying sentence generation during scene-viewing in Hindi with eye-tracking

Abstract

Eye movements and cognition: Eye movements have long been considered useful measures of different types of cognitive operations (see Wade & Tatler (2005) for an historical introduction). Since eye movements indicate the focus of visual attention, measurements of saccades and fixations can reveal the nature of transient mental representations that are formed during cognitive processing. For example, we listen and look at objects in the real world in order to comprehend discourse. Thus, measuring the allocation of attention during such tasks can reveal aspects of the underlying cognitive processes. Even if eye movements do not directly reflect any cognitive processing, they are nevertheless affected by the nature of the task and goals of the perceiver during that task (Cooper, 1974; Tanenhaus et al., 1995; Altmann & Kamide, 2009; Anderson & Spivey, 2009; Mishra, 2009). Eye movements also may reflect anticipatory processing in several cognitive tasks (see Mishra & Marmalejo-Ramos (2010) and Huettig, Mishra & Olivers (2012) for reviews). Viewers tend to programme anticipatory eye movements to objects in the visual field whose names or other semantic features match the current linguistic input (Huettig et al., 2006). When objects in the visual field that are relevant for current action and match to the speech input, one finds tight correlations between fixations and cognitive processing (Allopenna, Magnuson & Tanenhaus, 1998; see also Huettig, Rommers & Meyer, 2011). Thus, eye movements recorded during the simultaneous processing of language and visual stimuli reflect the dynamic nature of interactions between both visual and linguistic information (Knoeferle & Crocker, 2006; Mishra & Marmalejo-Ramos, 2010). The analysis of fixations and saccades can reveal finer aspects of cognitive processing including those related to visual, attentional and memory processes. For example, the recent use of eye-tracking in the study of online interaction between linguistic and visual representations has provided key insights into the nature of multimodal interactions (Huettig, Mishra & Olivers, 2012). Eye-movement data, therefore, have the potential to provide fine-grained data on temporal aspects of cognitive processing.