VisualNarrative · From Meaning to Perception: How narratives shape visual processing

Humans constantly generate expectations of what will happen in the future. Such expectations allow us to quickly absorb incoming data. For example, the visual system takes advantage of the world’s relative stability over space and time to anticipate upcoming input. However, most of what we currently know about predictive visual processing comes from studies using arbitrary associations between simple stimuli, such as gratings or abstract objects, which do not reflect the richness of daily life events. It is largely unknown how more complex, meaning-based expectations influence visual processing. To overcome this barrier, the proposed project will use an ecologically valid paradigm, using picture books, and investigate how such more complex, narrative-based expectations shape visual processing. I hypothesise that narratives will elicit strong expectation signals that impact even the earliest levels of sensory processing. To test this hypothesis, I will first quantify narrative-based expectations derived from the picture stories using state-of-the-art computational modelling. Then, I will assess whether these expectations shape the processing in visual cortical regions of adult human volunteers, using neuroimaging. I will inspect whether such signals become anticipatory as viewers become familiar with the narrative – in line with the idea that upon construction of a world model, the brain predicts upcoming stimuli. Moreover, I will also investigate what brain regions shape such learning of narratives. In a separate eye-tracking experiment, I will explore how narratives influence gaze behaviour, providing another assay into how meaning percolates to visual processing. This project will allow me to conduct creative, original and important research, which will propel my scientific career. More broadly, these lines of inquiry will push the envelope of understanding of human perception and learning, bridging from higher-order cognition to low-level visual processing.