MULTISENSE · The merging of the senses: understanding multisensory experience

The integration of information across different sensory channels is a fundamental problem that we have to solve in every moment of daily life. We will study four key aspects of multisensory processing that are currently unresolved: (1) the role of dynamic binding in multisensory experience; (2) the role of action and sensorimotor contingencies in multisensory processing; (3) the modulation of multisensory experience by top-down factors; (4) the adaptivity of these three mechanisms in conditions of altered multisensory experience. We will apply psychophysiological, behavioural and physiological approaches in studies on human subjects and combine these with advanced signal analysis approaches and computational modelling. The project objectives are: to test the role of dynamic crossmodal binding in multisensory processing; to address multisensory processing in rarely studied modalities such as pain and olfaction; to test the role of active sensorimotor exploration for multisensory experience; to study the role of top-down factors such as attention, or semantic knowledge; to address pathophysiological implications in congenitally blind subjects and synesthetes; to manipulate crossmodal interactions by application of multi-site TMS; to investigate dynamic crossmodal interactions in subjects undergoing training with an innovative sensory substitution device; to derive computational models of crossmodal integration to account for the empirical findings; and to explore the potential of our approach for enhancing human performance in application scenarios that exploit multisensory experience. Taken together, our proposal constitutes a highly innovative and systematic approach that will foster our understanding of the nature and mechanisms of unified multisensory experience.