MARIE · Multimodal Activity Recognition for Interactive Environments

With the increasingly ubiquitous use of computing devices in our environment, there is a clear need for new methods of human computer interaction. In order to support day-to-day human activity seamlessly, it is essential that computing systems have a sense of the user’s situation or context. As a consequence, activity recognition has become a central research challenge toward ambient intelligence. The proposed work aims to make three distinct contributions to the study of activity recognition: First, this project will introduce the use of eye movement patterns as a novel sensing modality for wearable activity recognition. Almost everything most of us do is guided by the use of our vision system., and Consequently, by studying what our eyes are doing, clues can be gathered as to what it is that we are doing, or intend to do. Beyond established gaze tracking, this work will look at the general patterns our eyes make during certain tasks and in certain situations, as context for interaction. Secondly, we aim to study robust activity recognition in a realistic, everyday setting. In a novel approach, we plan to exploit the synergy of information from wearable sensors - on user actions - with that of ambient sensors – on the environment being manipulated - and demonstrate this through recognition of a number of everyday activities. Thirdly, we will investigate how activity recognition can be used to infer user attention. Eye gaze is a good indicator of this (and has been used in the past), but this is not always a convenient modality to use. In this work we plan to assess the use of activity, in particular locomotion, recognition as an indicator of user attention. A final aspect of the proposal will be in the methodology applied to implementing and evaluating the various recognition problems encountered in this work – advancing the fellow's fundamental work on performance evaluation for activity recognition.