CAPREAL · Performance Capture of the Real World in Motion

Computer graphics technology for realistic rendering has improveddramatically; however, the technology to create scene models to be rendered,e.g., for movies, has not developed at the same pace. In practice, the stateof the art in model creation still requires months of complex manual design,and this is a serious threat to progress. To attack this problem, computergraphics and computer vision researchers jointly developed methods thatcapture scene models from real world examples. Of particular importance isthe capturing of moving scenes. The pinnacle of dynamic scene capturetechnology in research is marker-less performance capture. From multi-viewvideo, they capture dynamic surface and texture models of the real world.Performance capture is hardly used in practice due to profound limitations:recording is usually limited to indoor studios, controlled lighting, anddense static camera arrays. Methods are often limited to single objects, andreconstructed shape detail is very limited. Assumptions about materials,reflectance, and lighting in a scene are simplistic, and we cannot easilymodify captured data.In this project, we will pioneer a new generation of performance capturetechniques to overcome these limitations. Our methods will allow thereconstruction of dynamic surface models of unprecedented shape detail. Theywill succeed on general scenes outside of the lab and outdoors, scenes withcomplex material and reflectance distributions, and scenes in which lightingis general, uncontrolled, and unknown. They will capture dense and crowdedscenes with complex shape deformations. They will reconstruct convenientlymodifiable scene models. They will work with sparse and moving sets ofcameras, ultimately even with mobile phones. This far-reaching,multi-disciplinary project will turn performance capture from a researchtechnology into a practical technology, provide groundbreaking scientificinsights, and open up revolutionary new applications.