Multimodal Vision-based Avatar Face Reconstruction on Smart Glasses (in collaboration with Google XR)
Together with Research Scientists at Google Zürich, we will explore the future of facial expression tracking for AR/MR smart glasses and combine sensors inside the glasses (e.g., eye tracker, IMU, egocamera) with small additional cameras. The goal is to implement a new GenAI-based method (e.g., diffusion models) to generate facial expressions using input from the smart glasses alone. To achieve this, you will iterate together with a member of our team on a small capture setup that integrates smart glasses, additional cameras, and stationary cameras for reference recordings of facial expressions. This real-time reconstruction is intended to drive expressive 3D avatar animation that enables natural teleconferencing and presence in Augmented Reality.
