EgoSim
Vision as Proprioception
Estimating Humanoid Robot Pose from Video
Technical Report1Department of Computer Science, ETH Zürich2Department of Mechanical and Process Engineering, ETH Zürich
Abstract
Estimating a humanoid robot’s internal state from video has a wide range of real-world applications but remains unexplored. During interactions with a robot, its onboard state information may be unavailable due to privacy, ownership, or communication constraints. Meanwhile, online and AI-generated robot videos provide rich sources of data but typically lack corresponding joint configurations, limiting their direct use in robot learning. To address these issues, we present HumanoidPose, a new framework that treats vision as proprioception to recover full-body joint configurations in world space from video alone. Specifically, we combine pretrained human-pose features with robot kinematics in a unified model that predicts each embodiment’s joint configurations. We retarget human motions to sixteen different humanoid embodiments, enabling scalable learning entirely from synthetic data. To improve real-world generalization, we further leverage unlabeled online videos through self-training to develop a 2D robot-pose detector. In our evaluation, HumanoidPose improves joint-angle and joint-position accuracy over the evaluated baselines and reduces joint-angle error by 18–32% relative to the best-performing baseline on each of the three real humanoid platforms. We further show improved global localization and generalization to unseen embodiments. HumanoidPose enables visual understanding of other robots during interaction and extraction of robot motion from online and generated videos.
Video
Reference
Jiaxi Jiang, Matthias Heyrman, René Zurbrügg, Marco Hutter, and Christian Holz. Vision as Proprioception: Estimating Humanoid Robot Pose from Video. arXiv preprint arXiv:2610.25288 (2026).
BibTeX citation
@misc{jiang2026visionproprioceptionestimatinghumanoid, title={Vision as Proprioception: Estimating Humanoid Robot Pose from Video}, author={Jiaxi Jiang and Matthias Heyrman and René Zurbrügg and Marco Hutter and Christian Holz}, year={2026}, eprint={2610.25288}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2610.25288}, }
Video sources
Joint configurations are what robot learning consumes, and they are exactly what video does not carry. Reading the pose out of the pixels turns any footage into motion data.
Four sources in motion: the video above, the configuration HumanoidPose recovers below.
Method
A detector finds the robot and a square crop puts it at 55 % of the crop's height. The crop passes through an image and camera encoder pretrained on humans, kept frozen. A kinematics-conditioned decoder reads it together with joint descriptors from the robot's own description, and returns that robot's joint angles and root pose.
Because the robot description is an input rather than something baked into the weights, a robot the model has never seen is added by pointing at its kinematic description.
Training data
Rendered in Unreal Engine from human motion retargeted onto each robot, so every frame comes with the configuration that produced it. No real robot is used for training.
The modalities of a rendered frame. The model takes only the image; the rest ships with the corpus.
Detection
A detector trained only on renders misses much of what real video contains. Pseudo-labelling its own confident detections on unlabeled internet video closes most of that gap, without a single hand-drawn box.
One real clip through three detector stages: synthetic-only, then after each round of self-training.
The pseudo-labels themselves, round one against round two, on a public video.
Sixteen robots
The same checkpoint, conditioned on sixteen different kinematic descriptions. The robots differ by a factor of three in height and from 19 to 43 degrees of freedom. On synthetic sequences the ground truth is exact.
Input and recovered configuration for each of the sixteen embodiments.
Every method on a synthetic sequence.
Real robots
Three platforms were recorded while they logged their joint states. Nothing but the video enters the estimate; the encoder readings are only ever the reference.
Unitree G1, EngineAI PM01 and Fourier N1, with the configuration recovered from each frame.
Unitree G1, against its own joint encoders.
EngineAI PM01, walking into a deep squat.
Fourier N1.
Every evaluated method on one real clip, each rendered from its own prediction.
Internet videos
Footage that exists in quantity and carries no joint states at all. Reading the pose out of it turns a manufacturer’s promotional video into motion expressed in the robot’s own joints.
Six shots of one promotional video, input beside the recovered configuration.
Generated videos
No physical robot exists here, so no encoder reading could exist either. Conditioned on a kinematic description, the motion still comes back in the joints of a real platform.
Four generated videos, each conditioned on the kinematic description that fits it best.
