EgoPHI

Why force?

Contact tells us where a hand and object interact. Force provides an additional layer of information about how that interaction is taking place. Two grasps can touch nearly the same parts of an object while applying very different forces, corresponding to different ways of holding, pressing, stabilizing, or controlling it.

By estimating these forces directly from egocentric vision, EgoPHI takes a step toward physical interaction understanding without requiring tactile sensors at test time. Such information could eventually support applications including robot learning from human demonstrations, embodied scene understanding, and assistive systems that need to reason about how people manipulate objects.

Method overview

EgoPHI takes an egocentric RGB image together with the known 3D geometry of the manipulated object. Rather than reasoning about the image and the 3D meshes independently, it brings visual and geometric information into a shared representation and explicitly models how the two hands and object relate to one another in 3D.

A key challenge is that the object is often heavily occluded by the hands. EgoPHI therefore first refines the object’s estimated 3D pose, then uses the aligned hand and object meshes to predict where contact occurs and the force acting at each mesh vertex.

Figure 2: EgoPHI’s three-stage pipeline: (1) visual and geometric feature extraction with cross-modal fusion, (2) object pose estimation, and (3) contact and force estimation. Our novel Graph-Based Interaction Blocks perform the core 3D reasoning by encoding intra-mesh structure and inter-mesh relationships between the hands and object. These blocks represent the 3D pose and spatial configuration of each mesh relative to the others. We thus explicitly model proximity and geometric interaction, which is critical for inferring physically grounded forces.

Qualitative results

We evaluate EgoPHI on the ARCTIC dataset, augmented with our physics-simulated force supervision. The interactive 3D visualizations below show our predicted hand and object forces, as well as the full interaction motion, alongside the corresponding egocentric RGB video for four articulated rigid objects from ARCTIC.

EgoPHI predicts directly in 3D rather than a image-space heatmap. The widgets below allow comparing where EgoPHI predicts force on the hands with the corresponding regions on the object throughout the interaction.

Quantitative results

We quantitatively evaluate EgoPHI against image-space (PressureVision) and 3D mesh-level (HACO) baselines across both in-distribution (ARCTIC) and out-of-distribution (H2O) settings. Thus, H2O tests whether EgoPHI’s learned interaction reasoning transfers to different objects, scenes, and capture conditions.

For force error, lower MAE and RMSE are better. For the overlap- and contact-based metrics, higher values indicate better agreement with the ground truth.

Table 1: Comparison to PressureVision, a specialized 2D hand force estimation model, using 2D metrics computed on our projected 3D hand predictions.

Table 2: Contact and force estimation for HACO (pretrained on 14 datasets, including ARCTIC and H2O) and our models (trained only on the ARCTIC training set, excluding s05), comparing the full model with the version without the Iterative Refinement Module (IRM).

Overall, EgoPHI improves force estimation on ARCTIC and remains effective when evaluated on the unseen H2O dataset. 3D alignment matters for this task and improving the estimated relationship between the hands and object leads to better localization of contact and force.

Real-world evaluation

EgoPHI’s training supervision comes from physics simulation, so an important question is whether the learned predictions correspond to forces in the real world. To test this sim-to-real transfer, we built two instrumented objects: a cube and a cylinder, which allows us to measure where a person touches the surface and estimate the applied force.

The objects use illuminated acrylic: when a finger presses against the surface, light escapes into the skin, and stronger pressure produces a stronger optical response. A camera inside the setup records this signal, giving us dense measurements that can be compared directly with EgoPHI’s predictions. We recorded interactions from eight participants performing a range of touches, presses, grips, and full-hand grasps.

Quantitative sim-to-real results

Despite being trained using simulated force supervision rather than measurements from these objects, EgoPHI recovers the main real-world contact regions and meaningful force magnitudes. This provides evidence that the model is learning interaction cues that transfer beyond the simulated training setup.

Table 3: Real-world evaluation for object contact and force, trained on simulation alone. EgoPHI recovers the main spatial locations while achieving relatively low force errors on both objects.