SkillSpotter
Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos
ECCV 2026Department of Computer Science, ETH Zürich, Switzerland
Abstract
To enable personalized, real-time coaching using Augmented Reality glasses or fixed camera setups in domains such as sports, cooking, or music, a system must understand not just what a person does, but how well they execute an activity. In an ego-exo video setting, this requires simultaneously detecting individual skilled actions and classifying each as correct or needing improvement, which Ego-Exo4D’s proficiency demonstration benchmark formalized. We first adapt seven state-of-the-art temporal action detection architectures to this task, extend the evaluation protocol to disentangle detection from grading, and show that existing methods grade near-randomly. We then introduce SkillSpotter, a pose-aware multi-view architecture that jointly detects and grades skilled actions through three task-specific modules: (1) adaptive temporal suppression to handle the varying density of skilled actions across diverse activities, (2) gated 3D body pose fusion to leverage body kinematics as a complementary signal to visual features, and (3) bidirectional cross-view attention to combine ego and exo views effectively. SkillSpotter improves class-specific mAP from 12.40 to 21.82 (+76%) and balanced accuracy from 55.99% to 60.40% over the best baseline. SkillSpotter’s modules transfer to other temporal action detection models with consistent gains, and our method generalizes beyond Ego-Exo4D to HoloAssist.
Reference
Bjoern Braun and Christian Holz. SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos. In European Conference on Computer Vision 2026 (ECCV).
BibTeX citation
@article{braun2026skill, title={SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos}, author={Braun, Bj{\"o}rn and Holz, Christian}, journal={arXiv preprint arXiv:2606.31127}, year={2026} }
Heading
Figure 2. SkillSpotter’s architecture for skilled action detection and quality grading (Ego+Exos setting shown). SkillSpotter takes pre-extracted Omnivore features from ego and exo videos and estimated 3D body pose as input. Video features pass through an ActionFormer backbone and pose is encoded by a separate convolutional backbone. The output is a set of timestamped detections, each classified as good execution or tip for improvement. For single-view settings, the cross-view module is removed.
Figure 3. Cumulative distribution of temporal distances between consecutive ground-truth events, shown per scenario. We see that the event density varies strongly across scenarios: in Basketball, over 80% of events have a neighbor within 0.5 s, whereas in Music fewer than 20% do. This variation means no single fixed suppression radius can suit all scenarios, motivating our learned adaptive suppression.
Figure 4: (a) On Ego-Exo4D, the temporal distance between ground-truth events varies strongly across scenarios. (b) SkillSpotter learns to match this variation through our proposed adaptive temporal suppression module, which learns to predict scenario-specific suppression radii. Dense scenarios, such as Basketball, receive smaller radii, sparse ones, such as Music, larger radii (Spearman ρ=0.83, p=0.010).
