An Early Warning System Based on Visual Feedback for Light-Based Hand Tracking Failures in VR Head-Mounted Displays
Abstract
State-of-the-art Virtual Reality (VR) Head-Mounted Displays (HMDs) enable users to interact with virtual objects using their hands via built-in camera systems. However, the accuracy of the hand movement detection algorithm is often affected by limitations in both camera hardware and software, including image processing and machine learning algorithms used for hand skeleton detection. In this work, we investigated a visual feedback mechanism to create an early warning system that detects hand skeleton recognition failures in VR HMDs and warns users in advance. We conducted two user studies to evaluate the system's effectiveness. The first study involved a cup stacking task, where participants stacked virtual cups. In the second study, participants performed a ball sorting task, picking and placing colored balls into corresponding baskets. During both studies, we monitored the built-in hand tracking confidence of the VR HMD system and provided visual feedback to warn users when tracking confidence was low. The results showed that warning users before the hand tracking algorithm fails improved the system's usability while reducing frustration. The impact of our results extends beyond VR HMDs; any system that uses hand tracking, such as robotics, can benefit from this approach.
Methodology
We evaluated the early-warning visualization across two VR hand-interaction studies. The first used a cup-stacking task, while the second asked participants to sort colored balls into matching baskets. In both studies, the system monitored the headset's hand-tracking confidence and displayed visual feedback before tracking quality fell low enough to cause a failure.
Study 2 ball-sorting task setup with early warning visual feedback indicators.
Comparative view of normal vs. low tracking confidence during ball manipulation.
Visual feedback indicator triggering as hand approaches low confidence zone.
Dynamic color shift feedback warning user before tracking failure occurs.
Hand tracking skeleton failure state detected during VR HMD interaction.
Results
Providing an advance warning improved perceived usability and reduced frustration when compared with experiencing hand-tracking failures without warning. The results also characterize how the warning affected task-completion time, System Usability Scale scores, and NASA-TLX workload across the study conditions.
Task completion time comparison across warning conditions in Study 1 (cup stacking).
Task completion time comparison across warning conditions in Study 2 (ball sorting).
System Usability Scale (SUS) scores demonstrating improved usability with early warning.
NASA-TLX workload subscales showing significant reduction in user frustration.
Presentation
BibTeX
@article{bashar2025early,
author = {Bashar, Mohammad Raihanul and Batmaz, Anil Ufuk},
title = {An Early Warning System Based on Visual Feedback for Light-Based Hand Tracking Failures in VR Head-Mounted Displays},
journal = {IEEE Transactions on Visualization and Computer Graphics},
pages = {1--11},
year = {2025},
doi = {10.1109/TVCG.2025.3549544}
}