Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

Researchers developed a multimodal gesture recognition system using Apple Watch sensors and custom gloves for hands-free drone and robot control in hazardous environments. The framework achieves performance comparable to vision-based systems while being more computationally efficient and robust to environmental conditions.

• →New multimodal gesture recognition framework combines inertial data from Apple Watches with capacitive sensing from custom gloves for robot teleoperation.
• →System performs comparably to vision-based methods while reducing computational cost, model size, and training time.
• →Framework provides interpretability by quantifying individual modality contributions through log-likelihood ratio fusion.
• →New dataset includes 20 distinct gestures based on aircraft marshalling signals with synchronized sensor data.
• →Solution addresses limitations of vision-based systems including occlusions, lighting variations, and cluttered backgrounds.