Conference Agenda

Reliable tracking of physical objects and lower-body movements remains challenging in Mixed Reality (MR), particularly when relevant elements fall outside the headset field of view. This paper presents a low-cost hybrid sensing prototype that combines a Meta Quest 3 headset with an external Intel RealSense D415 RGB-D camera to track sports balls and lower-body joints in a shared MR coordinate system. The system uses YOLO-based models for object detection and lower-body pose estimation, while a QR-code-based calibration procedure estimates the rigid transformation between the external camera frame and the MR world frame. Detected objects and body joints are transmitted wirelessly to Unity and rendered as spatially aligned virtual overlays. Preliminary evaluation shows a spatial alignment error of approximately 0.5–1.5 cm within a 3 m range, with frame rates of about 31 FPS for object tracking and 27 FPS when lower-body pose estimation is added. A proof-of-concept application, named Hybrid Football, demonstrates how tracked physical objects and lower-body movements can trigger virtual interactions in MR. The results indicate the feasibility of hybrid external sensing for physically grounded MR interaction, while highlighting current limitations related to occlusions and fast-motion robustness.