Nail it! Vision-based Drift Correction For Accurate Mixed Reality Surgical Guidance

Abstract

Purpose: Mixed reality-guided surgery through head-mounted displays (HMDs) is gaining interest among surgeons. However, capitalizing on the benefits of mixed reality environments for surgical guidance requires precise tracking of the headset relative to the environment. Current image-based techniques for tracking the HMD suffer from millimeter- to centimeter-scale drift of the virtual space along with its contents, resulting in misaligned visualization of registered overlays. Methods and workflows capable of automatically correcting for drift after patient registration are essential to assuring accurate execution of surgical plans.
Methods: We present a mixed reality surgical navigation workflow that continuously corrects for drift after patient registration using only image-based methods. We demonstrate its feasibility and capabilities using the Microsoft HoloLens 2 on glenoid pin placement in total shoulder arthroplasty. A phantom study was conducted involving five users (two surgeons, two physical assistants, one medical device consultant) with each user placing pins on six glenoids of different deformity. A cadaver study was performed by an attending surgeon to validate the translation of our work to a real surgical scenario.
Results: In both the phantom and cadaver studies, all users were satisfied with the registration overlay before proceeding to drilling the pin. Postoperative CT scans show 1.5mm error in entry point deviation and 2.4° error in pin orientation on average in the phantom study and 2.5 mm and 1.5° error in the cadaver study. A trained user takes around 90 seconds to complete the workflow.
Conclusion: Our findings suggest that image-based drift correction can provide mixed reality environments precisely aligned with patient anatomy, enabling pin placement with consistently high accuracy. These techniques constitute a next step toward purely image-based mixed reality surgical guidance, without the need for patient markers or external tracking hardware.