First-in-human Realtime AI-assisted Instrument Deocclusion during
Augmented Reality Robotic Surgery
Abstract
The integration of Augmented Reality (AR) into daily surgical practice
is withheld by the correct registration of pre-operative data. This
includes intelligent 3D model superposition whilst simultaneously
handling real and virtual occlusions caused by the AR overlay.
Occlusions can negatively impact surgical safety and as such deteriorate
rather than improve surgical care. Robotic surgery is particularly
suited to tackle these integration challenges in a stepwise approach as
the robotic console allows for different inputs to be displayed in
parallel to the surgeon. Nevertheless, real-time de-occlusion requires
extensive computational resources which further complicates clinical
integration. This work tackles the problem of instrument occlusion and
presents, to our best knowledge, the first-in-human on edge deployment
of a real-time binary segmentation pipeline during three robot-assisted
surgeries: partial nephrectomy, migrated endovascular stent removal and
liver metastasectomy. To this end, a state-of-the-art real-time
segmentation and 3D model pipeline was implemented and presented to the
surgeon during live surgery. The pipeline allows real-time binary
segmentation of 37 non-organic surgical items, which are never occluded
during AR. The application features real-time manual 3D model
manipulation for correct soft tissue alignment. The proposed pipeline
can contribute towards surgical safety, ergonomics and acceptance of AR
in minimally invasive surgery.