loading page

First-in-human Realtime AI-assisted Instrument Deocclusion during Augmented Reality Robotic Surgery
  • +13
  • Jasper Hofman,
  • Pieter De Backer,
  • Ilaria Manghi,
  • Jente Simoens,
  • Ruben De Groote,
  • Hannes Van Den Bossche,
  • Mathieu D'Hondt,
  • Tim Oosterlinck,
  • Julie Lippens,
  • Charles Van Praet,
  • Federica Ferraguti,
  • Charlotte Debbaut,
  • Zhijin Li,
  • Oliver Kutter,
  • Alex Mottrie,
  • Karel Decaestecker
Jasper Hofman
Orsi Academy
Author Profile
Pieter De Backer
Orsi Academy

Corresponding Author:[email protected]

Author Profile
Ilaria Manghi
University of Modena and Reggio Emilia Department of Engineering Enzo Ferrari
Author Profile
Jente Simoens
Orsi Academy
Author Profile
Ruben De Groote
Orsi Academy
Author Profile
Hannes Van Den Bossche
az West
Author Profile
Mathieu D'Hondt
AZ Groeninge
Author Profile
Tim Oosterlinck
KU Leuven
Author Profile
Julie Lippens
Ghent University Faculty of Medicine and Health Sciences
Author Profile
Charles Van Praet
University Hospital Ghent
Author Profile
Federica Ferraguti
University of Modena and Reggio Emilia Department of Engineering Enzo Ferrari
Author Profile
Charlotte Debbaut
Ghent University, Faculty of Engineering and Architecture, and CRIG
Author Profile
Zhijin Li
NVIDIA Corp
Author Profile
Oliver Kutter
NVIDIA Corp
Author Profile
Alex Mottrie
Orsi Academy
Author Profile
Karel Decaestecker
University Hospital Ghent
Author Profile

Abstract

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre-operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real-time de-occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to our best knowledge, the first-in-human on edge deployment of a real-time binary segmentation pipeline during three robot-assisted surgeries: partial nephrectomy, migrated endovascular stent removal and liver metastasectomy. To this end, a state-of-the-art real-time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The application features real-time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics and acceptance of AR in minimally invasive surgery.
08 Nov 2023Submitted to Healthcare Technology Letters
10 Nov 2023Submission Checks Completed
10 Nov 2023Assigned to Editor
15 Nov 2023Reviewer(s) Assigned
17 Nov 2023Review(s) Completed, Editorial Evaluation Pending
21 Nov 2023Editorial Decision: Accept