The Segment Anything Model (SAM) is a powerful vision foundation model that is revolutionizing the traditional paradigm of segmentation. Despite this, reliance on prompting each frame and large computational cost limit its usage in robotically assisted surgery. Applications, such as augmented reality guidance, require little user intervention along with efficient inference to be usable clinically. In this study, we address these limitations by adopting lightweight SAM variants to meet the efficiency requirement and employing fine-tuning techniques to enhance their generalization in surgical scenes. Recent advancements in Tracking Any Point (TAP) have shown promising results in both accuracy and efficiency, particularly when points are occluded or leave the field of view. Inspired by this progress, we present a novel framework that combines an online point tracker with a lightweight SAM model that is fine-tuned for surgical instrument segmentation. Sparse points within the region of interest are tracked and used to prompt SAM throughout the video sequence, providing temporal consistency. The quantitative results surpass the state-of-the-art semi-supervised video segmentation method XMem on EndoVis2015 with 84.8 IoU and 91.0 Dice. In terms of efficiency, we tested our method on a single 4060/4090 GPU, achieving an over 25/90 FPS. Code is available at: https://github.com/wuzijian1997/SIS-PT-SAM

Force/torque sensing on hand-held tools enables control of applied forces, which is often essential in both tele-robotics and remote guidance of people. However, existing force sensors are either bulky, complex, or have insufficient load rating. This paper presents a novel force sensing modality based on differential magnetic field readings in a collection of sensor modules placed around a tool or device. The instrumentation is easy to install and low profile, but nonetheless achieves good performance. A detailed mathematical model and optimization-based design procedure are also introduced. The modeling, simulation, and optimization of the force sensor are described and then used in the electrical and mechanical design and integration of the sensor into an ultrasound probe. Through a neural network-based nonlinear calibration, the sensor achieves average root-mean-square test errors of 0.41 N and 0.027 Nm compared to an off-the-shelf ATI Nano25 sensor, which are 0.80% and 1.16% of the full-scale range respectively. The sensor has an average noise power spectral density of less than 0.0001 N/sqrt(Hz), and a 95% confidence interval resolution of 0.063 N and 0.0086 Nm. The practical readout rate is 1.3 kHz over USB serial and it can also operate over Bluetooth or Wi-Fi. This sensor can enable instrumentation of manual tools to improve the performance and transparency of teleoperated or autonomous systems.

For many augmented reality guidance, teleoperation, or human-robot interaction systems, accurate, fast, and robust 6 degree of freedom object pose tracking is essential. However, current solutions easily lose tracking when line-of-sight to markers is lost. In this paper we present a tracking system which matches or improves on current methods in speed and accuracy, achieving 1.77 mm and 1.51 degrees accuracy at 22 Hz. Reflective markers are segmented in infrared images using contour detection before using the known marker geometry to perform point correspondence and pose computation using novel approaches. At the same time, a new square root unscented Kalman filter is introduced which improves accuracy and flexibility by tracking the markers themselves rather than the computed pose, and enables fusion of an external inertial measurement unit. This reduces noise and makes the tracking robust to brief loss of line-of-sight. The algorithms and methods are described in detail with pseudo-code for ease of reproduction. The system is implemented in simulation and on a Microsoft HoloLens 2 using Unity for ease of entry and integration into graphical projects. The code is made available open source. Tests of the system are described, and the results analyzed .

We previously introduced a novel mixed reality (MR) teleguidance system, human teleoperation [1,2], in which a human (expert) leader and a human (novice) follower are tightly coupled through MR and haptics for applications such as tele-ultrasound. In this paper, a communication system suitable for human teleoperation is presented and characterized in various network conditions, over Ethernet, Wi-Fi, 4G LTE, and 5G. To study all types of latency in the system, the human response time is additionally characterized through step response tests with 11 volunteers. The step responses were obtained by tracking the position and force of the human hand in response to a change in the MR target. The round-trip communication latency is 40+/-10 ms over 5G, and down to 1+/-0.6 ms over Ethernet for typical throughputs. The human response time to a step change in position depends on the step magnitude, but is 485-535 ms, while the reaction time for forces is 150-200 ms. Both lags are decreased when tracking smooth motions. Thus, we demonstrate that the system is network agnostic and can achieve good teleoperation performance and secure, fast communication in appropriate network conditions. The presented tools and concepts are applicable to any high-performance teleoperation system, for example for remote surgery.

Purpose: In “human teleoperation” [1], augmented reality (AR) and haptics are used to tightly couple an expert leader to a human follower. To determine the feasibility of human teleoperation, we quantify the abil- ity of humans to track a position and/or force trajectory via AR cues. The human response time, precision, frequency response, and step response were characterized, and several rendering methods were compared. Methods: Volunteers (n=11) performed a series of tasks as the fol- lower in our human teleoperation system. The tasks involved tracking pre-recorded series of motions and forces, each time with a different rendering method. The order of tasks and rendering methods was ran- domized to avoid learning effects and bias. The volunteers then performed a series of frequency response tests and filled out a questionnaire. Results: Rendering the full ultrasound probe as a position target with an error bar displaying force led to the best position and force tracking. Following force and pose simultaneously was more difficult but did not lead to significant performance degradation versus following one at a time. On average, subjects tracked positions, orientations, and forces with rms tracking errors of 6.2 ± 1.9 mm, 5.9 ± 1.9˚, 1.0 ± 0.3 N, steady-state errors of 2.8 ± 2.1 mm, 0.26 ± 0.2 N, and lags of 345.5 ± 87.6 ms respectively. Performance decreased with input frequency, until the person could no longer follow, depending on the input amplitude. Conclusion: This paper characterizes human tracking ability in aug- mented reality human teleoperation, which shows the system’s feasi- bility and good performance, and is important for designing future human computer interfaces using augmented and virtual reality.

This paper introduces an integrated real-time intraoperative surgical guidance system, in which an endoscope camera of da Vinci surgical robot and a transrectal ultrasound (TRUS) transducer are co-registered using photoacoustic markers that are detected in both fluorescence (FL) and photoacoustic (PA) imaging. The co-registered system enables the TRUS transducer to track the laser spot illuminated by a pulsed-laser-diode attached to the surgical instrument, providing both FL and PA images of the surgical region-of- interest (ROI). As a result, the generated photoacoustic marker is visualized and localized in the da Vinci endoscopic FL images, and the corresponding tracking can be conducted by rotating the TRUS transducer to display the PA image of the marker. A quantitative evaluation revealed that the average registration and tracking errors were 0.84 mm and 1.16°, respectively. This study shows that the co-registered photoacoustic marker tracking can be effectively deployed intraoperatively using TRUS+PA imaging providing functional guidance of the surgical ROI.

Current teleguidance methods include verbal guidance and robotic teleoperation, which present tradeoffs between precision and latency versus flexibility and cost. We present a novel concept of “human teleoperation” which bridges the gap between these two methods. A prototype teleultrasound system was implemented which shows the concept’s efficacy. An expert remotely “teloperates” a person (the follower) wearing a mixed reality headset by controlling a virtual ultrasound probe projected into the person’s scene. The follower matches the pose and force of the virtual device with a real probe. The pose, force, video, ultrasound images, and 3-dimensional mesh of the scene are fed back to the expert. In this control framework, the input and the actuation are carried out by people, but with near robot-like latency and precision. This allows teleguidance that is more precise and fast than verbal guidance, yet more flexible and inexpensive than robotic teleoperation. The system was subjected to tests that show its effectiveness, including mean teleoperation latencies of 0.27 seconds and errors of 7 mm and 6◦ in pose tracking. The system was also tested with an expert ultrasonographer and four patients and was found to improve the precision and speed of two teleultrasound procedures.