https://arxiv.org/api/EOSSuC/QCBMgTPKTfnBE893a4ic 2026-10-01T02:39:56Z 25 41 0 http://arxiv.org/abs/2609.38483v1 2026-09-29T20:11:43Z

Indirect manipulation of deep-seated deformable anatomy inaccessible to the robot is challenging in robot-assisted minimally invasive surgery (RAMIS) because intervening tissues spatially filter deformation transmission. Passive observations can be ambiguous because the Decoupled and Blocked modes may produce similar motion responses. We propose CADeT, a causal-aware deformation transmission framework that integrates structural causal model (SCM) with active sensing to infer a latent transmission mode and estimate a state-dependent adhesion Jacobian online. During normal manipulation, control actions update the mode belief; when ambiguity persists, an additional probing action is selected to improve mode distinguishability. The mode belief and learned Jacobian are incorporated into a belief-aware model predictive controller for indirect target-shape control. Validation in simulation and on the da Vinci research kit (dVRK), using phantom and ex vivo porcine tissues, shows higher mode-identification accuracy and faster shape-error convergence than the evaluated model-free and model-based baselines. These results show that active sensing improves mode identification and indirect deformation control under the evaluated conditions.

2026-09-29T20:11:43Z Junlei Hu Dominic Jones Pietro Valdastri http://arxiv.org/abs/2609.16186v1 2026-09-14T18:20:02Z

Autonomous soft-tissue cancer surgery has been limited to interventions on organ surfaces, because current systems cannot perceive and adapt to anatomy once it deforms or is cut. We introduce the first vision-guided autonomous system capable of performing complete tumor resections for partial nephrectomy. Our system integrates conditional occupancy networks, trained entirely in a physics-based simulation, that infer full 3-D anatomy (tumor, margin tissue, and kidney) from single-view partial point clouds. These occupancy networks maintain intraoperative tracking even as tissue is cut and deformed, enabling adaptive planning and execution. The surgical platform combines a depth camera for capturing surface point clouds, dual robotic arms for electrosurgical cutting and vacuum-based tissue manipulation, and an autonomous control strategy for tumor resection. In patient-derived hydrogel phantoms under an open partial nephrectomy setting, the robot performed eight consecutive autonomous tumor resections comprising 77 electrosurgical cuts, with all cuts achieving negative surgical margins and 1.61 $\pm$ 0.48 mm mean absolute margin error. This work demonstrates, for the first time, a foundation for supervised autonomous closed-loop, imaging-driven, margin-negative tumor removal in phantoms.

2026-09-14T18:20:02Z Ethan Kilmer Pit Henrich Jiawei Ge Paul M. Scheikl Laura Connolly Soum D. Lokeshwar Joseph Chen Justin D. Opfermann Kaitlyn Kumar Lauren Shepard Ahmed Ghazi Nirmish Singla Richard J. Cha Kevin Cleary Franziska Mathis-Ullrich Axel Krieger http://arxiv.org/abs/2606.01572v1 2026-06-01T02:20:54Z

Predicting patient-specific facial soft-tissue deformation is critical for iterative orthognathic surgery planning. However, current computational methods face a strict accuracy-efficiency trade-off: high-fidelity Finite Element Methods (FEM) are computationally prohibitive, whereas pure deep learning models often produce biomechanically inconsistent results. While Physics-Informed Neural Networks (PINNs) offer a promising avenue, learning the complex heterogeneous mechanics of bone--soft-tissue interactions with only partial clinical supervision (i.e., outer facial surfaces) remains highly unstable. To overcome these challenges, we present PINNOCHIO, a novel physics-informed framework for facial soft-tissue simulation. PINNOCHIO introduces a hybrid sequential decomposition that explicitly decouples discontinuous bone--soft-tissue interface movements from continuous volumetric hyperelastic deformation. This structural separation enables stable training and facilitates a physics-enabled sim-to-real adaptation strategy, ensuring internal biomechanical consistency without requiring volumetric ground truth. Evaluated on a 40-patient clinical cohort, PINNOCHIO outperforms existing baselines in both surface accuracy and physical validity. Furthermore, it achieves a substantial speedup over FEM, successfully resolving the accuracy-efficiency trade-off to provide a highly reliable and practical tool for interactive surgical planning.

2026-06-01T02:20:54Z This work has been submitted to MICCAI 2026 Jungwook Lee Daeseung Kim Kevin Gu Zhangfeng Hu Tianshu Kuang Finn Hopeman Michael A. K. Liebschner Jaime Gateno Pingkun Yan http://arxiv.org/abs/2604.26781v1 2026-04-29T15:12:18Z

Surgical training involves didactic teaching, mentor-led learning, surgical skills laboratories, and direct exposure to surgery; however, increasing clinical pressures have limited operating room (OR) exposure. This work leverages virtual reality (VR) to provide a safe and immersive training environment. Existing VR training is often based on standardized scenarios not tailored to individual clinical cases. This study addresses this limitation using artificial intelligence (AI) based computer vision methods to generate patient-specific simulations from computed tomography (CT) and magnetic resonance imaging (MRI). This study focuses on patient-specific spinal decompression simulation for spinal stenosis in a virtual operating room. The objectives were (1) automatic creation of 3D anatomical models and (2) VR simulation of spinal decompression procedures including laminectomy, disc resection, and foraminotomy. Model construction required multimodal fusion (registration) of CT and MRI and segmentation of relevant structures. Segmentation was evaluated using the Dice Similarity Coefficient (DSC), and registration accuracy using Target Registration Error (TRE). Qualitative feedback was obtained from surgeons and trainees. High-fidelity patient-specific 3D models were generated efficiently (approximately 2.5 minutes per case, N = 15). Segmentation accuracy was high, with a DSC of 0.95 (+/- 0.03) for vertebral bone and 0.895 (+/- 0.02) for soft tissue structures. Registration accuracy showed a mean TRE of 1.73 (+/- 0.42) mm. Semi-structured interviews indicated improved spatial understanding, increased procedural confidence, and strong perceived educational value. This platform significantly reduced the time and costs of patient-specific modelling, thereby facilitating pre-operative planning, post-procedural assessments, and comprehensive surgical simulation.

2026-04-29T15:12:18Z Raj Kumar Ranabhat Tayler D Ross Tony Jiao Jeremie Larouche Joel Finkelstein Michael Hardisty http://arxiv.org/abs/2601.02088v3 2026-08-20T03:40:59Z

Orthognathic surgery repositions jaw bones to restore occlusion and enhance facial aesthetics. Accurate simulation of postoperative facial morphology is essential for preoperative planning. This study aims to develop and validate a physics-informed geometric deep learning framework named PhysSFI-Net for precise prediction of soft tissue deformation following orthognathic surgery. The model integrates a hierarchical feature extraction module with attention mechanisms to capture skeletal-facial interactions, an LSTM-based sequential predictor for incremental deformation, and a biomechanics-inspired reconstruction module for high-resolution facial modeling. The model was trained on 135 patients and externally validated on an independent cohort of 33 patients. Model performance was assessed using point cloud shape error, surface deviation error and landmark error between predicted facial shapes with corresponding ground truths. Quantitative analysis demonstrated that PhysSFI-Net achieved a global shape error of 1.070 +/- 0.088 mm, a surface deviation error of 1.296 +/- 0.349 mm and a landmark error of 2.445 +/- 1.326 mm. Comparative experiments indicated that PhysSFI-Net outperformed the state-of-the-art method ACMT-Net and baseline models. External validation further confirmed its robustness with a global HD of 1.431 +/- 0.087 mm and consistently lower subregional and mesh-based errors. In conclusion, PhysSFI-Net enables interpretable, high-resolution prediction of postoperative facial morphology, showing strong potential for clinical application in orthognathic surgical planning.

2026-01-05T13:14:19Z 16 pages, 9 figures Bao, J., Liu, H., Zhuang, Y. et al. PhysSFI-Net: physics-informed geometric learning of skeletal and facial interactions for orthognathic surgical outcome prediction. npj Digit. Med. (2026) Jiahao Bao Huazhen Liu Yu Zhuang Leran Tao Xinyu Xu Yongtao Shi Mengjia Cheng Yiming Wang Congshuang Ku Ting Zeng Yilang Du Siyi Chen Shunyao Shen Suncheng Xiang Hongbo Yu 10.1038/s41746-026-03075-6 http://arxiv.org/abs/2511.19198v1 2025-11-24T15:07:45Z

Surgical planning and training based on machine learning requires a large amount of 3D anatomical models reconstructed from medical imaging, which is currently one of the major bottlenecks. Obtaining these data from real patients and during surgery is very demanding, if even possible, due to legal, ethical, and technical challenges. It is especially difficult for soft tissue organs with poor imaging contrast, such as the prostate. To overcome these challenges, we present a novel workflow for automated 3D anatomical data generation using data obtained from physical organ models. We additionally use a 3D Generative Adversarial Network (GAN) to obtain a manifold of 3D models useful for other downstream machine learning tasks that rely on 3D data. We demonstrate our workflow using an artificial prostate model made of biomimetic hydrogels with imaging contrast in multiple zones. This is used to physically simulate endoscopic surgery. For evaluation and 3D data generation, we place it into a customized ultrasound scanner that records the prostate before and after the procedure. A neural network is trained to segment the recorded ultrasound images, which outperforms conventional, non-learning-based computer vision techniques in terms of intersection over union (IoU). Based on the segmentations, a 3D mesh model is reconstructed, and performance feedback is provided.

2025-11-24T15:07:45Z 6 pages, 4 figures, 1 table, IEEE International Conference on Intelligent Robots and Systems (IROS) Ann-Sophia Müller Moonkwang Jeong Meng Zhang Jiyuan Tian Arkadiusz Miernik Stefanie Speidel Tian Qiu http://arxiv.org/abs/2502.06336v1 2025-02-10T10:37:21Z

Soft-tissue surgeries, such as tumor resections, are complicated by tissue deformations that can obscure the accurate location and shape of tissues. By representing tissue surfaces as point clouds and applying non-rigid point cloud registration (PCR) methods, surgeons can better understand tissue deformations before, during, and after surgery. Existing non-rigid PCR methods, such as feature-based approaches, struggle with robustness against challenges like noise, outliers, partial data, and large deformations, making accurate point correspondence difficult. Although learning-based PCR methods, particularly Transformer-based approaches, have recently shown promise due to their attention mechanisms for capturing interactions, their robustness remains limited in challenging scenarios. In this paper, we present DefTransNet, a novel end-to-end Transformer-based architecture for non-rigid PCR. DefTransNet is designed to address the key challenges of deformable registration, including large deformations, outliers, noise, and partial data, by inputting source and target point clouds and outputting displacement vector fields. The proposed method incorporates a learnable transformation matrix to enhance robustness to affine transformations, integrates global and local geometric information, and captures long-range dependencies among points using Transformers. We validate our approach on four datasets: ModelNet, SynBench, 4DMatch, and DeformedTissue, using both synthetic and real-world data to demonstrate the generalization of our proposed method. Experimental results demonstrate that DefTransNet outperforms current state-of-the-art registration networks across various challenging conditions. Our code and data are publicly available.

2025-02-10T10:37:21Z Sara Monji-Azad Marvin Kinz Siddharth Kothari Robin Khanna Amrei Carla Mihan David Maennel Claudia Scherl Juergen Hesser http://arxiv.org/abs/2412.02359v1 2024-12-03T10:32:41Z

This paper tackles the challenge of automatically performing realistic surgical simulations from readily available surgical videos. Recent efforts have successfully integrated physically grounded dynamics within 3D Gaussians to perform high-fidelity simulations in well-reconstructed simulation environments from static scenes. However, they struggle with the geometric inconsistency in reconstructing simulation environments and unrealistic physical deformations in simulations of soft tissues when it comes to dynamic and complex surgical processes. In this paper, we propose SurgiSim, a novel automatic simulation system to overcome these limitations. To build a surgical simulation environment, we maintain a canonical 3D scene composed of 3D Gaussians coupled with a deformation field to represent a dynamic surgical scene. This process involves a multi-stage optimization with trajectory and anisotropic regularization, enhancing the geometry consistency of the canonical scene, which serves as the simulation environment. To achieve realistic physical simulations in this environment, we implement a Visco-Elastic deformation model based on the Maxwell model, effectively restoring the complex deformations of tissues. Additionally, we infer the physical parameters of tissues by minimizing the discrepancies between the input video and simulation results guided by estimated tissue motion, ensuring realistic simulation outcomes. Experiments on various surgical scenarios and interactions demonstrate SurgiSim's ability to perform realistic simulation of soft tissues among surgical procedures, showing its enormous potential for enhancing surgical training, planning, and robotic surgery systems. The project page is at https://namaenashibot.github.io/SurgiSim/.

2024-12-03T10:32:41Z Kailing Wang Chen Yang Keyang Zhao Xiaokang Yang Wei Shen http://arxiv.org/abs/2411.07353v1 2024-11-11T20:25:12Z

Advanced simulations of the mechanical behavior of soft tissues frequently rely on structure-based constitutive models, including smeared descriptions of collagen fibers. Among them, the so-called Discrete Fiber Dispersion (DFD) model is based on a discrete integration of the fiber-strain energy over all the fiber directions. In this paper, we recall the theoretical framework of the DFD model, including a derivation of the stress and stiffness tensors required for the finite element implementation. Specifically, their expressions for incompressible plane stress problems are obtained. The use of a Lebedev quadrature, built exploiting the octahedral symmetry, is then proposed, illustrating the particular choice adopted for the orientation of the integration points. Next, the convergence of this quadrature scheme is assessed by means of three numerical benchmark tests, highlighting the advantages with respect to other angular integration methods available in the literature. Finally, we propose as applicative example a simulation of Z-plasty, a technique commonly used in reconstructive skin surgery, considering multiple geometrical configurations and orientations of the fibers. Results are provided in terms of key mechanical quantities relevant for the surgical practice.

2024-11-11T20:25:12Z 29 pages, 12 figures Riccardo Alberini Michele Terzano Gerhard A. Holzapfel Andrea Spagnoli http://arxiv.org/abs/2404.15339v1 2024-04-10T07:49:07Z

The healthcare industry has a growing need for realistic modeling and efficient simulation of surgical scenes. With effective models of deformable surgical scenes, clinicians are able to conduct surgical planning and surgery training on scenarios close to real-world cases. However, a significant challenge in achieving such a goal is the scarcity of high-quality soft tissue models with accurate shapes and textures. To address this gap, we present a data-driven framework that leverages emerging neural radiance field technology to enable high-quality surgical reconstruction and explore its application for surgical simulations. We first focus on developing a fast NeRF-based surgical scene 3D reconstruction approach that achieves state-of-the-art performance. This method can significantly outperform traditional 3D reconstruction methods, which have failed to capture large deformations and produce fine-grained shapes and textures. We then propose an automated creation pipeline of interactive surgical simulation environments through a closed mesh extraction algorithm. Our experiments have validated the superior performance and efficiency of our proposed approach in surgical scene 3D reconstruction. We further utilize our reconstructed soft tissues to conduct FEM and MPM simulations, showcasing the practical application of our method in data-driven surgical simulations.

2024-04-10T07:49:07Z 14 pages, 4 figures. Accepted by International Journal of Computer Assisted Radiology and Surgery Yuehao Wang Bingchen Gong Yonghao Long Siu Hin Fan Qi Dou http://arxiv.org/abs/2404.05888v2 2024-08-07T20:20:46Z

Realistic real-time surgical simulators play an increasingly important role in surgical robotics research, such as surgical robot learning and automation, and surgical skills assessment. Although there are a number of existing surgical simulators for research, they generally lack the ability to simulate the diverse types of objects and contact-rich manipulation tasks typically present in surgeries, such as tissue cutting and blood suction. In this work, we introduce CRESSim, a realistic surgical simulator based on PhysX 5 for the da Vinci Research Kit (dVRK) that enables simulating various contact-rich surgical tasks involving different surgical instruments, soft tissue, and body fluids. The real-world dVRK console and the master tool manipulator (MTM) robots are incorporated into the system to allow for teleoperation through virtual reality (VR). To showcase the advantages and potentials of the simulator, we present three examples of surgical tasks, including tissue grasping and deformation, blood suction, and tissue cutting. These tasks are performed using the simulated surgical instruments, including the large needle driver, suction irrigator, and curved scissor, through VR-based teleoperation.

2024-04-08T22:01:28Z 7 pages 2024 21st International Conference on Ubiquitous Robots (UR), New York, NY, USA, 2024, pp. 64-70 Yafei Ou Sadra Zargarzadeh Paniz Sedighi Mahdi Tavakoli 10.1109/UR61395.2024.10597513 http://arxiv.org/abs/2309.11656v2 2024-05-29T18:42:40Z

Accurate deformable object manipulation (DOM) is essential for achieving autonomy in robotic surgery, where soft tissues are being displaced, stretched, and dissected. Many DOM methods can be powered by simulation, which ensures realistic deformation by adhering to the governing physical constraints and allowing for model prediction and control. However, real soft objects in robotic surgery, such as membranes and soft tissues, have complex, anisotropic physical parameters that a simulation with simple initialization from cameras may not fully capture. To use the simulation techniques in real surgical tasks, the "real-to-sim" gap needs to be properly compensated. In this work, we propose an online, adaptive parameter tuning approach for simulation optimization that (1) bridges the real-to-sim gap between a physics simulation and observations obtained 3D perceptions through estimating a residual mapping and (2) optimizes its stiffness parameters online. Our method ensures a small residual gap between the simulation and observation and improves the simulation's predictive capabilities. The effectiveness of the proposed mechanism is evaluated in the manipulation of both a thin-shell and volumetric tissue, representative of most tissue scenarios. This work contributes to the advancement of simulation-based deformable tissue manipulation and holds potential for improving surgical autonomy.

2023-09-20T21:41:14Z Xiao Liang Fei Liu Yutong Zhang Yuelei Li Shan Lin Michael Yip http://arxiv.org/abs/2309.00837v1 2023-09-02T06:13:58Z

In the context of surgery, robots can provide substantial assistance by performing small, repetitive tasks such as suturing, needle exchange, and tissue retraction, thereby enabling surgeons to concentrate on more complex aspects of the procedure. However, existing surgical task learning mainly pertains to rigid body interactions, whereas the advancement towards more sophisticated surgical robots necessitates the manipulation of soft bodies. Previous work focused on tissue phantoms for soft tissue task learning, which can be expensive and can be an entry barrier to research. Simulation environments present a safe and efficient way to learn surgical tasks before their application to actual tissue. In this study, we create a Robot Operating System (ROS)-compatible physics simulation environment with support for both rigid and soft body interactions within surgical tasks. Furthermore, we investigate the soft tissue interactions facilitated by the patient-side manipulator of the DaVinci surgical robot. Leveraging the pybullet physics engine, we simulate kinematics and establish anchor points to guide the robotic arm when manipulating soft tissue. Using demonstration-guided reinforcement learning (RL) algorithms, we investigate their performance in comparison to traditional reinforcement learning algorithms. Our in silico trials demonstrate a proof-of-concept for autonomous surgical soft tissue retraction. The results corroborate the feasibility of learning soft body manipulation through the application of reinforcement learning agents. This work lays the foundation for future research into the development and refinement of surgical robots capable of managing both rigid and soft tissue interactions. Code is available at https://github.com/amritpal-001/tissue_retract.

2023-09-02T06:13:58Z 10 Pages, 5 figures, MICCAI 2023 conference (AECAI workshop) Amritpal Singh Wenqi Shi May D Wang http://arxiv.org/abs/2307.10954v1 2023-07-20T15:26:01Z

In CMF surgery, the planning of bony movement to achieve a desired facial outcome is a challenging task. Current bone driven approaches focus on normalizing the bone with the expectation that the facial appearance will be corrected accordingly. However, due to the complex non-linear relationship between bony structure and facial soft-tissue, such bone-driven methods are insufficient to correct facial deformities. Despite efforts to simulate facial changes resulting from bony movement, surgical planning still relies on iterative revisions and educated guesses. To address these issues, we propose a soft-tissue driven framework that can automatically create and verify surgical plans. Our framework consists of a bony planner network that estimates the bony movements required to achieve the desired facial outcome and a facial simulator network that can simulate the possible facial changes resulting from the estimated bony movement plans. By combining these two models, we can verify and determine the final bony movement required for planning. The proposed framework was evaluated using a clinical dataset, and our experimental results demonstrate that the soft-tissue driven approach greatly improves the accuracy and efficacy of surgical planning when compared to the conventional bone-driven approach.

2023-07-20T15:26:01Z Early accepted by MICCAI 2023 Xi Fang Daeseung Kim Xuanang Xu Tianshu Kuang Nathan Lampen Jungwook Lee Hannah H. Deng Jaime Gateno Michael A. K. Liebschner James J. Xia Pingkun Yan http://arxiv.org/abs/2306.05509v1 2023-06-08T19:00:35Z

Image-guided surgery collocates patient-specific data with the physical environment to facilitate surgical decision making in real-time. Unfortunately, these guidance systems commonly become compromised by intraoperative soft-tissue deformations. Nonrigid image-to-physical registration methods have been proposed to compensate for these deformations, but intraoperative clinical utility requires compatibility of these techniques with data sparsity and temporal constraints in the operating room. While linear elastic finite element models are effective in sparse data scenarios, the computation time for finite element simulation remains a limitation to widespread deployment. This paper proposes a registration algorithm that uses regularized Kelvinlets, which are analytical solutions to linear elasticity in an infinite domain, to overcome these barriers. This algorithm is demonstrated and compared to finite element-based registration on two datasets: a phantom dataset representing liver deformations and an in vivo dataset representing breast deformations. The regularized Kelvinlets algorithm resulted in a significant reduction in computation time compared to the finite element method. Accuracy as evaluated by target registration error was comparable between both methods. Average target registration errors were 4.6 +/- 1.0 and 3.2 +/- 0.8 mm on the liver dataset and 5.4 +/- 1.4 and 6.4 +/- 1.5 mm on the breast dataset for the regularized Kelvinlets and finite element method models, respectively. This work demonstrates the generalizability of using a regularized Kelvinlets registration algorithm on multiple soft tissue elastic organs. This method may improve and accelerate registration for image-guided surgery applications, and it shows the potential of using regularized Kelvinlets solutions on medical imaging data.

2023-06-08T19:00:35Z 17 pages, 9 figures Morgan Ringel Jon Heiselman Winona Richey Ingrid Meszoely William Jarnagin Michael Miga http://arxiv.org/abs/2304.01027v1 2023-04-03T14:27:52Z

During brain tumour resection, localising cancerous tissue and delineating healthy and pathological borders is challenging, even for experienced neurosurgeons and neuroradiologists. Intraoperative imaging is commonly employed for determining and updating surgical plans in the operating room. Ultrasound (US) has presented itself a suitable tool for this task, owing to its ease of integration into the operating room and surgical procedure. However, widespread establishment of this tool has been limited because of the difficulty of anatomy localisation and data interpretation. In this work, we present a robotic framework designed and tested on a soft-tissue-mimicking brain phantom, simulating intraoperative US (iUS) scanning during brain tumour surgery.

2023-04-03T14:27:52Z 2023 ICRA-RAMI workshop Michael Dyck Alistair Weld Julian Klodmann Alexander Kirst Giulio Anichini Luke Dixon Sophie Camp Stamatia Giannarou Alin Albu-Schäffer http://arxiv.org/abs/2208.12958v1 2022-08-27T08:01:59Z

A stable shape for corneas experiencing refractive surgery has to be sustained so as to elude post-refractive surgery de-compensation. This de-compensation leads to visual complications and unsatisfactory procedure recovery. Variation in corneal lamellae and collagen fibres is induced by recent LASER refractive surgical procedures utilizing LASER ablation and disruption techniques. Conserving a steady response of central apex flattening and peripheral steepening in an elastic cornea pre- and post- procedure is the ultimate purpose of successful refractive surgery. Early diagnosis of ectatic corneal disorders and better understanding of corneal pathogenesis is achieved by assessment of corneal biomechanical properties. The ultimate objective of this research is to estimate the biomechanical properties for both normal and pathogenic corneal tissue pre- and post-operative refractive surgery. This achieved using ultrasonic acoustic radiation force impulse as a non-invasive method accounting for its high localization. Induced displacement tracking methods will be utilized for assessment of soft tissue biomechanical properties related to the investigated soft tissue. Ultrasound probe simulations will be carried out to optimize the probe design. FEM simulations will take place to precisely estimate in-situ corneal tissue biomechanics. In this research, corneal biomechanical properties are studied and estimated using acoustic radiation force impulse. This is achieved either by estimating the focal peak axial deformation value or by estimating the shear wave speed for the resulting propagating deformation wave.

2022-08-27T08:01:59Z Hassan M. Ahmed Nancy M. Salem Walid I. Al-Atabany http://arxiv.org/abs/2203.05438v1 2022-03-10T15:54:31Z

Autonomous robotic surgery requires deliberation, i.e. the ability to plan and execute a task adapting to uncertain and dynamic environments. Uncertainty in the surgical domain is mainly related to the partial pre-operative knowledge about patient-specific anatomical properties. In this paper, we introduce a logic-based framework for surgical tasks with deliberative functions of monitoring and learning. The DEliberative Framework for Robot-Assisted Surgery (DEFRAS) estimates a pre-operative patient-specific plan, and executes it while continuously measuring the applied force obtained from a biomechanical pre-operative model. Monitoring module compares this model with the actual situation reconstructed from sensors. In case of significant mismatch, the learning module is invoked to update the model, thus improving the estimate of the exerted force. DEFRAS is validated both in simulated and real environment with da Vinci Research Kit executing soft tissue retraction. Compared with state-of-the art related works, the success rate of the task is improved while minimizing the interaction with the tissue to prevent unintentional damage.

2022-03-10T15:54:31Z 2022 International Conference on Robotics and Automation Eleonora Tagliabue Daniele Meli Diego Dall'Alba Paolo Fiorini 10.1109/ICRA46639.2022.9811820 http://arxiv.org/abs/2110.00336v1 2021-10-01T11:59:33Z

The current research focus in Robot-Assisted Minimally Invasive Surgery (RAMIS) is directed towards increasing the level of robot autonomy, to place surgeons in a supervisory position. Although Learning from Demonstrations (LfD) approaches are among the preferred ways for an autonomous surgical system to learn expert gestures, they require a high number of demonstrations and show poor generalization to the variable conditions of the surgical environment. In this work, we propose an LfD methodology based on Generative Adversarial Imitation Learning (GAIL) that is built on a Deep Reinforcement Learning (DRL) setting. GAIL combines generative adversarial networks to learn the distribution of expert trajectories with a DRL setting to ensure generalisation of trajectories providing human-like behaviour. We consider automation of tissue retraction, a common RAMIS task that involves soft tissues manipulation to expose a region of interest. In our proposed methodology, a small set of expert trajectories can be acquired through the da Vinci Research Kit (dVRK) and used to train the proposed LfD method inside a simulated environment. Results indicate that our methodology can accomplish the tissue retraction task with human-like behaviour while being more sample-efficient than the baseline DRL method. Towards the end, we show that the learnt policies can be successfully transferred to the real robotic platform and deployed for soft tissue retraction on a synthetic phantom.

2021-10-01T11:59:33Z Accepted in IEEE International Symposium of Medical Robotics (ISMR 2021) Ameya Pore Eleonora Tagliabue Marco Piccinelli Diego Dall'Alba Alicia Casals Paolo Fiorini http://arxiv.org/abs/2109.07164v2 2022-02-18T10:26:00Z

Invasive intracranial electroencephalography (iEEG) or electrocorticography (ECoG) measures electric potential directly on the surface of the brain and can be used to inform treatment planning for epilepsy surgery. Combined with numerical modeling they can further improve accuracy of epilepsy surgery planning. Accurate solution of the iEEG or ECoG forward problem, which is a crucial prerequisite for solving the inverse problem in epilepsy seizure onset zone localization, requires accurate representation of the patient's brain geometry and tissue electrical conductivity after implantation of electrodes. However, implantation of subdural grid electrodes causes the brain to deform, which invalidates preoperatively acquired image data. Moreover, postoperative magnetic resonance imaging (MRI) is incompatible with implanted electrodes and computed tomography (CT) has insufficient range of soft tissue contrast, which precludes both MRI and CT from being used to obtain the deformed postoperative geometry. In this paper, we present a biomechanics-based image warping procedure using preoperative MRI for tissue classification and postoperative CT for locating implanted electrodes to perform non-rigid registration of the preoperative image data to the postoperative configuration. We solve the iEEG forward problem on the predicted postoperative geometry using the finite element method (FEM) which accounts for patient-specific inhomogeneity and anisotropy of tissue conductivity. Results for the simulation of a current source in the brain show large differences in electric potential predicted by the models based on the original images and the deformed images corresponding to the brain geometry deformed by placement of invasive electrodes. Computation of the lead field matrix (useful for solution of the iEEG inverse problem) also showed significant differences between the different models.

2021-09-15T09:00:02Z Benjamin F. Zwick George C. Bourantas Saima Safdar Grand R. Joldes Damon E. Hyde Simon K. Warfield Adam Wittek Karol Miller http://arxiv.org/abs/2109.02316v2 2021-09-28T11:41:53Z

Autonomy in robot-assisted surgery is essential to reduce surgeons' cognitive load and eventually improve the overall surgical outcome. A key requirement for autonomy in a safety-critical scenario as surgery lies in the generation of interpretable plans that rely on expert knowledge. Moreover, the Autonomous Robotic Surgical System (ARSS) must be able to reason on the dynamic and unpredictable anatomical environment, and quickly adapt the surgical plan in case of unexpected situations. In this paper, we present a modular Framework for Robot-Assisted Surgery (FRAS) in deformable anatomical environments. Our framework integrates a logic module for task-level interpretable reasoning, a biomechanical simulation that complements data from real sensors, and a situation awareness module for context interpretation. The framework performance is evaluated on simulated soft tissue retraction, a common surgical task to remove the tissue hiding a region of interest. Results show that the framework has the adaptability required to successfully accomplish the task, handling dynamic environmental conditions and possible failures, while guaranteeing the computational efficiency required in a real surgical scenario. The framework is made publicly available.

2021-09-06T09:31:55Z 2021 IEEE International Symposium on Medical Robotics (ISMR) D. Meli E. Tagliabue D. Dall'Alba P. Fiorini http://arxiv.org/abs/2102.01436v1 2021-02-02T11:16:12Z

Recent developments in surgical robotics have led to new advancements in the automation of surgical sub-tasks such as suturing, soft tissue manipulation, tissue tensioning and cutting. However, integration of dynamics to optimize these control policies for the variety of scenes encountered in surgery remains unsolved. Towards this effort, we investigate the integration of differentiable fluid dynamics to optimizing a suction tool's trajectory to clear the surgical field from blood as fast as possible. The fully differentiable fluid dynamics is integrated with a novel suction model for effective model predictive control of the tool. The differentiability of the fluid model is crucial because we utilize the gradients of the fluid states with respect to the suction tool position to optimize the trajectory. Through a series of experiments, we demonstrate how, by incorporating fluid models, the trajectories generated by our method can perform as good as or better than handcrafted human-intuitive suction policies. We also show that our method is adaptable and can work in different cavity conditions while using a single handcrafted strategy fails.

2021-02-02T11:16:12Z Submission to ICRA2021 Jingbin Huang Fei Liu Florian Richter Michael C. Yip http://arxiv.org/abs/2011.00800v3 2021-04-30T05:11:55Z

Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a registered, simulated dynamical model. In this work, we propose an online, continuous, real-to-sim registration method to bridge 3D visual perception with position-based dynamics (PBD) modeling of tissues. The PBD method is employed to simulate soft tissue dynamics as well as rigid tool interactions for model-based control. Meanwhile, a vision-based strategy is used to generate 3D reconstructed point cloud surfaces based on real-world manipulation, so as to register and update the simulation. To verify this real-to-sim approach, tissue experiments have been conducted on the da Vinci Research Kit. Our real-to-sim approach successfully reduces registration error online, which is especially important for safety during autonomous control. Moreover, it achieves higher accuracy in occluded areas than fusion-based reconstruction.

2020-11-02T07:59:55Z Fei Liu and Zihan Li contributed equally to this work. Final version of this paper is going to be published in ICRA 2021 Fei Liu Zihan Li Yunhai Han Jingpei Lu Florian Richter Michael C. Yip http://arxiv.org/abs/2010.13936v2 2021-02-28T16:25:07Z

The control and task automation of robotic surgical system is very challenging, especially in soft tissue manipulation, due to the unpredictable deformations. Thus, an accurate simulator of soft tissues with the ability of interacting with robot manipulators is necessary. In this work, we propose a novel 2D simulation framework for tool-tissue interaction. This framework continuously tracks the motion of manipulator and simulates the tissue deformation in presence of collision detection. The deformation energy can be computed for the control and planning task.

2020-10-26T22:47:32Z Accepted by IROS Workshop 2020 (Cognitive Robotic Surgery), 2 pages, 3 figures Yunhai Han Fei Liu Michael C. Yip http://arxiv.org/abs/2010.13823v3 2020-11-09T14:49:08Z

Imaging modalities provide clinicians with real-time visualization of anatomical regions of interest (ROI) for the purpose of minimally invasive surgery. During the procedure, low-resolution image data are acquired and registered with high-resolution preoperative 3D reconstruction to guide the execution of surgical preplan. Unfortunately, due to the potential large strain and nonlinearities in the deformation of soft biological tissues, significant mismatch may be observed between ROI shapes during pre- and intra-operative imaging stages, making the surgical preplan prone to failure. In an effort to bridge the gap between the two imaging stages, this paper presents a data-driven approach based on artificial neural network for predicting the ROI deformation in real-time with sparsely registered fiducial markers. For a head-and-neck tumor model with an average maximum displacement of 30 mm, the maximum surface offsets between benchmarks and predictions using the proposed approach for 98% of the test cases are under 1.0 mm, which is the typical resolution of high-quality interventional ultrasound. Each of the prediction processes takes less than 0.5 s. With the resulting prediction accuracy and computational efficiency, the proposed approach demonstrates its potential to be clinically relevant.

2020-10-26T18:17:11Z The first two authors contribute equally. Accepted to ML4Eng @ NeurIPS 2020 Haolin Liu Ye Han Daniel Emerson Houriyeh Majditehran Qi Wang Yoed Rabin Levent Burak Kara