Building a digital twin system for skull base surgery requires precise modeling, tracking, and updating of the patient’s anatomy, the surgeon’s tool(s), and the surgical camera. In our setup, the system includes a surgical phantom to simulate the patient’s anatomy, a surgical drill as the ablation tool, and a stereo surgical microscope. Twin-S acquires 3D poses of each component via an optical tracker (Fusion-Track 500, Atracsys¹). The pose measurements are streamed into a physics-based real-time virtual environment built upon AMBF [22,23]. The virtual environment provides computational analysis as feedback to the real-world processes. An overview is shown in Fig. 2.

The 3D model of the surgical tool is obtained from its manufacturer (ANSPACH, Johnson & Johnson²). In Twin-S, we assign the drill coordinate frame to the drill tip center. We mount the optical tracking markers at the tail of the drill shaft, defining the base coordinate of the drill. To find the transformation Fddb between the base coordinate and the drill coordinate, we calibrate the rotational Rddb and the translational tddb components.

The translational component tddb is obtained by pivot calibration [25]. For the rotational component Rddb, it is only necessary to calibrate two degrees of freedom as the drill is symmetrical along its shaft. The calibration reduces to aligning two linear line segments—the shaft axis in the drill coordinate (denoted as P) and the optical tracker coordinate (denoted as Q). Using the Kabsch algorithm [26], the rotation can be recovered as: (1) Rddb=HTH12H−1,whereH=PTQ.

The shaft axis in the drill coordinate P is defined to be along the Z-axis and thus is known. To recover the shaft axis in optical tracker coordinate Q, we fix the drill on a robot arm end effector, where the drill holder is by design aligns the drill shaft with the Z-axis of the robot to sufficient accuracy [27]. Therefore, by commanding the robot to move along its Z-axis, we recover the direction of motion Q in the optical tracker coordinate. Given P and Q, Rddb can be recovered.

We obtain the 3D structure of the surgical phantom using a CT scanner. The phantom is modeled as a binary volume of occupancy, where voxels corresponding to the bony tissues are marked as occupied and voxels representing air are marked as free space. To track the surgical phantom, we rigidly mount the phantom on a polycarbonate board with optical tracking markers, defining the base coordinate of the phantom. To calibrate the transformation from phantom base to phantom Fppb, we directly compute the transformation between the virtual model and the physical phantom via the point-to-plane ICP registration [28]. We sample 380 points on the physical phantom surface using a tracked pointer tool for the purpose of calibration.

When surgeons perform drilling in the real world, we update the surgical phantom in real time. We approximate the drill tip as a sphere. Twin-S detects collisions between the surgical phantom and the drill burr given the tracked positions. The voxels that collide with the drill tip sphere are moved and set to free space to simulate the tissue removal process. More details of the drilling algorithm can be found in prior work [22].

We obtain the intrinsic parameters and distortion coefficients of the stereo camera using a ChArUco pattern. Rectification is then performed to obtain a projective camera model [29]. To track the surgical camera, optical tracking markers are mounted on the handle of the camera, defining the base coordinate frame. A hand-eye calibration routine [30,31] is used to obtain Fccb. Given the tracked camera, Twin-S generates per-pixel segmentation mask and depth maps based on the object information [22], which can be used for different downstream applications.

We conduct experiments to evaluate the accuracy of the optical tracker, as its accuracy directly impacts the results of Twin-S. Throughout the experiment, we put the optical tracker at a fixed location. We mount the optical markers on a three-axis micrometer stage (Fig. 3a). For each experiment trial, we individually move 5 mm along the x-, y-, and z-axis in the micrometer stage local coordinate. We conduct three trials in total, each with a new orientation w.r.t. the optical tracker (Fig. 3b). The histogram of error is plotted in Fig. 3c. The mean error is 0.02 mm, with a standard deviation of 0.02 mm and a maximum error of 0.08 mm. We find the sub-millimeter accuracy of the optical tracker sufficient for our purpose.

We assess the accuracy of camera calibration as it is critical to downstream applications such as mixed reality. To perform a quantitative evaluation of hand-eye calibration, we record a video sequence of a moving camera observing a static ChArUco pattern. We take the first video frame as the reference frame. The reference frame is transformed and projected into subsequent video frames. We then evaluate the alignment of the projected pattern and observed pattern. We report the re-projection error (RPE) among all frames of size 1080 × 1920 of a video sequence.

The hand-eye calibration method has a 16-pixel mean RPE. We further convert the RPE to calibration errors by using the projective relationship between the pixel and Cartesian coordinate. The ratio between the camera-to-ChArUco distance tuc in mm and surgical camera focal length fpx in pixels is equal to the ratio between the calibration error ϵuc in mm and mean RPE in pixels. In other words, we have tuc/fpx=ϵuc/RPE. Given the estimate of tuc from physical setup and the calibrated camera focal length fpx, the hand-eye calibration error converts approximately to 1.9 mm.

One of the key features of Twin-S is real-time updating of the anatomic model as the patient anatomy is ablated. We conduct an experiment to evaluate the accuracy of the drilling process. A surgical expert is invited to drill the surgical phantom similar to in vivo surgery, emulating the tissue removal during mastoidectomy.

We compare the updated phantom model of our digital twins after drilling with the 3D structure obtained by a CT scanner (Loop-X, Brainlab³). Point clouds are generated from both the virtual phantom model and the segmentation of the phantom in the CT scan and aligned using ICP. We visualize the mismatch between the virtual estimate of the tissue ablation and the true state of the phantom after the surgery as a heat map in Fig. 4. As the goal is to evaluate the model update accuracy of Twin-S, we report the error in drilled regions only instead of the entire model. The mean error of our virtual model in region A is 1.18 mm with a standard deviation of 0.24 mm, and the mean error in region B is 1.61 mm with the standard deviation of 0.49 mm. The average error of all drilled regions is 1.39 mm with a standard deviation of 0.62 mm. Errors are mostly located around sharp edges, which are affected to some extent by the limited spatial resolution of the CT scan. We further examine if there is correlation between the drilling depth and error, with qualitative results shown in Fig. 5. We do not observe strong correlations between the two variables qualitatively, and the p-value of 1 from our Chi-square test confirms that the variables are indeed statistically independent.

Overall, the results indicate that Twin-S can update the anatomical model with a precision comparable to conventional optical navigation systems [32], which is expected since spatial tracking in our digital twins largely relies on those systems. However, the average drilling depth of 1.57 mm is shallower than clinical practices [33]. The line-of-sight issues of the optical tracking system currently prohibit us from drilling deeper. It is our future work to optimize the spatial arrangement of the optical tracking markers to resolve the line-of-sight issue and expand our evaluation in deeper regions.

We analyze the tracking error to shed insight onto sources of error in the tracking performance. We denote the measured transformation as F*, which contains error ΔF and deviates from the actual transformation F: (2) F*=F⋅ΔF,whereΔF=I+sk(α→),ε→0,1.

The α is the rotational error using axis-angle notation, and sk⋅ is the skew-symmetric matrix. The ε→ is the translation error.

We present an analysis of the relative pose error between phantom and drill, which is critical for modeling tool-to-tissue interactions. The measured phantom-to-drill transformation is expressed as: (3) Fp*d=Fd*−1o⋅Fp*o,=Fdb*o⋅Fd*db−1⋅Fpb*o⋅Fp*pb.

In the following content, we use the rotational component as an example to illustrate our derivation and analysis. We expand the error terms of the rotation part and obtain: (4) Rp*d=Rd*−1db⋅Rdb*−1o⋅Rpb*o⋅Rp*pb,Rpd⋅I+skα→pd=Rddb⋅I+skα→ddb−1⋅Rdbo⋅I+skα→dbo−1⋅Rpbo⋅I+skα→pbo⋅Rppb⋅I+skα→ppb.

With re-arrangement, we can obtain four independent components contributing to the rotational error: (5) α→pd=β1⋅α→ppb+β2,⋅α→pbo+β3⋅α→dbo+β4⋅α→ddb,

where β1=I, β2=Rp−1pb, β3=−Rp−1pb⋅Rpb−1o⋅Rdbo, and β4=−Rp−1d. Intuitively, the norm of βi represents how much each measurement error contributes to the final inaccuracy.

To derive the numerical upper bound on tracking error, we use the worst-case tracking accuracy of 0.08 mm for each optical marker obtained from “opticalspstracking” Section. We also assume a worst-case rotation error of 1° for ICP as it is not directly obtainable. We obtain the estimated worst-case error norm as: α→pd2≤β1⋅α→ppb2+β2,⋅oα→pb2+β3⋅oα→db2 (6) +β4⋅α→ddb2,≤1.0+0.3+0.4+1.0=2.7∘.

The rotational error α→pd is up to 2.7° and dominated by α→ppb and α→ddb, which are the calibration inaccuracies. A similar analysis can be done on the translation error ε→pd, with the worst error bound of 8.6 mm. The estimated numerical upper bounds of the tracking error represent the worst-case scenario that Twin-S may produce. In practice, we observe the actual drilling error of 1.39 mm (“Drilling simulation accuracy“ section) is much smaller than the estimated upper bounds. Details of the translation error analysis are given in appendix.

As mentioned in “Modeling of the surgical camera“ section, Twin-S can generate per-pixel segmentation mask from the virtual models for downstream applications. Therefore, we evaluate the accuracy of virtually generated segmentation masks and report the Dice scores, where higher Dicer scores indicate better accuracy [34]. To obtain ground truth, we manually label a sequence of 100 frames of size 480 × 640. The Dice score for the drill is computed as 0.725, and the Dice score for the phantom is 0.956. The lower Dice score of the drill in comparison with the phantom is due to the thin shape of the drill, where slight offsets will lead to much lower Dice scores.

We present computation analysis of Twin-S as the real-time update is important for intra-operative uses. The calibration step is a requirement prior to launching the digital twins system such that the virtual models are aligned with the physical objects. Therefore, the calibration process does not impact the run-time performance of Twin-S. The time spent for calibration is approximately 10 min. We conduct a computation evaluation of Twin-S on a laptop of Ubuntu 20.04 equipped with Nvidia 3060 Laptop Graphic card and Intel i5 CPU. The breakdown of the computation time is presented in Table 1. The data synchronization step is the most time-consuming process of 20.5 ms, which aligns the timestamps and pairs the incoming tracking poses from the optical tracker and stereo images from the surgical camera. The overall run time of Twin-S is 35.7 ms (28 FPS).