TRD 3: Intraoperative Devices

N Hata Jay Oliver Jonas K Tuncali
Nobuhiko Hata PhD Jayender Jagadeesan PhD Oliver Jonas PhD Kemal Tuncali MD

Led by Nobuhiko Hata, the focus areas of the Intraoperative Devices TRD are:

  • An Intraoperative needle guidance system for prostate biopsy that models needle deflection and derives optimal insertion paths for in-bore MRI-guided prostate interventions. 

  • An augmented reality guided navigation system for thoracoscopic lung surgery with compensation for lung deformation.

  • Implantable microdevices with integrated retrievability for difficult-to-access tissues, such as the brain.

Select Publications

Zhou H, Jayender J. EMDQ: Removal of Image Feature Mismatches in Real-Time. IEEE Trans Image Process. 2022;31 :706-20.Abstract
This paper proposes a novel method for removing image feature mismatches in real-time that can handle both rigid and smooth deforming environments. Image distortion, parallax and object deformation may cause the pixel coordinates of feature matches to have non-rigid deformations, which cannot be represented using a single analytical rigid transformation. To solve this problem, we propose an algorithm based on the re-weighting and 1-point RANSAC strategy (R1P-RNSC), which operates under the assumption that a non-rigid deformation can be approximately represented by multiple rigid transformations. R1P-RNSC is fast but suffers from the drawback that local smoothing information cannot be considered, thus limiting its accuracy. To solve this problem, we propose a non-parametric algorithm based on the expectation-maximization algorithm and the dual quaternion-based representation (EMDQ). EMDQ generates dense and smooth deformation fields by interpolating among the feature matches, simultaneously removing mismatches that are inconsistent with the deformation field. It relies on the rigid transformations obtained by R1P-RNSC to improve its accuracy. The experimental results demonstrate that EMDQ has superior accuracy compared to other state-of-the-art mismatch removal methods. The ability to build correspondences for all image pixels using the dense deformation field is another contribution of this paper.
Dominas C, Bhagavatula S, Stover EH, Deans K, Larocca C, Colson YL, Peruzzi PP, Kibel AS, Hata N, Tsai LL, et al. The Translational and Regulatory Development of an Implantable Microdevice for Multiple Drug Sensitivity Measurements in Cancer Patients. IEEE Trans Biomed Eng. 2022;69 (1) :412-21.Abstract
OBJECTIVE: The purpose of this article is to report the translational process of an implantable microdevice platform with an emphasis on the technical and engineering adaptations for patient use, regulatory advances, and successful integration into clinical workflow. METHODS: We developed design adaptations for implantation and retrieval, established ongoing monitoring and testing, and facilitated regulatory advances that enabled the administration and examination of a large set of cancer therapies simultaneously in individual patients. RESULTS: Six applications for oncology studies have successfully proceeded to patient trials, with future applications in progress. CONCLUSION: First-in-human translation required engineering design changes to enable implantation and retrieval that fit with existing clinical workflows, a regulatory strategy that enabled both delivery and response measurement of up to 20 agents in a single patient, and establishment of novel testing and quality control processes for a drug/device combination product without clear precedents. SIGNIFICANCE: This manuscript provides a real-world account and roadmap on how to advance from animal proof-of-concept into the clinic, confronting the question of how to use research to benefit patients.
Banach A, King F, Masaki F, Tsukada H, Hata N. Visually Navigated Bronchoscopy Using Three Cycle-Consistent Generative Adversarial Network for Depth Estimation. Med Image Anal. 2021;73 :102164.Abstract
[Background] Electromagnetically Navigated Bronchoscopy (ENB) is currently the state-of-the art diagnostic and interventional bronchoscopy. CT-to-body divergence is a critical hurdle in ENB, causing navigation error and ultimately limiting the clinical efficacy of diagnosis and treatment. In this study, Visually Navigated Bronchoscopy (VNB) is proposed to address the aforementioned issue of CT-to-body divergence. [Materials and Methods] We extended and validated an unsupervised learning method to generate a depth map directly from bronchoscopic images using a Three Cycle-Consistent Generative Adversarial Network (3cGAN) and registering the depth map to preprocedural CTs. We tested the working hypothesis that the proposed VNB can be integrated to the navigated bronchoscopic system based on 3D Slicer, and accurately register bronchoscopic images to pre-procedural CTs to navigate transbronchial biopsies. The quantitative metrics to asses the hypothesis we set was Absolute Tracking Error (ATE) of the tracking and the Target Registration Error (TRE) of the total navigation system. We validated our method on phantoms produced from the pre-procedural CTs of five patients who underwent ENB and on two ex-vivo pig lung specimens. [Results] The ATE using 3cGAN was 6.2 +/- 2.9 [mm]. The ATE of 3cGAN was statistically significantly lower than that of cGAN, particularly in the trachea and lobar bronchus (p < 0.001). The TRE of the proposed method had a range of 11.7 to 40.5 [mm]. The TRE computed by 3cGAN was statistically significantly smaller than those computed by cGAN in two of the five cases enrolled (p < 0.05). [Conclusion] VNB, using 3cGAN to generate the depth maps was technically and clinically feasible. While the accuracy of tracking by cGAN was acceptable, the TRE warrants further investigation and improvement.
Zhou H, Jayender J. Real-Time Nonrigid Mosaicking of Laparoscopy Images. IEEE Trans Med Imaging. 2021;40 (6) :1726-36.Abstract
The ability to extend the field of view of laparoscopy images can help the surgeons to obtain a better understanding of the anatomical context. However, due to tissue deformation, complex camera motion and significant three-dimensional (3D) anatomical surface, image pixels may have non-rigid deformation and traditional mosaicking methods cannot work robustly for laparoscopy images in real-time. To solve this problem, a novel two-dimensional (2D) non-rigid simultaneous localization and mapping (SLAM) system is proposed in this paper, which is able to compensate for the deformation of pixels and perform image mosaicking in real-time. The key algorithm of this 2D non-rigid SLAM system is the expectation maximization and dual quaternion (EMDQ) algorithm, which can generate smooth and dense deformation field from sparse and noisy image feature matches in real-time. An uncertainty-based loop closing method has been proposed to reduce the accumulative errors. To achieve real-time performance, both CPU and GPU parallel computation technologies are used for dense mosaicking of all pixels. Experimental results on in vivo and synthetic data demonstrate the feasibility and accuracy of our non-rigid mosaicking method.
Wang D, Zhang T, Li M, Bueno R, Jayender J. 3D Deep Learning Based Classification of Pulmonary Ground Glass Opacity Nodules With Automatic Segmentation. Comput Med Imaging Graph. 2021;88 :101814.Abstract
Classifying ground-glass lung nodules (GGNs) into atypical adenomatous hyperplasia (AAH), adenocarcinoma in situ (AIS), minimally invasive adenocarcinoma (MIA), and invasive adenocarcinoma (IAC) on diagnostic CT images is important to evaluate the therapy options for lung cancer patients. In this paper, we propose a joint deep learning model where the segmentation can better facilitate the classification of pulmonary GGNs. Based on our observation that masking the nodule to train the model results in better lesion classification, we propose to build a cascade architecture with both segmentation and classification networks. The segmentation model works as a trainable preprocessing module to provide the classification-guided 'attention' weight map to the raw CT data to achieve better diagnosis performance. We evaluate our proposed model and compare with other baseline models for 4 clinically significant nodule classification tasks, defined by a combination of pathology types, using 4 classification metrics: Accuracy, Average F1 Score, Matthews Correlation Coefficient (MCC), and Area Under the Receiver Operating Characteristic Curve (AUC). Experimental results show that the proposed method outperforms other baseline models on all the diagnostic classification tasks.
Wang D, Li M, Ben-Shlomo N, Corrales EC, Cheng Y, Zhang T, Jayender J. A Novel Dual-Network Architecture for Mixed-Supervised Medical Image Segmentation. Comput Med Imaging Graph. 2021;89 :101841.Abstract
In medical image segmentation tasks, deep learning-based models usually require densely and precisely annotated datasets to train, which are time-consuming and expensive to prepare. One possible solution is to train with the mixed-supervised dataset, where only a part of data is densely annotated with segmentation map and the rest is annotated with some weak form, such as bounding box. In this paper, we propose a novel network architecture called Mixed-Supervised Dual-Network (MSDN), which consists of two separate networks for the segmentation and detection tasks respectively, and a series of connection modules between the layers of the two networks. These connection modules are used to extract and transfer useful information from the detection task to help the segmentation task. We exploit a variant of a recently designed technique called 'Squeeze and Excitation' in the connection module to boost the information transfer between the two tasks. Compared with existing model with shared backbone and multiple branches, our model has flexible and trainable feature sharing fashion and thus is more effective and stable. We conduct experiments on 4 medical image segmentation datasets, and experiment results show that the proposed MSDN model outperforms multiple baselines.
Xu Z, Luo J, Yan J, Li X, Jayender J. F3RNet: Full-resolution Residual Registration Network for Deformable Image Registration. Int J Comput Assist Radiol Surg. 2021;16 (6) :923-32.Abstract
PURPOSE: Deformable image registration (DIR) is essential for many image-guided therapies. Recently, deep learning approaches have gained substantial popularity and success in DIR. Most deep learning approaches use the so-called mono-stream high-to-low, low-to-high network structure and can achieve satisfactory overall registration results. However, accurate alignments for some severely deformed local regions, which are crucial for pinpointing surgical targets, are often overlooked. Consequently, these approaches are not sensitive to some hard-to-align regions, e.g., intra-patient registration of deformed liver lobes. METHODS: We propose a novel unsupervised registration network, namely full-resolution residual registration network (F3RNet), for deformable registration of severely deformed organs. The proposed method combines two parallel processing streams in a residual learning fashion. One stream takes advantage of the full-resolution information that facilitates accurate voxel-level registration. The other stream learns the deep multi-scale residual representations to obtain robust recognition. We also factorize the 3D convolution to reduce the training parameters and enhance network efficiency. RESULTS: We validate the proposed method on a clinically acquired intra-patient abdominal CT-MRI dataset and a public inspiratory and expiratory thorax CT dataset. Experiments on both multimodal and unimodal registration demonstrate promising results compared to state-of-the-art approaches. CONCLUSION: By combining the high-resolution information and multi-scale representations in a highly interactive residual learning fashion, the proposed F3RNet can achieve accurate overall and local registration. The run time for registering a pair of images is less than 3 s using a GPU. In future works, we will investigate how to cost-effectively process high-resolution information and fuse multi-scale representations.
Moreira P, Grimble J, Iftimia N, Bay CP, Tuncali K, Park J, Tokuda J. In Vivo Evaluation of Angulated Needle-Guide Template for MRI-Guided Transperineal Prostate Biopsy. Med Phys. 2021;48 (5) :2553-65.Abstract
PURPOSE: Magnetic resonance imaging (MRI)-guided transperineal prostate biopsy has been practiced since the early 2000s. The technique often suffers from targeting error due to deviation of the needle as a result of physical interaction between the needle and inhomogeneous tissues. Existing needle guide devices, such as a grid template, do not allow choosing an alternative insertion path to mitigate the deviation because of their limited degree-of-freedom (DoF). This study evaluates how an angulated needle insertion path can reduce needle deviation and improve needle placement accuracy. METHODS: We extended a robotic needle-guidance device (Smart Template) for in-bore MRI-guided transperineal prostate biopsy. The new Smart Template has a 4-DoF needle-guiding mechanism allowing a translational range of motion of 65 and 58 mm along the vertical and horizontal axis, and a needle rotational motion around the vertical and horizontal axis and a vertical rotational range of , respectively. We defined a path planning strategy, which chooses between straight and angulated insertion paths depending on the anatomical structures on the potential insertion path. We performed (a) a set of experiments to evaluate the device positioning accuracy outside the MR-bore, and (b) an in vivo experiment to evaluate the improvement of targeting accuracy combining straight and angulated insertions in animal models (swine, ). RESULTS: We analyzed 46 in vivo insertions using either straight or angulated insertions paths. The experiment showed that the proposed strategy of selecting straight or angulated insertions based on the subject's anatomy outperformed the conventional approach of just straight insertions in terms of targeting accuracy (2.4 mm [1.3-3.7] vs 3.9 mm [2.4-5.0] {Median ); p = 0.041 after the bias correction). CONCLUSION: The in vivo experiment successfully demonstrated that an angulated needle insertion path could improve needle placement accuracy with a path planning strategy that takes account of the subject-specific anatomical structures.
Moreira P, Tuncali K, Tempany CM, Tokuda J. The Impact of Placement Errors on the Tumor Coverage in MRI-Guided Focal Cryoablation of Prostate Cancer. Acad Radiol. 2021;28 (6) :841-8.Abstract
RATIONALE AND OBJECTIVES: There have been multiple investigations defining and reporting the effectiveness of focal cryoablation as a treatment option for organ-confined prostate cancer. However, the impact of cryo-needle/probe placement accuracy within the tumor and gland has not been extensively studied. We analyzed how variations in the placement of the cryo-needles, specifically errors leading to incomplete ablation, may affect prostate cancer's resulting cryoablation. MATERIALS AND METHODS: We performed a study based on isothermal models using Monte Carlo simulations to analyze the impact of needle placement errors on tumor coverage and the probability of positive ablation margin. We modeled the placement error as a Gaussian noise on the cryo-needle position. The analysis used retrospective MRI data of 15 patients with biopsy-proven, unifocal, and MRI visible prostate cancer to calculate the impact of placement error on the volume of the tumor encompassed by the -40°C and -20°C isotherms using one to four cryo-needles. RESULTS: When the standard deviation of the placement error reached 3 mm, the tumor coverage was still above 97% with the -20°C isotherm, and above 81% with the -40°C isotherm using two cryo-needles or more. The probability of positive margin was significantly lower considering the -20°C isotherm (0.04 for three needles) than using the -40°C isotherm (0.66 for three needles). CONCLUSION: The results indicated that accurate cryo-needle placement is essential for the success of focal cryoablation of prostate cancer. The analysis shows that an admissible targeting error depends on the lethal temperature considered and the number of cryo-needles used.
More