Deep Learning Publications

Frisken SF. SurfaceNets for Multi-Label Segmentations with Preservation of Sharp Boundaries. J Comput Graph Tech. 2022;11 (1) :34-54.Abstract
We extend 3D SurfaceNets to generate surfaces of segmented 3D medical images composed of multiple materials represented as indexed labels. Our extension generates smooth, high-quality triangle meshes suitable for rendering and tetrahedralization, preserves topology and sharp boundaries between materials, guarantees a user-specified accuracy, and is fast enough that users can interactively explore the trade-off between accuracy and surface smoothness. We provide open-source code in the form of an extendable C++ library with a simple API, and a Qt and OpenGL-based application that allows users to import or randomly generate multi-label volumes to experiment with surface fairing parameters. In this paper, we describe the basic SurfaceNets algorithm, our extension to handle multiple materials, our method for preserving sharp boundaries between materials, and implementation details used to achieve efficient processing.
Sack J, Nitsch J, Meine H, Kikinis R, Halle M, Rutherford A. Quantitative Analysis of Liver Disease Using MRI-Based Radiomic Features of the Liver and Spleen. J Imaging. 2022;8 (10) :277.Abstract
BACKGROUND: Radiomics extracts quantitative image features to identify biomarkers for characterizing disease. Our aim was to characterize the ability of radiomic features extracted from magnetic resonance (MR) imaging of the liver and spleen to detect cirrhosis by comparing features from patients with cirrhosis to those without cirrhosis. METHODS: This retrospective study compared MR-derived radiomic features between patients with cirrhosis undergoing hepatocellular carcinoma screening and patients without cirrhosis undergoing intraductal papillary mucinous neoplasm surveillance between 2015 and 2018 using the same imaging protocol. Secondary analyses stratified the cirrhosis cohort by liver disease severity using clinical compensation/decompensation and Model for End-Stage Liver Disease (MELD). RESULTS: Of 167 patients, 90 had cirrhosis with 68.9% compensated and median MELD 8. Combined liver and spleen radiomic features generated an AUC 0.94 for detecting cirrhosis, with shape and texture components contributing more than size. Discrimination of cirrhosis remained high after stratification by liver disease severity. CONCLUSIONS: MR-based liver and spleen radiomic features had high accuracy in identifying cirrhosis, after stratification by clinical compensation/decompensation and MELD. Shape and texture features performed better than size features. These findings will inform radiomic-based applications for cirrhosis diagnosis and severity.
Bridge CP, Gorman C, Pieper S, Doyle SW, Lennerz JK, Kalpathy-Cramer J, Clunie DA, Fedorov AY, Herrmann MD. Highdicom: a Python Library for Standardized Encoding of Image Annotations and Machine Learning Model Outputs in Pathology and Radiology. J Digit Imaging. 2022.Abstract
Machine learning (ML) is revolutionizing image-based diagnostics in pathology and radiology. ML models have shown promising results in research settings, but the lack of interoperability between ML systems and enterprise medical imaging systems has been a major barrier for clinical integration and evaluation. The DICOM® standard specifies information object definitions (IODs) and services for the representation and communication of digital images and related information, including image-derived annotations and analysis results. However, the complexity of the standard represents an obstacle for its adoption in the ML community and creates a need for software libraries and tools that simplify working with datasets in DICOM format. Here we present the highdicom library, which provides a high-level application programming interface (API) for the Python programming language that abstracts low-level details of the standard and enables encoding and decoding of image-derived information in DICOM format in a few lines of Python code. The highdicom library leverages NumPy arrays for efficient data representation and ties into the extensive Python ecosystem for image processing and machine learning. Simultaneously, by simplifying creation and parsing of DICOM-compliant files, highdicom achieves interoperability with the medical imaging systems that hold the data used to train and run ML models, and ultimately communicate and store model outputs for clinical use. We demonstrate through experiments with slide microscopy and computed tomography imaging, that, by bridging these two ecosystems, highdicom enables developers and researchers to train and evaluate state-of-the-art ML models in pathology and radiology while remaining compliant with the DICOM standard and interoperable with clinical systems at all stages. To promote standardization of ML research and streamline the ML model development and deployment process, we made the library available free and open-source at .
Yu Y, Safdar S, Bourantas G, Zwick B, Joldes G, Kapur T, Frisken S, Kikinis R, Nabavi A, Golby A, et al. Automatic Framework for Patient-Specific Modelling of Tumour Resection-Induced Brain Shift. Comput Biol Med. 2022;143 :105271.Abstract
Our motivation is to enable non-biomechanical engineering specialists to use sophisticated biomechanical models in the clinic to predict tumour resection-induced brain shift, and subsequently know the location of the residual tumour and its boundary. To achieve this goal, we developed a framework for automatically generating and solving patient-specific biomechanical models of the brain. This framework automatically determines patient-specific brain geometry from MRI data, generates patient-specific computational grid, assigns material properties, defines boundary conditions, applies external loads to the anatomical structures, and solves differential equations of nonlinear elasticity using Meshless Total Lagrangian Explicit Dynamics (MTLED) algorithm. We demonstrated the effectiveness and appropriateness of our framework on real clinical cases of tumour resection-induced brain shift.
Giganti F, Cole AP, Fennessy FM, Clinton T, Moreira PLDF, Bernardes MC, Westin C-F, Krishnaswamy D, Fedorov A, Wollin DA, et al. Promoting the Use of the PI-QUAL Score for Prostate MRI Quality: Results From the ESOR Nicholas Gourtsoyiannis Teaching Fellowship. Eur Radiol. 2022 :1-11.Abstract
OBJECTIVES: The Prostate Imaging Quality (PI-QUAL) score is a new metric to evaluate the diagnostic quality of multiparametric magnetic resonance imaging (MRI) of the prostate. This study assesses the impact of an intervention, namely a prostate MRI quality training lecture, on the participant's ability to apply PI-QUAL. METHODS: Sixteen participants (radiologists, urologists, physicists, and computer scientists) of varying experience in reviewing diagnostic prostate MRI all assessed the image quality of ten examinations from different vendors and machines. Then, they attended a dedicated lecture followed by a hands-on workshop on MRI quality assessment using the PI-QUAL score. Five scans assessed by the participants were evaluated in the workshop using the PI-QUAL score for teaching purposes. After the course, the same participants evaluated the image quality of a new set of ten scans applying the PI-QUAL score. Results were assessed using receiver operating characteristic analysis. The reference standard was the PI-QUAL score assessed by one of the developers of PI-QUAL. RESULTS: There was a significant improvement in average area under the curve for the evaluation of image quality from baseline (0.59 [95 % confidence intervals: 0.50-0.66]) to post-teaching (0.96 [0.92-0.98]), an improvement of 0.37 [0.21-0.41] (p < 0.001). CONCLUSIONS: A teaching course (dedicated lecture + hands-on workshop) on PI-QUAL significantly improved the application of this scoring system to assess the quality of prostate MRI examinations. KEY POINTS: • A significant improvement in the application of PI-QUAL for the assessment of prostate MR image quality was observed after an educational intervention. • Appropriate training on image quality can be delivered to those involved in the acquisition and interpretation of prostate MRI. • Further investigation will be needed to understand the impact on improving the acquisition of high-quality diagnostic prostate MR examinations.
Zhang F, Wells WM, O'Donnell LJ. Deep Diffusion MRI Registration (DDMReg): A Deep Learning Method for Diffusion MRI Registration. IEEE Trans Med Imaging. 2022;41 (6) :1454-67.Abstract
In this paper, we present a deep learning method, DDMReg, for accurate registration between diffusion MRI (dMRI) datasets. In dMRI registration, the goal is to spatially align brain anatomical structures while ensuring that local fiber orientations remain consistent with the underlying white matter fiber tract anatomy. DDMReg is a novel method that uses joint whole-brain and tract-specific information for dMRI registration. Based on the successful VoxelMorph framework for image registration, we propose a novel registration architecture that leverages not only whole brain information but also tract-specific fiber orientation information. DDMReg is an unsupervised method for deformable registration between pairs of dMRI datasets: it does not require nonlinearly pre-registered training data or the corresponding deformation fields as ground truth. We perform comparisons with four state-of-the-art registration methods on multiple independently acquired datasets from different populations (including teenagers, young and elderly adults) and different imaging protocols and scanners. We evaluate the registration performance by assessing the ability to align anatomically corresponding brain structures and ensure fiber spatial agreement between different subjects after registration. Experimental results show that DDMReg obtains significantly improved registration performance compared to the state-of-the-art methods. Importantly, we demonstrate successful generalization of DDMReg to dMRI data from different populations with varying ages and acquired using different acquisition protocols and different scanners.
McGarry SD, Brehler M, Bukowy JD, Lowman AK, Bobholz SA, Duenweg SR, Banerjee A, Hurrell SL, Malyarenko D, Chenevert TL, et al. Multi-Site Concordance of Diffusion-Weighted Imaging Quantification for Assessing Prostate Cancer Aggressiveness. J Magn Reson Imaging. 2022;55 (6) :1745-58.Abstract
BACKGROUND: Diffusion-weighted imaging (DWI) is commonly used to detect prostate cancer, and a major clinical challenge is differentiating aggressive from indolent disease. PURPOSE: To compare 14 site-specific parametric fitting implementations applied to the same dataset of whole-mount pathologically validated DWI to test the hypothesis that cancer differentiation varies with different fitting algorithms. STUDY TYPE: Prospective. POPULATION: Thirty-three patients prospectively imaged prior to prostatectomy. FIELD STRENGTH/SEQUENCE: 3 T, field-of-view optimized and constrained undistorted single-shot DWI sequence. ASSESSMENT: Datasets, including a noise-free digital reference object (DRO), were distributed to the 14 teams, where locally implemented DWI parameter maps were calculated, including mono-exponential apparent diffusion coefficient (MEADC), kurtosis (K), diffusion kurtosis (DK), bi-exponential diffusion (BID), pseudo-diffusion (BID*), and perfusion fraction (F). The resulting parametric maps were centrally analyzed, where differentiation of benign from cancerous tissue was compared between DWI parameters and the fitting algorithms with a receiver operating characteristic area under the curve (ROC AUC). STATISTICAL TEST: Levene's test, P < 0.05 corrected for multiple comparisons was considered statistically significant. RESULTS: The DRO results indicated minimal discordance between sites. Comparison across sites indicated that K, DK, and MEADC had significantly higher prostate cancer detection capability (AUC range = 0.72-0.76, 0.76-0.81, and 0.76-0.80 respectively) as compared to bi-exponential parameters (BID, BID*, F) which had lower AUC and greater between site variation (AUC range = 0.53-0.80, 0.51-0.81, and 0.52-0.80 respectively). Post-processing parameters also affected the resulting AUC, moving from, for example, 0.75 to 0.87 for MEADC varying cluster size. DATA CONCLUSION: We found that conventional diffusion models had consistent performance at differentiating prostate cancer from benign tissue. Our results also indicated that post-processing decisions on DWI data can affect sensitivity and specificity when applied to radiological-pathological studies in prostate cancer. LEVEL OF EVIDENCE: 1 TECHNICAL EFFICACY: Stage 3.
Wang S, Zhang F, Huang P, Hong H, Jiaerken Y, Yu X, Zhang R, Zeng Q, Zhang Y, Kikinis R, et al. Superficial White Matter Microstructure Affects Processing Speed in Cerebral Small Vessel Disease. Hum Brain Mapp. 2022;43 (17) :5310-25.Abstract
White matter hyperintensities (WMH) are a typical feature of cerebral small vessel disease (CSVD), which contributes to about 50% of dementias worldwide. Microstructural alterations in deep white matter (DWM) have been widely examined in CSVD. However, little is known about abnormalities in superficial white matter (SWM) and their relevance for processing speed, the main cognitive deficit in CSVD. In 141 CSVD patients, processing speed was assessed using Trail Making Test Part A. White matter abnormalities were assessed by WMH burden (volume on T2-FLAIR) and diffusion MRI measures. SWM imaging measures had a large contribution to processing speed, despite a relatively low SWM WMH burden. Across all imaging measures, SWM free water (FW) had the strongest association with processing speed, followed by SWM mean diffusivity (MD). SWM FW was the only marker to significantly increase between two subgroups with the lowest WMH burdens. When comparing two subgroups with the highest WMH burdens, the involvement of WMH in the SWM was accompanied by significant differences in processing speed and white matter microstructure. Mediation analysis revealed that SWM FW fully mediated the association between WMH volume and processing speed, while no mediation effect of MD or DWM FW was observed. Overall, results suggest that the SWM has an important contribution to processing speed, while SWM FW is a sensitive imaging marker associated with cognition in CSVD. This study extends the current understanding of CSVD-related dysfunction and suggests that the SWM, as an understudied region, can be a potential target for monitoring pathophysiological processes.
Pujol S, Cabeen RP, Yelnik J, François C, Fernandez Vidal S, Karachi C, Bardinet E, Cosgrove RG, Kikinis R. Somatotopic Organization of Hyperdirect Pathway Projections From the Primary Motor Cortex in the Human Brain. Front Neurol. 2022;13 :791092.Abstract
Background: The subthalamic nucleus (STN) is an effective neurosurgical target to improve motor symptoms in Parkinson's Disease (PD) patients. MR-guided Focused Ultrasound (MRgFUS) subthalamotomy is being explored as a therapeutic alternative to Deep Brain Stimulation (DBS) of the STN. The hyperdirect pathway provides a direct connection between the cortex and the STN and is likely to play a key role in the therapeutic effects of MRgFUS intervention in PD patients. Objective: This study aims to investigate the topography and somatotopy of hyperdirect pathway projections from the primary motor cortex (M1). Methods: We used advanced multi-fiber tractography and high-resolution diffusion MRI data acquired on five subjects of the Human Connectome Project (HCP) to reconstruct hyperdirect pathway projections from M1. Two neuroanatomy experts reviewed the anatomical accuracy of the tracts. We extracted the fascicles arising from the trunk, arm, hand, face and tongue area from the reconstructed pathways. We assessed the variability among subjects based on the fractional anisotropy (FA) and mean diffusivity (MD) of the fibers. We evaluated the spatial arrangement of the different fascicles using the Dice Similarity Coefficient (DSC) of spatial overlap and the centroids of the bundles. Results: We successfully reconstructed hyperdirect pathway projections from M1 in all five subjects. The tracts were in agreement with the expected anatomy. We identified hyperdirect pathway fascicles projecting from the trunk, arm, hand, face and tongue area in all subjects. Tract-derived measurements showed low variability among subjects, and similar distributions of FA and MD values among the fascicles projecting from different M1 areas. We found an anterolateral somatotopic arrangement of the fascicles in the corona radiata, and an average overlap of 0.63 in the internal capsule and 0.65 in the zona incerta. Conclusion: Multi-fiber tractography combined with high-resolution diffusion MRI data enables the identification of the somatotopic organization of the hyperdirect pathway. Our preliminary results suggest that the subdivisions of the hyperdirect pathway projecting from the trunk, arm, hand, face, and tongue motor area are intermixed at the level of the zona incerta and posterior limb of the internal capsule, with a predominantly overlapping topographical organization in both regions. Subject-specific knowledge of the hyperdirect pathway somatotopy could help optimize target definition in MRgFUS intervention.
Cole AP, Langbein BJ, Giganti F, Fennessy FM, Tempany CM, Emberton M. Is Perfect the Enemy of Good? Weighing the Evidence for Biparametric MRI in Prostate Cancer. Br J Radiol. 2022;95 (1131) :20210840.Abstract
The role of multiparametric MRI in diagnosis, staging and treatment planning for prostate cancer is well established. However, there remain several challenges to widespread adoption. One such challenge is the duration and cost of the examination. Abbreviated exams omitting contrast-enhanced sequences may help address this challenge. In this review, we will discuss the rationale for biparametric MRI for detection and characterization of clinically significant prostate cancer prior to biopsy and synthesize the published literature. We will weigh up the advantages and disadvantages to this approach and lay out a conceptual cost/benefit analysis regarding adoption of biparametric MRI.
Zheng J, Yang Q, Makris N, Huang K, Liang J, Ye C, Yu X, Tian M, Ma T, Mou T, et al. Three-Dimensional Digital Reconstruction of the Cerebellar Cortex: Lobule Thickness, Surface Area Measurements, and Layer Architecture. Cerebellum. 2022.Abstract
The cerebellum is ontogenetically one of the first structures to develop in the central nervous system; nevertheless, it has been only recently reconsidered for its significant neurobiological, functional, and clinical relevance in humans. Thus, it has been a relatively under-studied compared to the cerebrum. Currently, non-invasive imaging modalities can barely reach the necessary resolution to unfold its entire, convoluted surface, while only histological analyses can reveal local information at the micrometer scale. Herein, we used the BigBrain dataset to generate area and point-wise thickness measurements for all layers of the cerebellar cortex and for each lobule in particular. We found that the overall surface area of the cerebellar granular layer (including Purkinje cells) was 1,732 cm2 and the molecular layer was 1,945 cm2. The average thickness of the granular layer is 0.88 mm (± 0.83) and that of the molecular layer is 0.32 mm (± 0.08). The cerebellum (both granular and molecular layers) is thicker at the depth of the sulci and thinner at the crowns of the gyri. Globally, the granular layer is thicker in the lateral-posterior-inferior region than the medial-superior regions. The characterization of individual layers in the cerebellum achieved herein represents a stepping-stone for investigations interrelating structural and functional connectivity with cerebellar architectonics using neuroimaging, which is a matter of considerable relevance in basic and clinical neuroscience. Furthermore, these data provide templates for the construction of cerebellar topographic maps and the precise localization of structural and functional alterations in diseases affecting the cerebellum.
Maier-Hein L, Eisenmann M, Sarikaya D, März K, Collins T, Malpani A, Fallert J, Feussner H, Giannarou S, Mascagni P, et al. Surgical Data Science - From Concepts Toward Clinical Translation. Med Image Anal. 2022;76 :102306.Abstract
Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.
Zekelman LR, Zhang F, Makris N, He J, Chen Y, Xue T, Liera D, Drane DL, Rathi Y, Golby AJ, et al. White Matter Association Tracts Underlying Language and Theory of Mind: An Investigation of 809 Brains from the Human Connectome Project. Neuroimage. 2022;246 :118739.Abstract
Language and theory of mind (ToM) are the cognitive capacities that allow for the successful interpretation and expression of meaning. While functional MRI investigations are able to consistently localize language and ToM to specific cortical regions, diffusion MRI investigations point to an inconsistent and sometimes overlapping set of white matter tracts associated with these two cognitive domains. To further examine the white matter tracts that may underlie these domains, we use a two-tensor tractography method to investigate the white matter microstructure of 809 participants from the Human Connectome Project. 20 association white matter tracts (10 in each hemisphere) are uniquely identified by leveraging a neuroanatomist-curated automated white matter tract atlas. The mean fractional anisotropy (FA), mean diffusivity (MD), and number of streamlines (NoS) are measured for each white matter tract. Performance on neuropsychological assessments of semantic memory (NIH Toolbox Picture Vocabulary Test, TPVT) and emotion perception (Penn Emotion Recognition Test, PERT) are used to measure critical subcomponents of the language and ToM networks, respectively. Regression models are constructed to examine how structural measurements of left and right white matter tracts influence performance across these two assessments. We find that semantic memory performance is influenced by the number of streamlines of the left superior longitudinal fasciculus III (SLF-III), and emotion perception performance is influenced by the number of streamlines of the right SLF-III. Additionally, we find that performance on both semantic memory & emotion perception is influenced by the FA of the left arcuate fasciculus (AF). The results point to multiple, overlapping white matter tracts that underlie the cognitive domains of language and ToM. Results are discussed in terms of hemispheric dominance and concordance with prior investigations.
Robles DJ, Dharani A, Rostowsky KA, Chaudhari NN, Ngo V, Zhang F, O'Donnell LJ, Green L, Sheikh-Bahaei N, Chui HC, et al. Older Age, Male Sex, and Cerebral Microbleeds Predict White Matter Loss After Traumatic Brain Injury. Geroscience. 2022;44 (1) :83-102.Abstract
Little is known on how mild traumatic brain injury affects white matter based on age at injury, sex, cerebral microbleeds, and time since injury. Here, we study the fractional anisotropy of white matter to study these effects in 109 participants aged 18-77 (46 females, age μ ± σ = 40 ± 17 years) imaged within [Formula: see text] 1 week and [Formula: see text] 6 months post-injury. Age is found to be linearly associated with white matter degradation, likely due not only to injury but also to cumulative effects of other pathologies and to their interactions with injury. Age is associated with mean anisotropy decreases in the corpus callosum, middle longitudinal fasciculi, inferior longitudinal and occipitofrontal fasciculi, and superficial frontal and temporal fasciculi. Over [Formula: see text] 6 months, the mean anisotropies of the corpus callosum, left superficial frontal fasciculi, and left corticospinal tract decrease significantly. Independently of other predictors, age and cerebral microbleeds contribute to anisotropy decrease in the callosal genu. Chronically, the white matter of commissural tracts, left superficial frontal fasciculi, and left corticospinal tract degrade appreciably, independently of other predictors. Our findings suggest that large commissural and intra-hemispheric structures are at high risk for post-traumatic degradation. This study identifies detailed neuroanatomic substrates consistent with brain injury patients' age-dependent deficits in information processing speed, interhemispheric communication, motor coordination, visual acuity, sensory integration, reading speed/comprehension, executive function, personality, and memory. We also identify neuroanatomic features underlying white matter degradation whose severity is associated with the male sex. Future studies should compare our findings to functional measures and other neurodegenerative processes.
Yu Y, Bourantas G, Zwick B, Joldes G, Kapur T, Frisken S, Kikinis R, Nabavi A, Golby A, Wittek A, et al. Computer Simulation of Tumour Resection-Induced Brain Deformation by a Meshless Approach. Int J Numer Method Biomed Eng. 2022;38 (1) :e3539.Abstract
Tumour resection requires precise planning and navigation to maximise tumour removal while simultaneously protecting nearby healthy tissues. Neurosurgeons need to know the location of the remaining tumour after partial tumour removal before continuing with the resection. Our approach to the problem uses biomechanical modelling and computer simulation to compute the brain deformations after the tumour is resected. In this study, we use Meshless Total Lagrangian Explicit Dynamics (MTLED) as the solver. The problem geometry is extracted from the patient-specific MRI data and includes the parenchyma, tumour, cerebrospinal fluid (CSF) and skull. Appropriate nonlinear material formulation is used. Loading is performed by imposing intra-operative conditions of gravity and reaction forces between the tumour and surrounding healthy parenchyma tissues. A finite frictionless sliding contact is enforced between the skull (rigid) and parenchyma. The meshless simulation results are compared to intra-operative MRI sections. We also calculate Hausdorff distances between the computed deformed surfaces (ventricles and tumour cavities) and surfaces observed intra-operatively. Over 80% of points on ventricle surface and 95% of points on tumour cavity surface were successfully registered (results within the limits of two times the original in-plane resolution of the intra-operative image). Computed results demonstrate the potential for our method in estimating the tissue deformation and tumour boundary during the resection. This article is protected by copyright. All rights reserved.
Mehrtash A, Kapur T, Tempany CM, Abolmaesumi P, Wells WM. Prostate Cancer Diagnosis With Sparse Biopsy Data and in Presence of Location Uncertainty. Proc IEEE Int Symp Biomed Imaging. 2021;2021 :443-447.Abstract
Prostate cancer is the second most prevalent cancer in men worldwide. Deep neural networks have been successfully applied for prostate cancer diagnosis in magnetic resonance images (MRI). Pathology results from biopsy procedures are often used as ground truth to train such systems. There are several sources of noise in creating ground truth from biopsy data including sampling and registration errors. We propose: 1) A fully convolutional neural network (FCN) to produce cancer probability maps across the whole prostate gland in MRI; 2) A Gaussian weighted loss function to train the FCN with sparse biopsy locations; 3) A probabilistic framework to model biopsy location uncertainty and adjust cancer probability given the deep model predictions. We assess the proposed method on 325 biopsy locations from 203 patients. We observe that the proposed loss improves the area under the receiver operating characteristic curve and the biopsy location adjustment improves the sensitivity of the models.
Liao R, Moyer D, Cha M, Quigley K, Berkowitz S, Horng S, Golland P, Wells WM. Multimodal Representation Learning via Maximization of Local Mutual Information. Med Image Comput Comput Assist Interv. 2021;12902 :273-83.Abstract
We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at:
Balagurunathan Y, Beers A, Mcnitt-Gray M, Hadjiiski L, Napel S, Goldgof D, Perez G, Arbelaez P, Mehrtash A, Kapur T, et al. Lung Nodule Malignancy Prediction in Sequential CT Scans: Summary of ISBI 2018 Challenge. IEEE Trans Med Imaging. 2021;40 (12) :3748-61.Abstract
Lung cancer is by far the leading cause of cancer death in the US. Recent studies have demonstrated the effectiveness of screening using low dose CT (LDCT) in reducing lung cancer related mortality. While lung nodules are detected with a high rate of sensitivity, this exam has a low specificity rate and it is still difficult to separate benign and malignant lesions. The ISBI 2018 Lung Nodule Malignancy Prediction Challenge, developed by a team from the Quantitative Imaging Network of the National Cancer Institute, was focused on the prediction of lung nodule malignancy from two sequential LDCT screening exams using automated (non-manual) algorithms. We curated a cohort of 100 subjects who participated in the National Lung Screening Trial and had established pathological diagnoses. Data from 30 subjects were randomly selected for training and the remaining was used for testing. Participants were evaluated based on the area under the receiver operating characteristic curve (AUC) of nodule-wise malignancy scores generated by their algorithms on the test set. The challenge had 17 participants, with 11 teams submitting reports with method description, mandated by the challenge rules. Participants used quantitative methods, resulting in a reporting test AUC ranging from 0.698 to 0.913. The top five contestants used deep learning approaches, reporting an AUC between 0.87 - 0.91. The team's predictor did not achieve significant differences from each other nor from a volume change estimate (p =.05 with Bonferroni-Holm's correction).
Xu Z, Yan J, Luo J, Wells W, Li X, Jagadeesan J. Unimodal Cyclic Regularization for Training Multimodal Image Registration Networks. Proc IEEE Int Symp Biomed Imaging. 2021;2021.Abstract
The loss function of an unsupervised multimodal image registration framework has two terms, i.e., a metric for similarity measure and regularization. In the deep learning era, researchers proposed many approaches to automatically learn the similarity metric, which has been shown effective in improving registration performance. However, for the regularization term, most existing multimodal registration approaches still use a hand-crafted formula to impose artificial properties on the estimated deformation field. In this work, we propose a unimodal cyclic regularization training pipeline, which learns task-specific prior knowledge from simpler unimodal registration, to constrain the deformation field of multimodal registration. In the experiment of abdominal CT-MR registration, the proposed method yields better results over conventional regularization methods, especially for severely deformed local regions.
Drakopoulos F, Tsolakis C, Angelopoulos A, Liu Y, Yao C, Kavazidi KR, Foroglou N, Fedorov A, Frisken S, Kikinis R, et al. Adaptive Physics-Based Non-Rigid Registration for Immersive Image-Guided Neuronavigation Systems. Front Digit Health. 2021;2 :613608.Abstract
Objective: In image-guided neurosurgery, co-registered preoperative anatomical, functional, and diffusion tensor imaging can be used to facilitate a safe resection of brain tumors in eloquent areas of the brain. However, the brain deforms during surgery, particularly in the presence of tumor resection. Non-Rigid Registration (NRR) of the preoperative image data can be used to create a registered image that captures the deformation in the intraoperative image while maintaining the quality of the preoperative image. Using clinical data, this paper reports the results of a comparison of the accuracy and performance among several non-rigid registration methods for handling brain deformation. A new adaptive method that automatically removes mesh elements in the area of the resected tumor, thereby handling deformation in the presence of resection is presented. To improve the user experience, we also present a new way of using mixed reality with ultrasound, MRI, and CT. Materials and methods: This study focuses on 30 glioma surgeries performed at two different hospitals, many of which involved the resection of significant tumor volumes. An Adaptive Physics-Based Non-Rigid Registration method (A-PBNRR) registers preoperative and intraoperative MRI for each patient. The results are compared with three other readily available registration methods: a rigid registration implemented in 3D Slicer v4.4.0; a B-Spline non-rigid registration implemented in 3D Slicer v4.4.0; and PBNRR implemented in ITKv4.7.0, upon which A-PBNRR was based. Three measures were employed to facilitate a comprehensive evaluation of the registration accuracy: (i) visual assessment, (ii) a Hausdorff Distance-based metric, and (iii) a landmark-based approach using anatomical points identified by a neurosurgeon. Results: The A-PBNRR using multi-tissue mesh adaptation improved the accuracy of deformable registration by more than five times compared to rigid and traditional physics based non-rigid registration, and four times compared to B-Spline interpolation methods which are part of ITK and 3D Slicer. Performance analysis showed that A-PBNRR could be applied, on average, in <2 min, achieving desirable speed for use in a clinical setting. Conclusions: The A-PBNRR method performed significantly better than other readily available registration methods at modeling deformation in the presence of resection. Both the registration accuracy and performance proved sufficient to be of clinical value in the operating room. A-PBNRR, coupled with the mixed reality system, presents a powerful and affordable solution compared to current neuronavigation systems.