Interleaved Text/Image Deep Mining on a Large-Scale Radiology Image Database

Authors

• H-C Shin
• L Lu
• L Kim
• A Seff
• J Yao
• RM Summers

Abstract

Various patient data of a large population are available at the NIH. However such data are not widely studied, due to the challenges encountered in analyzing a large dataset. Nonetheless, efficient analysis of large data can lead us to gain useful, possibly unpreceded insights in the area under study. With the big-data analysis on large collection of radiology images, we aim to achieve ‘predictive medicine’ – detecting diseases with large population patient image screening. In an attempt to achieve this, we collected 780,000 radiology reports from the patient archive and communication system at the NIH, comprising about 1 billion words. However, manually examining and annotating these is not only challenging, but also requires an expertise in radiology. To fill in this gap, we used a non-parametric topic-modeling algorithm, to analyze the large collection of reports and to divide them into a number of categories with semantic hierarchies. We then used artificial neural networks to classify the images into the report categories, and to predict the “keywords” mentioning the images, e.g. predicting “adenopathy”, “masses”, “lung”, given a CT image with lung cancer. The rate of predicted disease-related words matching the actual words in the reports’ sentences was 56%. This is the first study performing a large-scale image/text analysis on a hospital PACS database. Comparing this rate to that of other published studies attempting to describe images on the Internet (Flickr), where the rate of matching to the original user annotated text ranges from 16% to 55%, our findings are very promising.

Scientific Focus Area: Biomedical Engineering and Biophysics

This page was last updated on Friday, March 26, 2021