Deep Learning In Medical Imaging: General Overview
Review Article Experiment, Engineering, and SSN 1229-6929 · eISSN 2005-8330Korean J Radiol 2017;18(4):570-584Deep Learning in Medical Imaging: General OverviewJune-Goo Lee, PhD1, Sanghoon Jun, PhD2, 3, Young-Won Cho, MS2, 3, Hyunna Lee, PhD2, 3,Guk Bae Kim, PhD2, 3, Joon Beom Seo, MD, PhD2*, Namkug Kim, PhD2, 3*1Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Korea; 2Department ofRadiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Korea; 3Department ofConvergence Medicine, Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, KoreaThe artificial neural network (ANN)–a machine learning technique inspired by the human neuronal synapse system–wasintroduced in the 1950s. However, the ANN was previously limited in its ability to solve actual problems, due to the vanishinggradient and overfitting problems with training of deep architecture, lack of computing power, and primarily the absence ofsufficient data to train the computer system. Interest in this concept has lately resurfaced, due to the availability of big data,enhanced computing power with the current graphics processing units, and novel algorithms to train the deep neural network.Recent studies on this technology suggest its potentially to perform better than humans in some visual and auditory recognitiontasks, which may portend its applications in medicine and healthcare, especially in medical imaging, in the foreseeable future.This review article offers perspectives on the history, development, and applications of deep learning technology, particularlyregarding its applications in medical imaging.Keywords: Artificial intelligence; Machine learning; Convolutional neural network; Recurrent Neural Network; Computer-aided;Precision medicine; RadiologyINTRODUCTIONMachine learning (ML) is defined as a set of methodsthat automatically detect patterns in data, and then utilizethe uncovered patterns to predict future data or enabledecision making under uncertain conditions (1). ML is asubset of “artificial intelligence” (AI). In general, thereare three approaches to AI: symbolism (rule based, suchas IBM Watson), connectionism (network and connectionbased, such as deep learning or artificial neural net), andBayesian (based on the Bayesian theorem). The mostrepresentative characteristic of ML is that it is drivenby data, and the decision process is accomplished withminimum interventions by a human. The program can learnby analyzing training data, and then make a predictionwhen new data is put in.Received December 20, 2016; accepted after revision March 29, 2017.*These authors contributed equally to this work.Corresponding authors: Namkug Kim, PhD, Department of Convergence Medicine and Radiology, Research Institute of Radiology andInstitute of Biomedical Engineering, University of Ulsan College of Medicine, Asan Medical Center, 88 Olympic-ro 43-gil, Songpa-gu, Seoul05505, Korea. Tel: (822) 3010-6573 Fax: (822) 476-4719 E-mail: namkugkim@gmail.com; andJoon Beom Seo, MD, PhD, Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, AsanMedical Center, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea. Tel: (822) 3010-4393 Fax: (822) 476-4719 E-mail: joonbeomseo@gmail.comThis is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium,provided the original work is properly cited.570Copyright 2017 The Korean Society of Radiology
Deep Learning in Medical ImagingDeep learning is a part of ML and a special type ofartificial neural network (ANN) that resembles themultilayered human cognition system. Deep learning iscurrently gaining a lot of attention for its utilization withbig healthcare data. Even though ANN was introducedin 1950, there were severe limitations in its applicationto solve real dilemmas, due to vanishing gradient andoverfitting problems, which hindered the training in deeparchitecture, lack of computing power, and primarily theabsence of sufficient data to train the computer system.However, many limitations have now been resolved, giventhe current availability of big data, enhanced computingpower with graphics processing units (GPU), and newalgorithms to train a deep neural network (DNN). Thesedeep learning approaches have exhibited impressiveperformances in mimicking humans in various fields,including medical imaging. One of the typical tasks inradiology practice is detecting structural abnormalities andclassifying them into disease categories. Since the 1980s,numerous ML algorithms with different implementations,mathematical bases, and logical theories have beenexecuted to perform such classification tasks. Accordingly,several computer-aided detection (CAD) systems weredeveloped and introduced in the clinical workflow in theearly 2000s. However, adverse impacts of these systemshave been reported in clinical studies (2, 3). In particular,CAD systems were found to generate more false positivesthan human readers, which led to a greater assessment timeand additional biopsies (2). Thus, the net benefit gained byusing CAD was unclear (3). It is expected that current deeplearning technology may help overcome the limitations ofprevious CAD systems, achieve greater detection accuracy,and help make human readers more productive by allowingthem to shift humdrum, repetitive radiology tasks to AI.Deep learning is well suited to medical big data, andcan be used to extract useful knowledge from it. Thisnew AI technology has a potential to perform automaticlesion detection, suggest differential diagnoses, andcompose preliminary radiology reports. In fact, theglobally integrated enterprise IBM is already developingthe radiology applications of Dr. Watson. This systemincludes all the above-mentioned functions, includingautomatic detection and quantitative feature analysisof the lesion in medical imaging. The rapid rise in AItechnology requires radiologists to have knowledge aboutthe technology, in order to understand the ability of AI andhow it might change and influence radiologic practice inkjronline.orgKorean J Radiol 18(4), Jul/Aug 2017the near future. We believe that eventually, the adoptionof these ML-based analytic tools in radiology practice willhappen. However, we also believe that it does not mean areplacement of radiologists, although some specific humantasks will be replaced. These “replacements” will not reallybe an ultimate replacement, but an overall augmentationof the entire radiology practice, as it will complementirreplaceable and remarkable human skills. In this review,we introduce the history and describe the general, medical,and radiological applications of deep learning.From Traditional Machine Learning Methods toDeep LearningFor training the algorithm, the ML learning methodsare classified as supervised learning and unsupervisedlearning. Supervised learning generates a function thatreproduces output by inferring from training data. For thismethod, training data is prepared with numerical or nominalvectors that represent the characteristics of input dataand the corresponding output data. When the output datahas a continuous value, the training process is generallyreferred to as regression. However, if the output data has acategorized value, the process is referred to as classification.In contrast to supervised learning, unsupervised learningdoes not involve the consideration of output data, butinstead infers a function to describe hidden structures fromunlabeled input data. Since the examples are unlabeled,there is no objective evaluation of the accuracy. Thoughunsupervised learning encompasses many other solutionsinvolving summarizing and explaining key features of thedata, unsupervised learning is similar to a cluster analysis instatistics, and focuses on the manner which composes thevector space representing the hidden structure, includingdimensionality reduction and clustering (Fig. 1).A naïve Bayesian model that focuses on the probabilitydistribution of input data is a typical classificationalgorithm. The algorithm is relatively simple, but showsbest performance in specific areas such as rRNA sequenceassignment (4). The support vector machine (SVM) is themost popular classification algorithm, and typically exhibitsthe highest performance ranks for most classificationproblems, given its advantages of regularization and convexoptimization (5, 6). Recently, ensemble learning, combinedwith the diverse classification algorithm for preciseprediction, is commonly being used for more advancedclassifications (7).571
Lee et al.With regard to regression, the linear and logisticregression systems are widely used due to their simplearchitecture. The parameters of linear regression areestimated to ensure the best fit of the straight line in thedata space. Logistic regression employs the logistic functionto differentiate binomial distribution, and is usually usedas a classifier. Support vector regression (SVR) and ANN arebeing increasingly used in recent years, and have shownbetter performances in the regression of certain problems.SVR is a version of SVM for regression (8), and has roximationreliable performance in forecasting weather and financialdata (9, 10). ANN is a popular regression and classificationalgorithm for ML, modeling the computational units ofmultiple layers by imitating signal transmission, and bylearning the architecture of neurons and synapses in thehuman brain.Figure 2 shows the concept of neural networks derivedthrough biological inspiration. A single neuron consistsof dendrites, axon, cell body, and synapse. The simple cellneuron integrates the various input signals and transmitsNotworkingStartSGDclassifierNotK SpectralclusteringGMMNoK meansYesClusteringYes 10 KsamplesYesElastic netLassoSGDregressorPredictingcategoryNo 100 KsamplesYesSVR (kernel ‘rbf’)ensemble regressorsYesNoYesFew featuresshould beimportantNotworkingRidge regressionSVR(kernel Number ofcategoriesknownNoMini batchK meansDo you havelabeleddata 50samplesYes 100 Ksamples 10 KsamplesJustlookingNoNoRandomizedPCANotworkingYes 10 KsamplesMean onKernelapproximationFig. 1. Categories of machine learning, including classification, regression, clustering, and dimensionality reduction. Adapted e learning map/ (101). GMM Gaussian mixture model, LLE locally-linear embedding, PCA principal component analysis, SGD stochastic gradient descent, SVC support vector classification, SVR support vector regression, VBGMM variational Bayesian Gaussian mixture modelSynapsesDendritesAxonCell bodySynapseχ1W1jχ2W2jχ3W3jχnWnjInput fromother neuronsBody of neuron(signal integration)Σy1yDendritesActivationfunctionOutput axonABFig. 2. Conceptual analogy between real neurons (A) and artificial neurons (B).572Korean J Radiol 18(4), Jul/Aug 2017kjronline.org
Deep Learning in Medical Imagingthem to other neurons (Fig. 2A). The ANN is composed ofinterconnected artificial neurons. Each artificial neuronimplements a simple classifier model that outputs adecision signal based on the weighted sum of evidences(Fig. 2B). Hundreds of these basic computing units areassembled together to establish the ANN. The weights ofthe network are trained by a learning algorithm, such asback propagation, where pairs of input signals and desiredoutput decisions are presented, mimicking the conditionwhere the brain relies on external sensory stimuli to learnto achieve specific tasks (11).Numerical or nominal values used as input data aregenerally referred to as features in ML. Defining meaningfuland powerful features was an important process in previousML studies. Many domain experts and data scientistssought to discover and generate handcrafted features afterapplying diverse evaluation approaches, including statisticalanalysis and performance tests of ML. To enhance thisprocess and achieve training models with higher accuracy,various data cleaning and feature selection methods havebeen developed to obtain significant improvements inperformance. After defining and selecting good handcraftedfeatures, ML algorithms are applied for modeling regression,classification, or unsupervised analysis.Previous studies show that ANN has remarkableperformance in various fields, but had limitations such asa decrease in the local minimum during optimization, andover-training for given values (overfitting). Researcherstherefore attempted to use deep architecture to determinesolutions, but its complex operation and heavy trainingcosts limited the ability to generate successful models.Input layerHidden layer 1Hidden layer 2DNN consists of a series of stacked layers (Fig. 3A). Thefirst layer (input) represents the observed values basedon which a prediction is made. The last layer (output)produces a value or class prediction. The layers betweenthe input and output layers are called hidden layers, sincetheir state does not correspond to observable data (input oroutput). The tiered structure of the neural networks allowsthem to produce much more complex decisions, based on acombination of simpler decisions. For example, starting withsimple localized interpretation of each part of an input,deeper hidden layers can model more complicated networksin the data, thus enabling the classification of a tumor frompixel to curve to shape and to feature. Each edge requiresoptimized weights for specific training samples. Theseweights used by DNNs can sum up to billions of parameters,and are randomly initialized and progressively configuredby an optimization algorithm such as gradient descent, tofind a local minimum of a function by steps proportional tothe negative of the gradient of the function at the currentpoint (12). After applying training samples to the network,a loss function between the prediction and the target classor regression value, is quantitatively evaluated. All theparameters are then slightly updated in the direction thatwill favor the minimization of the loss function.Based on these neural networks, there are differentcategories of deep learning with different approaches. DNNextends the depth of layers as compared to traditionalANN, and has shown better performance in prediction andrecognition studies, when the layers become complex (13).Recently, ML researchers have developed technicalsolutions for implementing deeper architecture (Fig.Hidden layerHidden layer 3Input layerOutput layerOutput layerABFig. 3. Comparison between shallow learning and deep learning in neural network.A. Typical deep learning neural network with 3 deep layers between input and output layers. B. Typical artificial neural network with 1 layerbetween input and output layers.kjronline.orgKorean J Radiol 18(4), Jul/Aug 2017573
Lee et al.3A), as compared to the traditional ANN (Fig. 3B). Usingthe unsupervised restricted Boltzmann machine (Fig. 4)proposed by Hinton et al. (14), the layers of deep neuralarchitecture were trained separately in an unsupervisedmanner. As a result, the limitations of DNN, such as localminimum optimization and overfitting, were overcome. Asthe model could learn from data with deep architecture inan unsupervised manner, it could generate features fromraw data. The learning process of this DNN architecture canbe observed from the external web based application (15).The development of hardware technology, such asgeneral-purpose computing on a GPU, has enabledcomplex operations in shorter computation time fortraining DNN. Thus, deep learning models now generatemeaningful and powerful features after analyzing a largeamount of uncategorized data and training the modelfor accurate prediction by using these features. Thisprocess is surprisingly similar to the process of obtainingknowledge in humans with regard to self-organization.These breakthroughs have led to innovative improvementsin performances in various research fields, such as speechrecognition, image classification, and face recognition.There are several currently available open source deeplearning libraries, like Caffe (16), Microsoft Cognitive Toolkit(CNTK) (17), Tensorflow (18), Theano (19), and Torch (20).Convolutional Neural NetThe convolutional neural network (CNN), whichconsists of multiple layers of neuron-like computationalconnections with step-by-step minimal processing, hasachieved significant improvements in the computervision research area. The overall learning process of CNNsimulates the organization of the animal visual cortex (21),and a successfully trained CNN can compose hierarchicalinformation during pre-processing, such as an edge-shapecomponent-object structure in image classification.The architecture of CNN is composed of convolutional,pooling layers and fully connected layers (Fig. 5). Theprimary purpose of a convolutional layer is to detectdistinctive local motif-like edges, lines, and other visualelements. The parameters of specialized filter operators,termed as convolutions, are learned. This mathematicaloperation describes the multiplication of local neighbors ofa given pixel by a small array of learned parameters calleda kernel (Fig. 6A). By learning meaningful kernels, thisoperation mimics the extraction of visual features, such asedges and colors, similar to that noted for the visual cortex.AlgorithmsUnsupervised pre-trainingDropoutFig. 4. Two breakthrough algorithms in deep learning,including unsupervised pre-training and dropout.Maxpooling4420194 333334931964Input image49Conv. layer #146464Dropout64Local responsenormalizationConv. layer #2620 41004MaxpoolingLocal responsenormalizationConv. layer #3Conv. layer #4FC #1FC #2Fig. 5. Architecture of convolutional neural networks, including input, Conv., and FC layers. Conv. convolutional, FC fully connected574Korean J Radiol 18(4), Jul/Aug 2017kjronline.org
Deep Learning in Medical ImagingInput imageMax poolingAverage poolingConvolutionkernelOutput pixelAFig. 6. Illustration of convolution and pooling methods.BA. Convolution method. B. Max and average pooling methods.This process can be performed by using filter banks. Eachfilter is a square-shaped object that travels over the givenimage. The image values on this moving grid are summedusing the weights of the filter. The convolutional layerapplies multiple filters and generates multiple feature maps.Convolutions are a key component of CNN, and are vital forsuccess in image processing tasks such as segmentation andclassification.To capture an increasingly larger field of view, featuremaps are progressively and spatially reduced by poolingthe pixels together (Fig. 6B). By propagating only themaximum or average activation through a layer of max oraverage pooling, convolutional layers subsequently becomeless sensitive to small shifts or distortions of the targetobject in extracted feature maps. The pooling layer is usedto effectively reduce the dimensions of feature maps, andremain robust to the shape and position of the detectedsemantic features within the image. In most cases, themax pooling in a feature map is empirically used. Theseconvolutional and pooling layers are repeated several times.The fully connected layers are incorporated to integrate allthe feature responses from the entire image and providethe final results. This CNN architecture can be furtherunderstood from the external resource (22).By
Deep Learning in Medical Imaging kjronline.org Korean J Radiol 18(4), Jul/Aug 2017 Deep learning is a part of ML and a special type of artificial neural network (ANN) that resembles the multilayered human cognition system. Deep learning is currently gaining a lot of attention for its utilization with big healthcare data. Even though ANN was .File Size: 2MB
1. Medical imaging coordinate naming 2. X-ray medical imaging Projected X-ray imaging Computed tomography (CT) with X-rays 3. Nuclear medical imaging 4. Magnetic resonance imaging (MRI) 5. (Ultrasound imaging covered in previous lecture) Slide 3: Medical imaging coordinates The anatomical terms of location Superior / inferior, left .
Medical X-Ray Medical Imaging N/A N/A 5-100 Tc-99m Medical Imaging (SPECT) 6.02 hours J 140.5 Tl-201 Medical Imaging (SPECT) 73 hours H 135, 167 In-111 Medical Imaging (SPECT) 2.83 days H 171, 245 F-18 Medical Imaging (PET) 1.83 hours E 511 Ga-68 Medical Imaging (PET) 68 minutes E 511 Cs-137 Fission Product 30.17 years E- 662
Deep Learning: Top 7 Ways to Get Started with MATLAB Deep Learning with MATLAB: Quick-Start Videos Start Deep Learning Faster Using Transfer Learning Transfer Learning Using AlexNet Introduction to Convolutional Neural Networks Create a Simple Deep Learning Network for Classification Deep Learning for Computer Vision with MATLAB
BIBLIOGRAPHY Physics for Medical Imaging P. Allisy-Roberts, J. Williams – Farr’s Physics for Medical Imaging Radiological Physics P. Dendy, B. Heaton – Physics for Radiologists Medical Imaging J. Bushberg et al – The Essential Physics of Medical Imaging S. Webb – The Physics of Medical Imaging Nuclear Medicine
Rhode Island Medical Imaging: Greenwich Medical Center 1351 South County Trail, Suite 105, East Greenwich, RI 02818 : Rhode Island Medical Imaging: Blackstone Valley Medical Building 333 School Street, Suite 105, Pawtucket, RI 02860 . Rhode Island Medical Imaging: Moshassuck Medical Center 1 Randall Square, Suite 103, Providence, RI 02904
2.3 Deep Reinforcement Learning: Deep Q-Network 7 that the output computed is consistent with the training labels in the training set for a given image. [1] 2.3 Deep Reinforcement Learning: Deep Q-Network Deep Reinforcement Learning are implementations of Reinforcement Learning methods that use Deep Neural Networks to calculate the optimal policy.
Head to Toe Imaging Conference 34th Annual Morton A. Bosniak The Department of Radiology Presents: New York Hilton Midtown December 14–18, 2015 Monday, Dec. 14 Abdominal Imaging & Emergency Imaging Tuesday, Dec. 15 Thoracic Imaging & Cardiac Imaging Wednesday, Dec. 16 Neuroradiology & Pediatric Imaging Thursday, Dec. 17 Musculoskeletal Imaging & Interventional
During the American Revolution both the American Continental Army and the British Army had spies to keep track of their enemy. You have been hired by the British to recruit a spy in the colonies. You must choose your spy from one of the colonists you have identified. When making your decisions use the following criteria: 1. The Spy cannot be someone who the Patriots mistrust. The spy should be .
