Convolutional Neural Networks: An Overview And Application .

614feature map (Fig. 3a–c). This procedure is repeated applyingmultiple kernels to form an arbitrary number of feature maps,which represent different characteristics of the input tensors;different kernels can, thus, be considered as different featureextractors (Fig. 3d). Two key hyperparameters that define theconvolution operation are size and number of kernels. Theformer is typically 3 3, but sometimes 5 5 or 7 7. Thelatter is arbitrary, and determines the depth of output featuremaps.The convolution operation described above does not allowthe center of each kernel to overlap the outermost element ofthe input tensor, and reduces the height and width of the outputfeature map compared to the input tensor. Padding, typicallyzero padding, is a technique to address this issue, where rowsand columns of zeros are added on each side of the input tensor,so as to fit the center of a kernel on the outermost element andkeep the same in-plane dimension through the convolutionoperation (Fig. 4). Modern CNN architectures usually employzero padding to retain in-plane dimensions in order to applymore layers. Without zero padding, each successive featuremap would get smaller after the convolution operation.The distance between two successive kernel positionsis called a stride, which also defines the convolution operation. The common choice of a stride is 1; however, astride larger than 1 is sometimes used in order to achievedownsampling of the feature maps. An alternative technique to perform downsampling is a pooling operation, asdescribed below.The key feature of a convolution operation is weight sharing: kernels are shared across all the image positions. Weightsharing creates the following characteristics of convolutionoperations: (1) letting the local feature patterns extracted bykernels translation b invariant as kernels travel across all theimage positions and detect learned local patterns, (2) learningspatial hierarchies of feature patterns by downsampling inconjunction with a pooling operation, resulting in capturingan increasingly larger field of view, and (3) increasing modelefficiency by reducing the number of parameters to learn incomparison with fully connected neural networks.As described later, the process of training a CNN modelwith regard to the convolution layer is to identify the kernelsthat work best for a given task based on a given trainingdataset. Kernels are the only parameters automatically learnedduring the training process in the convolution layer; on theother hand, the size of the kernels, number of kernels, padding, and stride are hyperparameters that need to be set beforethe training process starts (Table 1).Nonlinear activation functionThe outputs of a linear operation such as convolution are thenpassed through a nonlinear activation function. Althoughsmooth nonlinear functions, such as sigmoid or hyperbolicInsights Imaging (2018) 9:611–629bcFig. 3 a–c An example of convolution operation with a kernel size of 3 3, no padding, and a stride of 1. A kernel is applied across the input tensor,and an element-wise product between each element of the kernel and theinput tensor is calculated at each location and summed to obtain theoutput value in the corresponding position of the output tensor, called afeature map. d Examples of how kernels in convolution layers extractfeatures from an input tensor are shown. Multiple kernels work as different feature extractors, such as a horizontal edge detector (top), a verticaledge detector (middle), and an outline detector (bottom). Note that the leftimage is an input, those in the middle are kernels, and those in the rightare output feature maps

Insights Imaging (2018) 9:611–629615000000002053008000000004900 1860 10002 11314000000000000003 11309700060007000000000000902 1400 10 14 11 11004206000000000002 150990000000000000004 60 ## ## ## ## ## 95 61 3200 2900000000000 10 16 ## ## ## ## ## ## ## ## ## ## ## 10000000000000 14 ## ## ## ## ## ## ## ## ## ## ## ## ##18002420002 98 ## ## ## ## ## ## ## ## ## ## ## ## ## 49308 13 ## ## ## ## 33 ## 5200 11 10013201012003 10 16 ## ## ## 49 000230200001210000000000000 110-1 -2 000000476000000 28 20000004 238110000000 18 137 138007 140 1381000007 14700000000000 12000000000006000000 38 ## ## ## ## ##0000000000000000076 20 ## ## ## ## ## ## ## ## 5600000000000 1200052 39 ## ##00 30 ## ## ## ## ##000000005 1000 38 ## ## ## ## ## ## ## ## ##00000005 10 114 37 15 ## ## 2705800 ## ## ## ## ## ## ## ## ## ## 18 25000000000 80 ## ## ## ## ##0400 13709 188000000000000000000000000000000019 1830180000002 11 28 20009 18 23800005 ## ## ##00 ## ## ## 200 ## ## 40000020 10 13 ## ## ## 36 10000005638 20 97 ## 8400 ## ## ## 410000 ## ## ## 56 1000000 ## ## ## ## 19 ## 77 ## 1700077500000002 16 39 ## ## ##0000000000 ## ## ## 40000000 ## ## ##4026 ## ## ## ## 25 11930 ## ## ## ## ##00000500009 ## ## ## ##00000000 ## ## ## 99000000 ## ## ## ## ## ## ## ## 24000 87 ## ## ## ## 6001 ## ## ## ## ##0 160000450000 83 ## ## ## 99000000 ## ## ## ## 10 20 1000000000 13 ## ## ## ## ## ## ## ## ## ## ## 360 190000004 10 1051 13 ## ## ## ## ##0000 ## ## ## ## 53 37 1900032105 ## ## ## ## ## ## ## ## ## 1707060000015 10 11842 22 37 1500794000005860000000001000000510000000000 ## ## ## ## 27000000000000004 97 ## ## ## ## ## ## ## ## ## 10040000000000000000 ## ## ## ## ## ## ##000040000000 22 ## ## ## ## ## ## 24 ## ## ## ## ##9080000000000000 80 ## ## ## ## ## ## ## ## ##000000000000 ## ## ## ## ## 24006 39 ## ## ## 56000000000000000 ## ## ## ##0000000000000 ## ## ## ##7 110002 62 ## ## ##390000000000 ## ## ##0000 77 ## 80 179000000000 ## ## ## ##9 200 133 13 ## ## ## 61010000000000 ## ## ##0000000000 ## ## ## ## ## 98 55 19 ## ## ## ## ## 5240000000 ## ## ## ## ##00000000 18 ## ## ## ## ## ## ## ## ## ## ## ##5 10000000000000000000 23 ## ## ## ## ## ## ## ## ## ## 14 120000000000000 18 ## ## ## ## ## ## ## ## ## ## ## ## 32 110000000000000610 52 ## ## ## ## ## 3700410000000000002 1100000000055000066001000000000000000000002002437 1600000000 101031000 70 ## ## ## 624 58 ## ## ## ## ## ## ## 110 141060000000002000600000 14920373000001000 19 ## ## 320000000000000 62 ## ## 420000000000 ## ## ## ## 4840000 18 ## ## ## ## ## ## ## 31 ## ## ## ## ## ## 14 24 1000001000 13029000005000100890 -1000000 1500 211 15 1109 233 2000000000006000 13 13900 20304 200000007 120 ## ## 758 29000000000504 12006 1371 11800 1900000000 ## ## ## ## 58000000000700000 77 ## ## 56 75 4800 2903 ## ## ##00000000000000 ## ## ## ## ## ## 45 10 580000 1200 ## ## ## ##0 87 ## ##000000000000000 78 ## ## ## ## ## ## ## 30800010 ## ## ##0 ## ## 620004200000 ## ##00 115000003 ##0 50 ## ## 28000 98 ## ## ## 2400302208000 12100000 ## ##0 14 ## ## 100000 ## ## ## 28005000 74 ## ##00 ## ## 170038000000 ## ## ## 21 ## 47000 87 ## ## ## 20005000 ## ## ##00 ## ## 19056000000 ## ## ## ## 6800000 ## ## ## 62 21000000 37 ## ##0 94 ## ## 2000034100000000 ## ## ## ## 19 32002000000 ## ## ## ## 1000 ## ## ## ##06000 1410 ## ## 25 11 1300 ## ## ## ##9000037400000 ## ## ##0000055000000 68 6300 ## ## ## ## 17610012200000 170 12 ## ## ## ##0 ## ##1000000000060 ## ## ## ## 51 1920 38 2200000003 19 120063001 ## ## ## ##2468800000000000000 ## ## ##0000 ## ## ## 584 1600000000000 74 ## ## ## ##00000 ## ## ##3 1700000000000 ## ## ## ## 5300000 ## ## ##6 19000 ## ## ##0000000000 ## ## ## ## ##00007 110000000000 37 ## ## ## ##0000 34 ## ## ## 13 110000000000000 ## ## 74000 94 ## ## ## 44 14 2000000000000000000 ## ## ## ## ## ## 186 100000000000000000 ## ## ## ## ## 18 192100000000000 1400000 ## ## ## 25 10 1101800000000009001000 22 2603 130000000 18 4700 ##60 6500700008 -100000000000000 5200000000000000000000000360 7510 190 28000 110630070 156 21 15000 ## ## ##0 ## ## ## 380 ## 12 ## ## ##000000000000 18 57 ## 31 6500 46000000 52 ## 85 40 84 68 15 12 410 8500 40 84 63 6800 15 120 410000000000 ##00000000000000022700 ## ## ## ## ## ## ## 1100000 ## ## ## ## ##00000 ##0000105 31 ## ##009 ## ## ## ##0000 57000 310 ## ## 510000000 ## ## ##8 22700000 ##0 10500 190 ## ## ##00 ##06003 310 ## ## ##008000000 41 ## ## ##017000000 ## ## ## ##00000-1 -1 -1-10 28 ## ##003-1 -1 -100 2310 -25090 -10002 11 ## ## ## ## 331003 110 ## ## ##2006 45 ## ## ## ## ## ## ## 33000000 73 ## ## ## ## ## ## ## 85000000 41 ## ##000 ## ##00 ## ## ## ##0000 110000 13 150 ## ## ## ## ## ## ## 29000 ## ## 28 ##0000050000000000 ## 85 ##000000000 ## ## ##0 600000000 ## ## ##00000000 ## ##00000 ## ## ## ##00000000 13 ## ## ## ## ## ##0 7100000 ## ## ## 190 ##00000000000000 26 23 37 930 44000 13 120000 ## ## ## ## ## ## ## ## ## ##000 13 15 110000 ## ##500000000000 ##000000000 2900000000000000000000000000000 ## ## ## ## ##000 ## ## ## ##000000000 ## 84 ## 920 ## ## ## ## ## ## ## ## ##00000 ## ## ##00 14000000 6000000000000000 ## ## ##00000000 68000000000 ## ## ##0000000 ## ## ##00000000000000 13 ## ## ## ##0000 ## ## ## ##00000000000000000000000000 ## ## ## ## ## ## ## ## ## 78 ## 7300 710000008 ## ## ## 73 38 ## ## ## 210000000000000 ## ## ## ## ##00000000000000000 26 260000003 36001000000000003 23 12 37 93000000 44000000 ##000 ## ## ## ##0080 75 19 460 ## ## ## ## ## ##0 ## ## ## ##0 ## 3400 21 21 ##00 ## ## ##00 ## ## ## ## ## 14000 ## ##0 ## ## ## 688 ## ## ## ## 210 3600Fig. 3 (continued)tangent (tanh) function, were used previously because they aremathematical representations of a biological neuron behavior,the most common nonlinear activation function used presentlyis the rectified linear unit (ReLU), which simply computes thefunction: f(x) max(0, x) (Fig. 5) [1, 3, 17–19].and distortions, and decrease the number of subsequent learnable parameters. It is of note that there is no learnable parameter in any of the pooling layers, whereas filter size, stride, andpadding are hyperparameters in pooling operations, similar toconvolution operations.Pooling layerMax poolingA pooling layer provides a typical downsampling operationwhich reduces the in-plane dimensionality of the feature mapsin order to introduce a translation invariance to small shiftsThe most popular form of pooling operation is max pooling,which extracts patches from the input feature maps, outputsthe maximum value in each patch, and discards all the other

616Insights Imaging (2018) 9:611–629Fig. 4 A convolution operationwith zero padding so as to retainin-plane dimensions. Note that aninput dimension of 5 5 is kept inthe output feature map. In thisexample, a kernel size and a strideare set as 3 3 and 1, respectivelyvalues (Fig. 6). A max pooling with a filter of size 2 2 with astride of 2 is commonly used in practice. This downsamplesthe in-plane dimension of feature maps by a factor of 2. Unlikeheight and width, the depth dimension of feature maps remains unchanged.Global average poolingAnother pooling operation worth noting is a global average pooling [20]. A global average pooling performs anextreme type of downsampling, where a feature map withsize of height width is downsampled into a 1 1 arrayby simply taking the average of all the elements in eachfeature map, whereas the depth of feature maps isretained. This operation is typically applied only once before the fully connected layers. The advantages of applying global average pooling are as follows: (1) reduces thenumber of learnable parameters and (2) enables the CNNto accept inputs of variable size.Fully connected layerThe output feature maps of the final convolution orpooling layer is typically flattened, i.e., transformed intoTable 1 A list of parameters andhyperparameters in aconvolutional neural network(CNN)Convolution layerPooling layerFully connected layerOthersa one-dimensional (1D) array of numbers (or vector), andconnected to one or more fully connected layers, alsoknown as dense layers, in which every input is connectedto every output by a learnable weight. Once the featuresextracted by the convolution layers and downsampled bythe pooling layers are created, they are mapped by a subset of fully connected layers to the final outputs of thenetwork, such as the probabilities for each class in classification tasks. The final fully connected layer typicallyhas the same number of output nodes as the number ofclasses. Each fully connected layer is followed by a nonlinear function, such as ReLU, as described above.Last layer activation functionThe activation function applied to the last fully connectedlayer is usually different from the others. An appropriateactivation function needs to be selected according to eachtask. An activation function applied to the multiclass classification task is a softmax function which normalizes output real values from the last fully connected layer to targetclass probabilities, where each value ranges between 0 and1 and all values sum to 1. Typical choices of the last rnel size, number of kernels, stride, padding, activation functionPooling method, filter size, stride, paddingNumber of weights, activation functionModel architecture, optimizer, learning rate, loss function, mini-batchsize, epochs, regularization, weight initialization, dataset splittingNote that a parameter is a variable that is automatically optimized during the training process and ahyperparameter is a variable that needs to be set beforehand

Insights Imaging (2018) 9:611–629617Fig. 5 Activation functions commonly applied to neural networks: a rectified linear unit (ReLU), b sigmoid, and c hyperbolic tangent (tanh)activation function for various types of tasks are summarized in Table 2.mathematically, a partial derivative of the loss with respect toeach learnable parameter, and a single update of a parameter isformulated as follows:Training a networkw ¼ w α*Training a network is a process of finding kernels in convolution layers and weights in fully connected layers which minimize differences between output predictions and givenground truth labels on a training dataset. Backpropagationalgorithm is the method commonly used for training neuralnetworks where loss function and gradient descent optimization algorithm play essential roles. A model performance under particular kernels and weights is calculated by a loss function through forward propagation on a training dataset, andlearnable parameters, namely kernels and weights, are updated according to the loss value through an optimizationalgorithm called backpropagation and gradient descent,among others (Fig. 1).where w stands for each learnable parameter, α stands for alearning rate, and L stands for a loss function. It is of note that,in practice, a learning rate is one of the most importanthyperparameters to be set before the training starts. In practice,for reasons such as memory limitations, the gradients of theloss function with regard to the parameters are computed byusing a subset of the training dataset called mini-batch, andapplied to the parameter updates. This method is calledmini-batch gradient descent, also frequently referred to as stochastic gradient descent (SGD), and a mini-batch size is also ahyperparameter. In addition, many improvements on the gradient descent algorithm have been proposed and widely used,such as SGD with momentum, RMSprop, and Adam [21–23],though the details of these algorithms are beyond the scope ofthis article.Loss functionA loss function, also referred to as a cost function, measuresthe compatibility between output predictions of the networkthrough forward propagation and given ground truth labels.Commonly used loss function for multiclass classification iscross entropy, whereas mean squared error is typically appliedto regression to continuous values. A type of loss function isone of the hyperparameters and needs to be determined according to the given tasks.Gradient descentGradient descent is commonly used as an optimization algorithm that iteratively updates the learnable parameters, i.e.,kernels and weights, of the network so as to minimize the loss.The gradient of the loss function provides us the direction inwhich the function has the steepest rate of increase, and eachlearnable parameter is updated in the negative direction of thegradient with an arbitrary step size determined based on ahyperparameter called learning rate (Fig. 7). The gradient is, L wData and ground truth labelsData and ground truth labels are the most important components in research applying deep learning or other machinelearning methods. As a famous proverb originating in computer science notes: BGarbage in, garbage out. Careful collection of data and ground truth labels with which to train andtest a model is mandatory for a successful deep learning project, but obtaining high-quality labeled data can be costly andtime-consuming. While there may be multiple medical imagedatasets open to the public [24, 25], special attention should bepaid in these cases to the quality of the ground truth labels.Available data are typically split into three sets: a training, avalidation, and a test set (Fig. 8), though there are some variants, such as cross validation. A training set is used to train anetwork, where loss values are calculated via forward propagation and learnable parameters are updated via backpropagation.A validation set is used to evaluate the model during the training

618Insights Imaging (2018) 9:611–629abFig. 6 a An example of max pooling operation with a filter size of 2 2,no padding, and a stride of 2, which extracts 2 2 patches from the inputtensors, outputs the maximum value in each patch, and discards all theother values, resulting in downsampling the in-plane dimension of anTable 2 A list of commonly applied last layer activation functions forvarious tasksTaskLast layer activation functionBinary classificationMulticlass single-class classificationMulticlass multiclass classificationRegression to continuous valuesSigmoidSoftmaxSigmoidIdentityinput tensor by a factor of 2. b Examples of the max pooling operationon the same images in Fig. 3b. Note that images in the upper row aredownsampled by a factor of 2, from 26 26 to 13 13process, fine-tune hyperparameters, and perform model selection. A test set is ideally used only once at the very end of theproject in order to evaluate the performance of the final modelthat was fine-tuned and selected on the training process withtraining and validation sets.Separate validation and test sets are needed because training amodel always involves fine-tuning its hyperparameters andperforming model selection. As this process is performed basedon the performance on the validation set, some informationabout this validation set leaks into the model itself, i.e.,

Insights Imaging (2018) 9:611–629619models during the training process. On the other hand, inmedicine, Bvalidation usually stands for the process of verifying the performance of a prediction model, which is analogous to the term Btest in machine learning. In order to avoidthis confusion, the word Bdevelopment set is sometimes usedas a substitute for Bvalidation set .OverfittingFig. 7 Gradient descent is an optimization algorithm that iterativelyupdates the learnable parameters so as to minimize the loss, whichmeasures the distance between an output prediction and a ground truthlabel. The gradient of the loss function provides the direction in which thefunction has the steepest rate of increase, and all parameters are updated inthe negative direction of the gradient with a step size determined based ona learning rateoverfitting to the validation set, even though the model is neverdirectly trained on it for the learnable parameters. For that reason, it is guaranteed that the model with fine-tunedhyperparameters on the validation set will perform well on thissame validation set. Therefore, a completely unseen dataset, i.e.,a separate test set, is necessary for the appropriate evaluation ofthe model performance, as what we care about is the modelperformance on never-before-seen data, i.e., generalizability.It is worthy of

operations described in this section are for 2D-CNN, similar operations can also be performed for three-dimensional (3D)-CNN. Convolution layer A convolution layer is a fundamental component of the CNN architecture that performs feature extraction, which typically Convolution Convolution is a specialized type of linear operation used for