Flexible, High Performance Convolutional Neural Networks .
Proceedings of the Twenty-Second International Joint Conference on Artificial IntelligenceFlexible, High Performance ConvolutionalNeural Networks for Image ClassificationDan C. Cireşan, Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jürgen SchmidhuberIDSIA, USI and SUPSIGalleria 2, 6928 Manno-Lugano, chAbstract(CNNs) [LeCun et al., 1998; Behnke, 2003; Simard et al.,2003], whose weights (filters) are randomly initialized andchanged in a supervised way using back-propagation (BP).Despite the hardware progress of the past decades, computational speed is still a limiting factor for CNN architecturescharacterized by many building blocks typically set by trialand error. To systematically test the impact of various architectures on classification performance, we present a fast CNNimplementation on Graphics Processing Units (GPUs). Previous GPU implementations of CNNs [Chellapilla et al., 2006;Uetz and Behnke, 2009; Strigl et al., 2010] were hard-codedto satisfy GPU hardware constraints or use general purposelibraries, whereas our implementation is flexible and fully online (i.e., weight updates after each image). A notable exception is [Jarrett et al., 2009] who performed a thorough analysis of the influence of all building blocks of a multistage architecture on recognition performance. Our implementationallows for training large CNNs within days instead of months,such that we can investigate the influence of various structuralparameters by exploring large parameter spaces [Pinto et al.,2009] and performing error analysis on repeated experiments.We evaluate various networks on the handwritten digitbenchmark MNIST [LeCun et al., 1998] and two image classification benchmarks: NORB [LeCun et al., 2004] and CIFAR10 [Krizhevsky, 2009].We present a fast, fully parameterizable GPU implementation of Convolutional Neural Networkvariants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned ina supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10)and handwritten digit recognition (MNIST), witherror rates of 2.53%, 19.51%, 0.35%, respectively.Deep nets trained by simple back-propagation perform better than more shallow ones. Learning issurprisingly rapid. NORB is completely trainedwithin five epochs. Test error rates on MNISTdrop to 2.42%, 0.97% and 0.48% after 1, 3 and 17epochs, respectively.1IntroductionThe human visual system efficiently recognizes and localizes objects within cluttered scenes. For artificial systems,however, this is still difficult due to viewpoint-dependent object variability, and the high in-class variability of many object types. Deep hierarchical neural models roughly mimickthe nature of mammalian visual cortex, and by communityconsensus are among the most promising architectures forsuch tasks. The most successful hierarchical object recognition systems all extract localized features from input images, convolving image patches with filters. Filter responsesare then repeatedly sub-sampled and re-filtered, resulting in adeep feed-forward network architecture whose output featurevectors are eventually classified. One of the first hierarchical neural systems was the Neocognitron [Fukushima, 1980]which inspired many of the more recent variants.Unsupervised learning methods applied to patches of natural images tend to produce localized filters that resembleoff-center-on-surround filters, orientation-sensitive bar detectors, Gabor filters [Schmidhuber et al., 1996; Olshausen andField, 1997; Hoyer and Hyvärinen, 2000]. These findingsin conjunction with experimental studies of the visual cortex justify the use of such filters in the so-called standardmodel for object recognition [Riesenhuber and Poggio, 1999;Serre et al., 2007; Mutch and Lowe, 2008], whose filters arefixed, in contrast to those of Convolutional Neural Networks2Convolutional neural networksCNNs are hierarchical neural networks whose convolutionallayers alternate with subsampling layers, reminiscent of simple and complex cells in the primary visual cortex [Wieseland Hubel, 1959]. CNNs vary in how convolutional and subsampling layers are realized and how the nets are trained.2.1Image processing layerThe image processing layer is an optional pre-processinglayer of predefined filters that are kept fixed during training. Thus additional information besides the raw input image can be provided to the network, such as edges and gradients. In particular, we find that a contrast-extracting layer[Fukushima, 2003] helps to improve the recognition rate forNORB.1237
2.2Convolutional layerA convolutional layer is parametrized by the size and thenumber of the maps, kernel sizes, skipping factors, and theconnection table. Each layer has M maps of equal size (Mx ,My ). A kernel (blue rectangle in Fig 1) of size (Kx , Ky ) isshifted over the valid region of the input image (i.e. the kernelhas to be completely inside the image). The skipping factorsSx and Sy define how many pixels the filter/kernel skips in xand y-direction between subsequent convolutions. The sizeof the output map is then defined as:Mxn Mxn 1 Kxn 1;Sxn 1Myn Myn 1 Kyn 1 (1)Syn 1where index n indicates the layer. Each map in layer Ln isconnected to at most M n 1 maps in layer Ln 1 . Neurons ofa given map share their weights but have different receptivefields.2.3Figure 1: Architecture of a convolutional neural network withfully connected layers, kernel sizes of 5 x 5 and skipping factors of 1.Max-pooling layerThe biggest architectural difference between our implementation and the CNN of [LeCun et al., 1998] is the use ofa max-pooling layer instead of a sub-sampling layer. Nosuch layer is used by [Simard et al., 2003] who simply skipsnearby pixels prior to convolution, instead of pooling or averaging. [Scherer et al., 2010] found that max-pooling canlead to faster convergence, select superior invariant features,and improve generalization. A theoretical analysis of featurepooling in general and max-pooling in particular is given by[Boureau et al., 2010]. The output of the max-pooling layer isgiven by the maximum activation over non-overlapping rectangular regions of size (Kx , Ky ). Max-pooling enables position invariance over larger local regions and downsamples theinput image by a factor of Kx and Ky along each direction.2.4textures and shared memory. Code obtained by this pragmatic strategy is fast enough. We use the following typesof optimization: pre-computed expressions, unrolled loopswithin template kernels, strided matrices to obtain coalescedmemory accesses and registers wherever possible. Additionalmanual optimizations are possible in case future image classification problems will require even more computing power.3.1Classification layerKernel sizes of convolutional filters and max-pooling rectangles as well as skipping factors are chosen such that eitherthe output maps of the last convolutional layer are downsampled to 1 pixel per map, or a fully connected layer combinesthe outputs of the topmost convolutional layer into a 1D feature vector. The top layer is always fully connected, with oneoutput unit per class label.3Data structuresBoth outputs y and deltas δ of layer Ln are 2D strided. Theiroriginal size is Mx M My , but they are horizontally stridedwith a pitch of 32 floats (we use this stride for all 2D data),resulting in coalesced memory accesses. The vertical strideavoids additional bounding tests in CUDA kernels.All connections between maps of consecutive layers Ln 1and Ln are stored in matrix C n . Each row of C n containsall connections that feed into a particular map in layer Ln .Because we aim for a flexible architecture with partially connected layers, in the first column we store the number of previous connections. This index is useful for Forward Propagation (FP) and Adjusting Weights (AW) CUDA kernels. Thesecond column stores the number of connections, followedby corresponding indices of maps in Ln 1 connected to thecurrent map.For BP and FP, analogous information about connectionsis needed. We therefore store backward connections inCBP . AW requires a list of all map connections (see Subsection 3.4), stored as an array of map index pairs. Dealingwith biases in BP kernel requires to know where the weightsof particular connections start; this information is stored in a2D array WIDXBP of size M n M n 1 .GPU implementationThe latest generation of NVIDIA GPUs, the 400 and 500 series (we use GTX 480 & GTX 580), has many advantagesover older GPUs, most notably the presence of a R/W L2global cache for device memory. This permits faster programs and simplifies writing the code. In fact, the corresponding transfer of complexity into hardware alleviatesmany software and optimization problems. Our experimentsshow that the CNN program becomes 2-3 times faster just byswitching from GTX 285 to GTX 480.Manual optimization of CUDA code is very timeconsuming and error prone. We optimize for the new architecture, relying on the L2 cache for many of the devicememory accesses, instead of manually writing code that uses3.2Forward propagationA straightforward way of parallelizing FP is to assign a threadblock to each map that has to be computed. For maps withmore than 1024 neurons, the job is further split into smaller1238
blocks by assigning a block to each line of the map, becausethe number of threads per block is limited (1024 for GTX480). A one to one correspondence between threads andthe map’s neurons is assumed. Because of weight sharing,threads inside a block can access data in parallel, in particular the same weights and inputs from the previous layer. Eachthread starts by initializing its sum with the bias, then loopsover all map connections, convolving the appropriate patchof the input map with the corresponding kernel. The output isobtained by passing the sum through a scaled tanh activationfunction, and is then written to device memory.3.3max j Ky 1Sy 1 ,0 y min j, My 1 .Sy 1The above inequalities state that the delta of neuron (i, j)from Ln 1 is computed from deltas of neurons in a rectangular area in maps of Ln (Fig. 2). After summing up thedeltas, each thread multiplies the result by the derivative ofthe activation function.Backward propagationBP of deltas can be done in two ways: by pushing or bypulling. Pushing deltas means taking each delta from the current layer and computing the corresponding deltas for the previous layer. For an architecture with shared weights this hasthe disadvantage of being hard to code. Each delta from thecurrent layer contributes to many deltas in the previous layer,which translates into a lot of programming. There are twoways of avoiding this: either writing partial deltas to a separated block of memory and then putting everything togetherby calling another kernel (slow because of a tremendous increase in the number of memory accesses, and the need ofanother kernel), or using atomic writes (to avoid data hazards) to update deltas (very slow because many writings areserialized). We implement pulling deltas, which has almostnone of the above speed-limiting drawbacks, but is a bit morecomplicated.The (uni- or bi-dimensional) thread grid assigns a (bi- oruni-dimensional) thread block to each map in the previouslayer and a thread to each neuron in every map. Similar to FP,for maps with more than 1024 neurons, the 2D grid is furthersplit into smaller 1D blocks by assigning a 2D block to eachrow of the map. Each thread computes the delta of its corresponding neuron by pulling deltas from the current layer. Forevery neuron in the previous layer we have to determine thelist of neurons in the current layer which are connected to it.Let us consider neuron (i, j) from a map in layer Ln 1 , andthen assume that (x, y) are the coordinates of neurons in mapsof Ln that contribute to the delta of neuron (i, j). The (x, y)neuron is connected to kernel size number neurons (Kx Ky )from each connected map in the previous layer. The indices inLn 1 of the neurons connected through a kernel to the (x, y)neuron are:x(Sx 1)y(Sy 1) i j Figure 2: Back propagating deltas. A connection betweentwo maps from two consecutive layers is displayed. The mapin Ln 1 has 29 x 29 neurons; the map in Ln has 13 x 13neurons. They are linked through a 5 x 5 kernel K. Skippingfactors of Sx 1 and Sy 1 are assumed. Arrows andcolors depict the correspondence between neurons in Ln 1and their sources in Ln .3.44x(Sx 1) Kx 1,y(Sy 1) Ky 1.We can now compute the inequalities for (x, y):i Kx 1Sx 1j Ky 1Sy 1 x y Adjusting weightsFP and BP have a grid over the list of maps, but the AW threadgrid is over the list of kernels (filters) between maps of twoconsecutive layers. The 1D grid has a block for each connection between two maps. Thread blocks are 2D, with a corresponding thread for every kernel weight. The bias weightis included as an entire row of threads, thus requiring threadblocks to have (Kx 1) Ky threads. Most of the time theseadditional Ky threads will wait, thread (0,0) being activatedonly for blocks that have to process the bias.i,Sx 1j.Sy 1Because (x, y) has to be inside the map, the final inequalitiesare: ii Kx 1max, Mx 1 ,,0 x minSx 1Sx 11239ExperimentsWe use a system with a Core i7-920 (2.66GHz), 12 GB DDR3and four graphics cards: 2 x GTX 480 and 2 x GTX 580. Thecorrectness of the CPU version is checked by comparing theanalytical gradient with its finite difference approximation.On GPU this is not possible because all computations are performed with single precision floating point numbers. Hencethe GPU implementation’s correctness is checked by comparing its results to those of a randomly initialized net aftertraining it for several epochs on the more accurate CPU version. Obtaining identical results after trillions of operationsis a strong indication of correctness.The implemented CNN’s plain feed-forward architecture istrained using on-line gradient descent. All images from thetraining set are used for training and also for validation. Ifdeformations are enabled, only the images from the trainingset will be deformed. Weights are initialized according to auniform random distribution in the range [ 0.05, 0.05]. Each
elastic distortions seem to have a negative impact on generalization. These deformations improve recognition rates fordigits that are intrinsically 2D [Ciresan et al., 2010], but seeminadequate for 3D objects.Initial experiments on NORB show that unlike withMNIST where we use deformations, the CNN needs only 3to 6 epochs to reach zero validation error. This allows us toquickly run numerous repetitive experiments with huge networks with hundreds of maps per layer. We decided to usea CNN with five hidden layers: layer1, a convolutional layerwith 300 maps, kernel size 6 6 and skipping factors 1 1;layer2, a max-pooling layer over a 2 2 region; layer3, aconvolutional layer with 500 maps, kernel size 4 4, skipping factors 0 0; layer4, a max-pooling layer over a 4 4region; layer5, a fully connected layer with 500 neurons. Thelearning rate is initialized by 0.001 and multiplied by 0.95after every epoch.Table 2 summarizes the results of four different experiments by switching on/off translation as well as the fixed image processing layer. We report the average error rate as wellas the standard deviation of N independent runs with identical architectures but different weight initializations. For thefirst experiment without translation and no image processing(IP), an average test error rate of 7.86% is obtained. Withadditional translations of at most 5%, the average error ratedrops to 4.71%, contradicting the common belief that CNNsare translation invariant. These results are on par or betterthan others in the literature: 5.90% error rate for a combination of CNNs and SVMs [LeCun et al., 2004] and 5.20%error rate for restricted Boltzman machines [Nair and Hinton,2009].neuron’s activation function is a scaled hyperbolic tangent:y(a) 1.7159 tanh(0.6666a) [LeCun et al., 1998].We pick the trained CNN with the lowest validation error,and evaluate it on the test set (Test for best Validation - TfbV).The best test error (bT) is also listed for all experiments. Thereported computation times per epoch include training, validation and testing as well as all data transfers.4.1Experiments on MNISTFor the MNIST dataset the networks are trained on deformedimages, continually generated in on-line fashion. Affine(translation, rotation, scaling, horizontal shearing) and elastic deformations [Simard et al., 2003] are combined. We usea variable learning rate that shrinks by a multiplicative constant after each epoch, from 10 3 down to 3 · 10 5 after 500epochs.Table 1: Error rates on MNIST test set for randomly connected CNNs with 2 to 6 convolutional layers with M Mapsand an optional fully connected Layer with N Neurons, various kernel sizes and skipping factors were used.#M, #NTfbV[%]in Hidden 150N20M-40M-60M-80M-100M-120M-150N 0.35Fully connected convolutional layers lead to an explodingnumber of network connections and weights, making training of big and deep CNNs for hundreds of epochs impracticaleven on GPUs. Partial connectivity alleviates this problemand is also biologically more plausible. We reduce the number of connections between convolutional layers in a randomway [LeCun et al., 1998; Jarrett et al., 2009]. Table 1 listsresults of various networks with 2 to 7 hidden layers with random connections. Additional layers result in better networks,the best one achieving a test error of 0.35% for best validationand a best test error of 0.27%. The best previous CNN resulton MNIST is 0.40% [Simard et al., 2003]. A 0.35% error ratewas recently also obtained by a big, deep MLP [Ciresan et al.,2010] with many more free parameters. Deeper nets requiremore computation time to complete an epoch, but we observethat they also need fewer epochs to achieve good test errors.The deepest CNN from Table 1 reaches 2.42%, 0.97% and0.48% after one, three and seventeen epochs, respectively.On the other hand, the network with 4 instead of 7 hiddenlayers reaches 4.71%, 1.58%, 0.68% after one, three and seventeen epochs, achieving a test error below 0.50% after only34 epochs. This shows once more that deep networks, contrary to common belief, can be trained successfully by backpropagation. Despite the numerous free parameters, deep networks seem to learn faster (better recognition rates after fewerepochs) than shallow ones.4.2Table 2: Average error rates and standard deviations of N runsfor a five hidden layer CNN on the NORB test set (see textfor details).trans. [%] IPTfbV [%]runs time/epoch [s]0no 7.86 0.555011415no 4.71 0.575015630yes 3.94 0.485016585yes 2.53 0.40 1002080The best previously published result on NORB (2.87%)was obtained by a hierarchical neural network which to everyconvolutional layer provides a subsampled version plus edgeinformation of the original image [Uetz and Behnke, 2009].This motivates us to implement a pre-processing layer withfixed filters. We try simple edge masks (Sobel, Scharr), butfind that a contrast-extraction layer [Fukushima, 2003] realized by mexican hat shaped filters of size 21 21 works best.We use two filters, one with a concentric on-center receptivefield and one with a concentric off-center receptive field.The first filter extracts positive contrast in brightness,whereas the latter extracts negative contrast. Each imagefrom the original NORB is filtered, consequently the inputof the CNN has six maps, the original image plus the positiveand negative contrast for each of the two stereo channels. Using such a pre-processing layer results in lower average errorrates, 3.94% without translation and 2.53% with translation.Experiments on NORBNORB contains stereo images of 3D objects. Hence there aretwo maps on
2 Convolutional neural networks CNNs are hierarchical neural networks whose convolutional layers alternate with subsampling layers, reminiscent of sim-ple and complex cells in the primary visual cortex [Wiesel and Hubel, 1959]. CNNs vary in how convolutional and sub-sampling layers are realized and how the nets are trained. 2.1 Image processing .
Learning a Deep Convolutional Network for Image Super-Resolution . a deep convolutional neural network (CNN) [15] that takes the low- . Convolutional Neural Networks. Convolutional neural networks (CNN) date back decades [15] and have recently shown an explosive popularity par-
Performance comparison of adaptive shrinkage convolution neural network and conven-tional convolutional network. Model AUC ACC F1-Score 3-layer convolutional neural network 97.26% 92.57% 94.76% 6-layer convolutional neural network 98.74% 95.15% 95.61% 3-layer adaptive shrinkage convolution neural network 99.23% 95.28% 96.29% 4.5.2.
Video Super-Resolution With Convolutional Neural Networks Armin Kappeler, Seunghwan Yoo, Qiqin Dai, and Aggelos K. Katsaggelos, Fellow, IEEE Abstract—Convolutional neural networks (CNN) are a special type of deep neural networks (DNN). They have so far been suc-cessfully applied to image super-resolution (SR) as well as other image .
ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012 M. Zeiler and R. Fergus, Visualizing and Understanding Convolutional Networks, ECCV 2014 K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR 2015
Deep Neural Networks Convolutional Neural Networks (CNNs) Convolutional Neural Networks (CNN, ConvNet, DCN) CNN a multi‐layer neural network with – Local connectivity: Neurons in a layer are only connected to a small region of the layer before it – Share weight parameters across spatial positions:
of networks are updated according to learning rate, cost function via stochastic gradient descent during the back propagation. In the following, we briefly introduce the structures of di erent DNNs applied in NLP tasks. 2.1.1 Convolutional Neural Network Convolutional neural networks (CNNs) learn local features and assume that these features
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky University of Toronto kriz@cs.utoronto.ca Ilya Sutskever University of Toronto ilya@cs.utoronto.ca Geoffrey E. Hinton University of Toronto hinton@cs.utoronto.ca Abstract We trained a large, deep convolutional neural network to classify the 1.2 million
Massachusetts Curriculum Framework for English Language Arts and Literacy 3 Grade 5 Language Standards [L]. 71 Resources for Implementing the Pre-K–5 Standards. 74 Range, Quality, and Complexity of Student Reading Pre-K–5 . 79 Qualitative Analysis of Literary Texts for Pre-K–5: A Continuum of Complexity. 80 Qualitative Analysis of Informational Texts for Pre-K–5: A .
