Deep Neural Networks I - University Of California, Davis

5/31/2018Deep neural networks IMay 29th, 2018Yong Jae LeeUC DavisMany slides from Rob Fergus, Svetlana Lazebnik, Jia‐Bin Huang, Derek HoiemDiscriminative classifiersNeural networksNearest neighbor106 examplesShakhnarovich, Viola, Darrell 2003Berg, Berg, Malik 2005.Support Vector MachinesGuyon, VapnikHeisele, Serre, Poggio,2001, LeCun, Bottou, Bengio, Haffner 1998Rowley, Baluja, Kanade 1998 BoostingConditional Random FieldsViola, Jones 2001,Torralba et al. 2004,Opelt et al. 2006, McCallum, Freitag, Pereira2000; Kumar, Hebert 2003 2Slide adapted from Antonio TorralbaDiscriminative classifiersNeural networksNearest neighbor106 examplesShakhnarovich, Viola, Darrell 2003Berg, Berg, Malik 2005.Support Vector MachinesGuyon, VapnikHeisele, Serre, Poggio,2001, LeCun, Bottou, Bengio, Haffner 1998Rowley, Baluja, Kanade 1998 BoostingConditional Random FieldsViola, Jones 2001,Torralba et al. 2004,Opelt et al. 2006, McCallum, Freitag, Pereira2000; Kumar, Hebert 2003 3Slide adapted from Antonio Torralba1

5/31/2018Outline Deep Neural Networks Convolutional Neural Networks (CNNs)Traditional Image Categorization:Training uresClassifierTrainingTrainedClassifierTraditional Image Categorization:Testing sifierPredictionTestingImageFeaturesOutdoorTest Image2

5/31/2018Features have been key.HOG [Dalal and Triggs CVPR 05]SIFT [Loewe IJCV 04]Hand‐craftedSPM [Lazebnik et al. CVPR 06]DPM [Felzenszwalb et al. PAMI 10]Color Descriptor [Van De Sande et al. PAMI 10]What about learning the features? Learn a feature hierarchy all the way from pixels toclassifier Each layer extracts features from the output ofprevious layer Layers have (nearly) the same structure Train all layers jointly (“end‐to‐end”)Image/VideoPixelsLayer 1Layer 2Layer 3SimpleClassifierLearning Feature HierarchyGoal: Learn useful higher‐level features from imagesFeature representationInput data3rd layer“Objects”2nd layer“Object parts”Lee et al., ICML 2009;CACM 20111st layer“Edges”PixelsSlide: Rob Fergus3

5/31/2018Learning Feature Hierarchy Better performance Other domains (unclear how to hand engineer):– Kinect– Video– Multi spectral Feature computation time– Dozens of features now regularly used– Getting prohibitive for large datasets (10’s sec /image)Slide: R. Fergus“Shallow” vs. “deep” architecturesTraditional recognition: “Shallow” e extractionTrainableclassifierObjectClassDeep learning: “Deep” architectureImage/VideoPixelsLayer 1 Layer NSimpleclassifierObjectClassBiological neuron and PerceptronsA biological neuronAn artificial neuron (Perceptron)- a linear classifier4

5/31/2018Simple, Complex, and Hyper‐complex cellsvideoDavid H. Hubel and Torsten WieselSuggested a hierarchy of feature detectorsin the visual cortex, with higher level featuresresponding to patterns of activation in lowerlevel cells, and propagating activationupwards to still higher level cells.David Hubel's Eye, Brain, and VisionHubel/Wiesel Architecture and Multi‐layer Neural NetworkHubel and Weisel’s architectureMulti-layer Neural Network- A non-linear classifierNeuron: Linear Perceptron Inputs are feature values Each feature has a weight Sum is the activation If the activation is:– Positive, output 1– Negative, output ‐1Slide credit: Pieter Abeel and Dan Klein5

5/31/2018Two‐layer perceptron networkSlide credit: Pieter Abeel and Dan KleinTwo‐layer perceptron networkSlide credit: Pieter Abeel and Dan KleinTwo‐layer perceptron networkSlide credit: Pieter Abeel and Dan Klein6

5/31/2018Learning w Training examples Objective: a misclassification loss Procedure:– Gradient descent / hill climbingSlide credit: Pieter Abeel and Dan KleinHill climbing Simple, general idea:– Start wherever– Repeat: move to the bestneighboring state– If no neighbors better thancurrent, quit– Neighbors small perturbationsof w What’s bad?– Optimal?Slide credit: Pieter Abeel and Dan KleinTwo‐layer perceptron networkSlide credit: Pieter Abeel and Dan Klein7

5/31/2018Two‐layer perceptron networkSlide credit: Pieter Abeel and Dan KleinTwo‐layer neural networkSlide credit: Pieter Abeel and Dan KleinNeural network properties Theorem (Universal function approximators): A two‐layer network with a sufficient number of neuronscan approximate any continuous function to anydesired accuracy Practical considerations:– Can be seen as learning the features– Large number of neurons Danger for overfitting– Hill‐climbing procedure can get stuck in bad local optimaApproximation by Superpositions of Sigmoidal Function,1989Slide credit: Pieter Abeel and Dan Klein8

5/31/2018Multi‐layer Neural Network A non‐linear classifier Training: find network weights w to minimize theerror between true training labels and estimatedlabels Minimization can be done by gradient descentprovided f is differentiable This training method is calledback‐propagationOutline Deep Neural Networks Convolutional Neural Networks (CNNs)Convolutional Neural Networks(CNN, ConvNet, DCN) CNN a multi‐layer neural network with– Local connectivity: Neurons in a layer are only connected to a small regionof the layer before it– Share weight parameters across spatial positions: Learning shift‐invariant filter kernelsImage credit: A. Karpathy9

5/31/2018Neocognitron [Fukushima, Biological Cybernetics 1980]Deformation-ResistantRecognitionS-cells: (simple)- extract local featuresC-cells: (complex)- allow for positional errorsLeNet [LeCun et al. 1998] Stack multiple stages of featureextractorsHigher stages compute moreglobal, more invariant featuresClassification layer at the endGradient‐based learning applied to documentrecognition [LeCun, Bottou, Bengio, Haffner 1998]LeNet-1 from 1993Convolutional Neural NetworksFeature mapsSpatial poolingNon‐linearityConvolution(Learned)Input Image10

5/31/2018What is a Convolution? Weighted moving sum.Feature Activation MapInputWhy Convolution? Few parameters (filter weights)Dependencies are localTranslation invariance.Feature MapInputConvolutional Neural NetworksFeature mapsSpatial poolingNon‐linearityConvolution(Learned).Input ImageInputFeature Map11

5/31/2018Convolutional Neural NetworksFeature mapsRectified Linear Unit (ReLU)Spatial poolingNon‐linearityConvolution(Learned)Input Imageslide credit: S. LazebnikNon‐Linearity Per‐element (independent) Options:– Tanh– Sigmoid: 1/(1 exp(‐x))– Rectified linear unit (ReLU) Makes learning faster Simplifies backpropagation Avoids saturation issues Preferred optionConvolutional Neural NetworksFeature mapsNormalizationMax poolingSpatial ing: a non-linear down-samplingProvide translation invarianceInput Image12

5/31/2018Spatial Pooling Average or max Non‐overlapping / overlapping regions Role of pooling: Invariance to small transformationsLarger receptive fields (see more of input)MaxAverageEngineered vs. learned featuresLabelConvolutional filters are trained in asupervised manner by back-propagatingclassification n/poolFeature extractionConvolution/poolImageImageCompare: SIFT DescriptorLowe [IJCV 2004]ImagePixelsApplyoriented filtersSpatial pool(Sum)Normalize to unitlengthFeatureVector13

5/31/2018Compare: Spatial Pyramid MatchingSIFTfeaturesLazebnik,Schmid,Ponce[CVPR 2006]Filter withVisual WordsTake max VWresponseMulti‐scalespatial pool(Sum)GlobalimagedescriptorPrevious Convnet successes Handwritten text/digits– MNIST (0.17% error [Ciresan et al. 2011])– Arabic & Chinese [Ciresan et al. 2012] Simpler recognition benchmarks– CIFAR‐10 (9.3% error [Wan et al. 2013])– Traffic sign recognition 0.56% error vs 1.16% for humans[Ciresan et al. 2011]ImageNet Challenge 2012 14 million labeled images, 20k classes Images gathered from Internet Human labels via Amazon Turk ImageNet Challenge: 1.2 million trainingimages, 1000 classes[Deng et al. CVPR 2009]A. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet Classification with DeepConvolutional Neural Networks, NIPS 201214

5/31/2018AlexNetSimilar framework to LeCun’98 but: Bigger model (7 hidden layers, 650,000 units, 60,000,000 params) More data (106 vs. 103 images) GPU implementation (50x speedup over CPU) Trained on two GPUs for a weekA. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet Classification with DeepConvolutional Neural Networks, NIPS 2012AlexNet for image classificationAlexNet“car”Fixed input size: 224x224x3ImageNet Classification lks/2016/ILSVRC2016 10 09 clsloc.pdf15

5/31/2018Industry Deployment Used in Facebook, Google, MicrosoftStartupsImage Recognition, Speech Recognition, .Fast at test timeTaigman et al. DeepFace: Closing the Gap to Human-Level Performance in FaceVerification, CVPR’14Visualizing CNNs What input pattern originally caused a givenactivation in the feature maps?Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Layer 1Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]16

5/31/2018Layer 2Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Layer 3Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Layer 4 and 5Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]17

5/31/2018Beyond classification DetectionSegmentationRegressionPose estimationMatching patchesSynthesisand many more R‐CNN: Regions with CNN features Trained on ImageNet classification Finetune CNN on PASCALRCNN [Girshick et al. CVPR 2014]Labeling Pixels: Semantic LabelsFully Convolutional Networks for Semantic Segmentation [Long et al. CVPR 2015]18

5/31/2018Labeling Pixels: Edge DetectionDeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection[Bertasius et al. CVPR 2015]CNN for RegressionDeepPose [Toshev and Szegedy CVPR 2014]CNN as a Similarity Measure for MatchingStereo matching [Zbontar and LeCun CVPR 2015]Compare patch [Zagoruyko and Komodakis 2015]FlowNet [Fischer et al 2015]FaceNet [Schroff et al. 2015]Match ground and aerial images[Lin et al. CVPR 2015]19

5/31/2018CNN for Image GenerationLearning to Generate Chairs with Convolutional Neural Networks [Dosovitskiy et al. CVPR 2015]Chair MorphingLearning to Generate Chairs with Convolutional Neural Networks [Dosovitskiy et al. CVPR 2015]Transfer Learning Improvement of learning in a new task through thetransfer of knowledge from a related task that hasalready been learned. Weight initialization for CNNLearning and Transferring Mid‐Level Image Representations usingConvolutional Neural Networks [Oquab et al. CVPR 2014]20

5/31/2018Deep learning libraries TensorflowCaffeTorchMatConvNetFooling CNNsIntriguing properties of neural networks [Szegedy ICLR 2014]Questions?See you Thursday!6321

Deep Neural Networks Convolutional Neural Networks (CNNs) Convolutional Neural Networks (CNN, ConvNet, DCN) CNN a multi‐layer neural network with – Local connectivity: Neurons in a layer are only connected to a small region of the layer before it – Share weight parameters across spatial positions: