Deep Neural Networks - University Of California, Davis

6/1/17Deep neural networksJune 1st, 2017Yong Jae LeeUC DavisMany slides from Rob Fergus, Svetlana Lazebnik, Jia-Bin Huang, Derek HoiemAnnouncements Post quesMons on Piazza for review-session(6/8 lecture)2Outline Deep Neural Networks ConvoluMonal Neural Networks (CNNs)1

6/1/17TradiMonal Image CategorizaMon:Training l Image CategorizaMon:TesMng assifierPredictionTestingImageFeaturesOutdoorTest ImageFeatures have been key.SIFT [Loewe IJCV 04]HOG [Dalal and Triggs CVPR 05]Hand-cra(edSPM [Lazebnik et al. CVPR 06]DPM [Felzenszwalb et al. PAMI 10]Color Descriptor [Van De Sande et al. PAMI 10]2

6/1/17What about learning the features? Learn a feature hierarchy all the way from pixels toclassifier Each layer extracts features from the output ofprevious layer Layers have (nearly) the same structure Train all layers jointlyImage/VideoPixelsLayer 1Layer 2SimpleClassifierLayer 3Learning Feature HierarchyGoal: Learn useful higher-level features from imagesFeature representaMonInput data3rd layer“Objects”2nd layer“Object parts”Lee et al., ICML 2009;CACM 20111st layer“Edges”PixelsSlide: Rob FergusLearning Feature Hierarchy Better performance Other domains (unclear how to hand engineer):– Kinect– Video– Multi spectral Feature computation time– Dozens of features now regularly used– Getting prohibitive for large datasets (10’s sec /image)Slide: R. Fergus3

6/1/17“Shallow” vs. “deep” architecturesTraditional recognition: “Shallow” architectureImage/VideoPixelsHand-designedfeature extracMonTrainableclassifierObjectClassDeep learning: “Deep” architectureImage/VideoPixelsLayer 1 Layer NSimpleclassifierObjectClassBiological neuron and PerceptronsA biological neuronAn artificial neuron (Perceptron)- a linear classifierSimple, Complex, and Hyper-complex cellsvideoDavid H. Hubel and Torsten WieselSuggested a hierarchy of feature detectorsin the visual cortex, with higher level featuresresponding to patterns of activation in lowerlevel cells, and propagating activationupwards to still higher level cells.David Hubel's Eye, Brain, and Vision4

6/1/17Hubel/Wiesel Architecture and MulM-layer Neural NetworkHubel and Weisel’s architectureMulti-layer Neural Network- A non-linear classifierNeuron: Linear Perceptron Inputs are feature values Each feature has a weight Sum is the acMvaMon If the acMvaMon is:– PosiMve, output 1– NegaMve, output -1Slide credit: Pieter Abeel and Dan KleinTwo-layer perceptron networkSlide credit: Pieter Abeel and Dan Klein5

6/1/17Two-layer perceptron networkSlide credit: Pieter Abeel and Dan KleinTwo-layer perceptron networkSlide credit: Pieter Abeel and Dan KleinLearning w Training examples ObjecMve: a misclassificaMon loss Procedure:– Gradient descent / hill climbingSlide credit: Pieter Abeel and Dan Klein6

6/1/17Hill climbing Simple, general idea:– Start wherever– Repeat: move to the bestneighboring state– If no neighbors befer thancurrent, quit– Neighbors small perturbaMonsof w What’s bad?– OpMmal?Slide credit: Pieter Abeel and Dan KleinTwo-layer perceptron networkSlide credit: Pieter Abeel and Dan KleinTwo-layer perceptron networkSlide credit: Pieter Abeel and Dan Klein7

6/1/17Two-layer neural networkSlide credit: Pieter Abeel and Dan KleinNeural network properMes Theorem (Universal func9on approximators): Atwo-layer network with a sufficient number ofneurons can approximate any conMnuous funcMon toany desired accuracy Prac9cal considera9ons:– Can be seen as learning the features– Large number of neurons Danger for overfijng– Hill-climbing procedure can get stuck in bad local opMmaApproximation by Superpositions of Sigmoidal Function,1989Slide credit: Pieter Abeel and Dan KleinMulM-layer Neural Network A non-linear classifier Training: find network weights w to minimize theerror between true training labels and esMmatedlabels MinimizaMon can be done by gradient descentprovided f is differenMable This training method is calledback-propaga9on8

6/1/17Outline Deep Neural Networks Convolu9onal Neural Networks (CNNs)ConvoluMonal Neural Networks(CNN, ConvNet, DCN) CNN a mulM-layer neural network with– Local connecMvity: Neurons in a layer are only connected to a small regionof the layer before it– Share weight parameters across spaMal posiMons: Learning shil-invariant filter kernelsImage credit: A. KarpathyNeocognitron [Fukushima, Biological CyberneMcs 1980]Deformation-ResistantRecognitionS-cells: (simple)- extract local featuresC-cells: (complex)- allow for positional errors9

6/1/17LeNet [LeCun et al. 1998] Stack mulMple stages of featureextractorsHigher stages compute moreglobal, more invariant featuresClassificaMon layer at the end Gradient-based learning applied to documentrecogniMon [LeCun, Bofou, Bengio, Haffner 1998]LeNet-1 from 1993ConvoluMonal Neural NetworksFeature mapsSpaMal poolingNon-linearityConvoluMon(Learned)Input ImageWhat is a ConvoluMon? Weighted moving sum.InputFeature AcMvaMon Map10

6/1/17Why ConvoluMon? Few parameters (filter weights)Dependencies are localTranslaMon invariance.Feature MapInputConvoluMonal Neural NetworksFeature mapsSpaMal poolingNon-linearityConvoluMon(Learned).Input ImageInputFeature MapConvoluMonal Neural NetworksFeature mapsSpaMal poolingRectified Linear Unit (ReLU)Non-linearityConvoluMon(Learned)Input Imageslide credit: S. Lazebnik11

6/1/17Non-Linearity Per-element (independent) OpMons:– Tanh– Sigmoid: 1/(1 exp(-x))– RecMfied linear unit (ReLU) Makes learning faster Simplifies backpropagaMon Avoids saturaMon issuesà Preferred opMonConvoluMonal Neural NetworksFeature mapsNormalizaMonMax poolingSpaMal : a non-linear down-sampProvide translation invarianceInput ImageSpaMal Pooling Average or max Non-overlapping / overlapping regions Role of pooling: Invariance to small transformaMonsLarger recepMve fields (see more of input)MaxAverage12

6/1/17Engineered vs. learned featuresConvolutional filters are trained in asupervised manner by back-propagatingclassification Mon/poolFeature extracMonConvoluMon/poolImageImageCompare: SIFT DescriptorImagePixelsLowe [IJCV 2004]Applyoriented filtersSpaMal pool(Sum)Normalize to unitlengthFeatureVectorCompare: SpaMal Pyramid MatchingSIFTfeaturesFilter withVisual WordsLazebnik,Schmid,Ponce[CVPR 2006]Take max VWresponseMulM-scalespaMal pool(Sum)Globalimagedescriptor13

6/1/17Previous Convnet successes Handwrifen text/digits– MNIST (0.17% error [Ciresan et al. 2011])– Arabic & Chinese [Ciresan et al. 2012] Simpler recogniMon benchmarks– CIFAR-10 (9.3% error [Wan et al. 2013])– Traffic sign recogniMon 0.56% error vs 1.16% for humans[Ciresan et al. 2011]ImageNet Challenge 2012 14 million labeled images, 20k classes Images gathered from Internet Human labels via Amazon Turk ImageNet Challenge: 1.2 million trainingimages, 1000 classes[Deng et al. CVPR 2009]A. Krizhevsky, I. Sutskever, and G. Hinton,ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012AlexNetSimilar framework to LeCun’98 but: Bigger model (7 hidden layers, 650,000 units, 60,000,000 params) More data (106 vs. 103 images) GPU implementaMon (50x speedup over CPU) Trained on two GPUs for a weekA. Krizhevsky, I. Sutskever, and G. Hinton,ImageNet Classification with Deep Convolutional Neural Networks, NIPS 201214

6/1/17AlexNet for image classificaMonAlexNet“car”Fixed input size: 224x224x3ImageNet ClassificaMon lks/2016/ILSVRC2016 10 09 clsloc.pdfIndustry Deployment Used in Facebook, Google, MicrosolStartupsImage RecogniMon, Speech RecogniMon, .Fast at test MmeTaigman et al. DeepFace: Closing the Gap to Human-Level Performance in FaceVerification, CVPR’1415

6/1/17Visualizing CNNs What input pafern originally caused a givenacMvaMon in the feature maps?Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Layer 1Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Layer 2Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]16

6/1/17Layer 3Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Layer 4 and 5Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]Beyond classificaMon DetecMonSegmentaMonRegressionPose esMmaMonMatching patchesSynthesisand many more 17

6/1/17R-CNN: Regions with CNN features Trained on ImageNet classificaMon Finetune CNN on PASCALRCNN [Girshick et al. CVPR 2014]Labeling Pixels: SemanMc LabelsFully Convolutional Networks for Semantic Segmentation [Long et al. CVPR 2015]Labeling Pixels: Edge DetecMonDeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection[Bertasius et al. CVPR 2015]18

6/1/17CNN for RegressionDeepPose [Toshev and Szegedy CVPR 2014]CNN as a Similarity Measure for MatchingStereo matching [Zbontar and LeCun CVPR 2015]Compare patch [Zagoruyko and Komodakis 2015]FlowNet [Fischer et al 2015]FaceNet [Schroff et al. 2015]Match ground and aerial images[Lin et al. CVPR 2015]CNN for Image GeneraMonLearning to Generate Chairs with Convolutional Neural Networks [Dosovitskiy et al. CVPR 2015]19

6/1/17Chair MorphingLearning to Generate Chairs with Convolutional Neural Networks [Dosovitskiy et al. CVPR 2015]Transfer Learning Improvement of learning in a new task through thetransfer of knowledge from a related task that hasalready been learned. Weight iniMalizaMon for CNNLearning and Transferring Mid-Level Image RepresentaMons usingConvoluMonal Neural Networks [Oquab et al. CVPR 2014]Deep learning libraries TensorflowCaffeTorchMatConvNet20

6/1/17Fooling CNNsIntriguing properties of neural networks [Szegedy ICLR 2014]What is going on?x E xx x α E xExplaining and Harnessing Adversarial Examples [Goodfellow ICLR -convnets/QuesMons?See you Tuesday!6321

ConvoluMonal Neural Networks Input Image ConvoluMon (Learned) Non-linearity SpaMal pooling Feature maps ConvoluMonal Neural Networks . ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012 . 6/1/17 1 5 AlexNet for image classificaMon “car” AlexNet Fixed input size: 224x224x3