Visual Object Recognition - University Of Texas At Austin

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingVisual Object RecognitionBastian LeibeComputer Vision LaboratoryETH ZurichChicago, 14.07.2008&Kristen GraumanDepartment of Computer SciencesUniversity of Texas in Austin

OutlineVisualPerceptualObjectandRecognitionSensory AugmentedTutorial Computing1. Detection with Global Appearance & Sliding Windows2. Local Invariant Features: Detection & Description3. Specific Object Recognition with Local Features― Coffee Break ―4. Visual Words: Indexing, Bags of Words Categorization5. Matching Local Features6. Part-Based Models for Categorization7. Current Challenges and Research DirectionsHighlight of some research topics not covered in the main tutorialK. Grauman, B. Leibe2

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingBenchmark Data What degree of difficulty do current datasets have?

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingExample: Caltech-101A dataset that hasbeen aboutmastered Images from the Caltech-101:101-way multi-class classification problemK. Grauman, B. Leibe

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingExample: Caltech256Images from the Caltech-256:256 multi-class recognition problemK. Grauman, B. Leibe

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingExample: Pascal Visual Object Classes ChallengePascal VOC 2007:Binary detection s/VOC/K. Grauman, B. Leibe

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingExample: LabelMehttp://labelme.csail.mit.edu/K. Grauman, B. Leibe

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingCurrent challenges & ongoing research Multi-cue integrationFiner level categorizationView invariant recognitionUnsupervised category discoveryLearning from noisily labeled imagesIntegration of segmentation and recognitionLearning with text and images/videoUse of videoContext and scene layout

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingMulti-cue integration Single cues often not sufficient. Integrate multiple local and global cues.K. Grauman, B. Leibe9

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingMulti-Category Discrimination Distinguish similar categories. Need to look at specific details!K. Grauman, B. Leibe10

Multi-Aspect Recognition Detectors for different viewpoints How can this beVisualPerceptualObjectandRecognitionSensory AugmentedTutorial Computingimproved?K. Grauman, B. Leibe11

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingMulti-Aspect Recognition[Hoiem, Rother, Winn, CVPR’07]K. Grauman, B. Leibe[Thomas et al., CVPR’06]12

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingMulti-Aspect Recognition[Rothganger et al., CVPR’03][Savarese & Fei-Fei, ICCV’07]K. Grauman, B. Leibe13

Unsupervised, semi-supervised category ry AugmentedTutorial ComputingTopic models for imagesProbabilistic LatentSemantic Analysis(pLSA)“face”Latent DirichletAllocation(LDA)“beach”cDπzNwSivic et al. ICCV 2005, Fei-Fei et al. ICCV 2005Figure credit: Fei-Fei Li

Unsupervised, semi-supervised category ry AugmentedTutorial ComputingClustering cluttered imagesLearning from noisy keyword-based image search resultsGrauman & Darrell, CVPR 2006Fergus et al. ECCV 2004, ICCV 2005Li & Fei-Fei, CVPR 2007

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingLearning with text and images/videoBarnard et al. JMLR 2003Berg, Berg, Edwards,& Forsyth, NIPS 2006Gupta et al. ECML 2008

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingIntegrating segmentation recognitionKumar et al. CVPR 2005Tu, Chen, Yuille, Zhu, ICCV 2003Borenstein & Ullman, ECCV 2002Kannan, Winn, & Rother, NIPS 2006

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingRole of context, understanding scene layoutAntonio Torralba, IJCV 2003

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingRole of context, understanding scene layoutImageWorldHoiem, Efros, & Hebert, CVPR 2006

Integration with Scene Geometry Goal: Find the ground planeRestrict object location¾ Assume Gaussian size prior Significantly reduced search spaceVisualPerceptualObjectandRecognitionSensory AugmentedTutorial ough VolumeDense stereoB. Leibe20

ory AugmentedTutorial Computing Combination with 3D Geometry[Leibe, Cornelis, Cornelis, Van Gool, CVPR’07] Mobile Pedestrian Detection[Ess, Leibe, Van Gool, ICCV’07]K. Grauman, B. Leibe21

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingDetections Using Ground Plane Constraintsleft camera1175 framesB. Leibe[Leibe07]22

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingExtensions: Tracking-by-Detection Spacetime trajectory analysis¾¾Link up detections to form physically plausible ST trajectoriesSelect set of ST trajectories that best explain the data[Leibe07]23

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingDynamic Scene Analysis ResultsB. Leibe[Leibe07]24

Extensions (2)VisualPerceptualObjectandRecognitionSensory AugmentedTutorial Computing Combination 3D Reconstruction[Cornelis, Leibe, Cornelis, Van Gool, 3DPVT’06]K. Grauman, B. Leibe25

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingTextured 3D ModelOriginal3D Reconstruction Run-times¾¾SfM Bundle adjustment: 27-30 fps on CPUDense reconstruction:36 fps on GPU[Cornelis, Cornelis, Van Gool, CVPR’06]B. Leibe26

Improved 3D City ModelVisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingEnhancing your driving experience Original3D Reconstruction[Cornelis, Leibe, Cornelis, Van Gool, 3DPVT’06] 27

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingPutting It All Together ysB. LeibexππdQVVTSoiIH11.ndiDtH i ,tiH2zx28

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingMobile Pedestrian Tracking[Ess, Leibe, Schindler, Van Gool, CVPR’08]29

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingMobile Tracking Through Crowds[Ess, Leibe, Schindler, Van Gool, CVPR’08]30

Extension: Recovering ensory AugmentedTutorial Computing1.N Idea: Only perform articulated tracking where it’s easy! Multi-person tracking¾Solves hard data association problem Articulated tracking¾Only on individual “tracklets” between occlusions[Gammeter, Ess, Jaeggli, Schindler, Leibe, Van Gool, ECCV’08]B. Leibe31

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingArticulated Multi-Person Tracking Multi-Person tracking¾¾¾Recovers trajectories and solves data associationEstimates 3D walking direction and speedDetects occlusion events[Gammeter, Ess, Jaeggli, Schindler, Leibe, Van Gool, ECCV’08]B. Leibe32

VisualPerceptualObjectandRecognitionSensory AugmentedTutorial ComputingArticulated Tracking under Egomotion[Gammeter, Ess, Jaeggli, Schindler, Leibe, Van Gool, ECCV’08]B. Leibe33

K. Grauman, B. LeibeVisualPerceptualObjectandRecognitionSensory AugmentedTutorial Computing34

SummaryVisualPerceptualObjectandRecognitionSensory AugmentedTutorial Computing Visual recognition is a challenging and very activeresearch area. We’ve covered some basic models and representationsthat have been shown to be effective, and highlightedsome ongoing issues. See tutorial website for slides, links, references.http://www.vision.ee.ethz.ch/ bleibe/teaching/tutorial-aaai08/Thank you!K. Grauman, B. Leibe

Perceptual and Sensory Augmented ComputingVisual Object Recognition Tutorial Unsupervised, semi-supervised category discovery w N c z D π “beach” Latent Dirichlet Allocation (LDA) Sivic et al. ICCV 2005, Fei-Fei et al. ICCV 2005 Figure credit: Fei-Fei Li Probabilistic Latent Semanti