CSC420: Intro To Image Understanding Introduction
CSC420: Intro to Image UnderstandingIntroductionSanja FidlerSeptember 9, 2019Sanja FidlerIntro to Image Understanding1 / 63
The TeamInstructor:Sanja Fidler (fidler@cs.toronto.edu)Office: DH 3084Office hours: Monday 11am-12.30pm, or by appointmentTA:Huan Ling (linghuan@cs.toronto.edu)Sanja FidlerIntro to Image Understanding2 / 63
Course InformationClass time: Monday at 9-11amLocation: MN 1170Tutorials: TUT0101 on Monday 11am-12pm (MN 2110),TUT0102 on Monday 12-1pm (MN 3100), demos and Q&A,we’ll do it on demandClass Website:http://www.cs.toronto.edu/ fidler/teaching/2019Fall/CSC420.htmlThe class will use Piazza for announcements and 9/csc420utm/homeYour grade will not depend on your participation on Piazza.It’s just a good way for asking questions, discussing with yourinstructor, TAs and your peersSanja FidlerIntro to Image Understanding3 / 63
Course InformationClass time: Monday at 9-11amLocation: MN 1170Tutorials: TUT0101 on Monday 11am-12pm (MN 2110),TUT0102 on Monday 12-1pm (MN 3100), demos and Q&A,we’ll do it on demandClass Website:http://www.cs.toronto.edu/ fidler/teaching/2019Fall/CSC420.htmlThe class will use Piazza for announcements and 9/csc420utm/homeYour grade will not depend on your participation on Piazza.It’s just a good way for asking questions, discussing with yourinstructor, TAs and your peersSanja FidlerIntro to Image Understanding3 / 63
Course InformationTextbook: We won’t directly follow any book, but extra readingin this textbook will be useful:Rick SzeliskiComputer Vision: Algorithms and Applicationsavailable free online:http://szeliski.org/Book/Links to other material (papers, code, etc) will be posted on theclass webpageSanja FidlerIntro to Image Understanding4 / 63
Course PrerequisitesCourse Prerequisites:Data structuresLinear AlgebraVector calculusWithout this you’ll need some serious catching up to do!Knowing some basics in this is a plus:Python, Matlab, C Machine LearningNeural NetworksSolving assignments sooner rather than laterSanja FidlerIntro to Image Understanding5 / 63
RequirementsEach student expected to complete 4 assignments and a projectAssignments:Short theoretical questions and programming exercisesWill be given roughly every two weeks (starting second week of class)You will have a week to hand in the solution to each assignmentYou need to solve the assignment aloneSanja FidlerIntro to Image Understanding6 / 63
RequirementsEach student expected to complete 4 assignments and a projectAssignments:Short theoretical questions and programming exercisesWill be given roughly every two weeks (starting second week of class)You will have a week to hand in the solution to each assignmentYou need to solve the assignment aloneProject:You will be able to choose from a list of projects or come up with yourown project (discussed prior with your instructor)Need to hand in a report and do an oral presentationCan work individually or in pairsSanja FidlerIntro to Image Understanding6 / 63
RequirementsEach student expected to complete 4 assignments and a projectAssignments:Short theoretical questions and programming exercisesWill be given roughly every two weeks (starting second week of class)You will have a week to hand in the solution to each assignmentYou need to solve the assignment aloneProject:You will be able to choose from a list of projects or come up with yourown project (discussed prior with your instructor)Need to hand in a report and do an oral presentationCan work individually or in pairsSanja FidlerIntro to Image Understanding6 / 63
GradingGrade breakdownAssignments: 60% (15% each)Project: 40%For the project you will need to doShort project proposalProject reportProject presentation (oral)I will be asking questions about relevant part of the class materialduring project presentations which will influence the gradeSanja FidlerIntro to Image Understanding7 / 63
Term Work DatesTerm WorkPost DateDue DateAssignment 1Sept 20Sept 27Assignment 2Oct 4Oct 11Assignment 3Oct 25Nov 1Assignment 4Nov 8Nov 15Project ReportFirst week of DecemberProject PresentationSecond week of DecemberAll dates are for 2019Dates are approximate (depend on what material we cover in class)Sanja FidlerIntro to Image Understanding8 / 63
Programming Language?Your assignments / project can be in Python, Matlab, C As long as it compiles, runs, and you know how to defend it,we’re happyHOWEVER, most code and examples we will provide during theclass will be in Python, MatlabChoose wiselySanja FidlerIntro to Image Understanding9 / 63
LatenessDeadline The solutions to assignments / project should besubmitted by 11.59pm on the date they are due.Anything from 1 minute late to 24 hours will count as onelate day.Lateness Each student will be given a total of 3 free late days.This means that you can hand in three of the assignmentsone day late, or one assignment three days late. It is up tothe you to make a good planning of your work. After youhave used the 3 day budget, the late assignments willnot be accepted.Sanja FidlerIntro to Image Understanding10 / 63
SyllabusTentative syllabusWeek nb.12345678910111213Sanja FidlerDateSept 9Sept 16Sept 23Sept 30Oct 7Oct 21Oct 28Nov 4Nov 11Nov 18Nov 25Dec 2Dec ?TopicIntroLinear filters, edgesImage featuresKeypoint detectionMatchingGroupingStereo, multi-viewStereo, multi-viewObject recognitionObject detectionNeural NetworksSegmentationProject PresentationsIntro to Image Understanding11 / 63
IntroductionSanja FidlerIntro to Image Understanding12 / 63
Let’s begin!Introduction to Intro to Image UnderstandingWhat is Computer Vision?Why study Computer Vision?Which cool applications can we do with it?Is vision a hard problem?Sanja FidlerIntro to Image Understanding13 / 63
What is Computer Vision?Sanja FidlerIntro to Image Understanding14 / 63
What is Computer Vision?A field trying to develop automatic algorithms that would “see”Sanja FidlerIntro to Image Understanding15 / 63
Embodied AgentsUnderstand the scene in order to take actions: perception, planning,reasoningFigure: How do I make dinner in this household?Many simulators: Carla, Thor, House3D, VirtualHome, etcSanja FidlerIntro to Image Understanding16 / 63
What is Computer Vision?What does it mean to see?[text adopted from A. Torralba]To know what is where by looking – Marr, 1982Sanja FidlerIntro to Image Understanding17 / 63
What is Computer Vision?What does it mean to see?[text adopted from A. Torralba]To know what is where by looking – Marr, 1982Understand where things are in the worldSanja FidlerIntro to Image Understanding17 / 63
What is Computer Vision?What does it mean to see?[text adopted from A. Torralba]To know what is where by looking – Marr, 1982Understand where things are in the worldWhat are their 3D/material properties?imageSanja FidlerIntro to Image Understanding17 / 63
What is Computer Vision?What does it mean to see?[text adopted from A. Torralba]To know what is where by looking – Marr, 1982Understand where things are in the worldWhat are their 3D/material properties?What actions are taking place?Depth pic from http://vladlen.infoSanja FidlerIntro to Image Understanding17 / 63
What is Computer Vision?What does it mean to see?[text adopted from A. Torralba]To know what is where by looking – Marr, 1982Understand where things are in the worldWhat are their 3D/material properties?What actions are taking place?Pic from www.cobblehillpuzzles.comSanja FidlerIntro to Image Understanding17 / 63
“Full” Image Understanding?Full understanding of an image?Sanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about it[M. Malinowski, M. Fritz, A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input,NIPS, 2014]Sanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
“Full” Image Understanding?Full understanding of an image? You can answer any question about itSanja FidlerIntro to Image Understanding18 / 63
Why study Computer Vision?Sanja FidlerIntro to Image Understanding19 / 63
Why study Computer Vision?You are curious how to one day make the robot walk your doghttp://www.cs.toronto.edu/ fidler/videos/robotsmovies.movSanja FidlerIntro to Image Understanding20 / 63
Why study Computer Vision?. and fold your laundryhttps://www.youtube.com/watch?v gy5g33S0GzoSanja Fidlerhttps://www.youtube.com/watch?v KKUaVzf3OqwIntro to Image Understanding21 / 63
Why study Computer Vision?. and drive you to workAmnon Shashua’s Mobileye autonomous driving systemhttps://www.youtube.com/watch?v 4fxFDypHZLsSanja FidlerIntro to Image Understanding22 / 63
Why study Computer Vision?Allows you to manipulate your imagesScene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007Sanja FidlerIntro to Image Understanding23 / 63
Why study Computer Vision?Allows you to manipulate your imagesScene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007Sanja FidlerIntro to Image Understanding23 / 63
Why study Computer Vision?Allows you to manipulate your imagesScene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007Sanja FidlerIntro to Image Understanding23 / 63
Why study Computer Vision?Allows you to manipulate your imagesScene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007Sanja FidlerIntro to Image Understanding23 / 63
Why study Computer Vision?Allows you to manipulate your imagesScene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007Sanja FidlerIntro to Image Understanding23 / 63
Why study Computer Vision?Allows you to manipulate your imagesScene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007Sanja FidlerIntro to Image Understanding23 / 63
Why study Computer Vision?Allows you to manipulate your imageshttps://www.youtube.com/watch?v p5U4NgVGAwgGauGan, Ming-Yu Liu et al., http://nvidia-research-mingyuliu.com/gaugan/]Sanja FidlerIntro to Image Understanding24 / 63
Why study Computer Vision?Change style of images[Gatys, Ecker, Bethge. A Neural Algorithm of Artistic Style. Arxiv’15.]Sanja FidlerIntro to Image Understanding25 / 63
Why study Computer Vision?Change style of videoshttps://www.youtube.com/watch?v Khuj4ASldmU[Ruder, Dosovitskiy, Brox. Artistic style transfer for videos, 2016]Sanja FidlerIntro to Image Understanding26 / 63
Why study Computer Vision?Change style of videoshttps://arxiv.org/pdf/1701.04928.pdfSanja FidlerIntro to Image Understanding27 / 63
Why study Computer Vision?. and make cool videos using a single imagehttp://www.cs.cmu.edu/ om3d/3D Object Manipulation in a Single Photograph using Stock 3D Models,Kholgade, Simon, Efros, Sheikh, SIGGRAPH 2014Sanja FidlerIntro to Image Understanding28 / 63
Why study Computer Vision?Fancy visualization and game analysis in sportsSanja FidlerIntro to Image Understanding29 / 63
Why study Computer Vision?Fancy visualization and special e ects in movies[Source: http://cvfxbook.com andSanja Fidlerhttp://vimeo.com/100095868]Intro to Image Understanding30 / 63
Why study Computer Vision?Reconstruct the world in 3D from online photos!https://www.youtube.com/watch?v IgBQCoEfiMsPhotosynth, https://photosynth.net/ (try it!)Sanja FidlerIntro to Image Understanding31 / 63
Why study Computer Vision?Figure out what people are wearinghttp://clothingparsing.comSanja FidlerIntro to Image Understanding(try it!)32 / 63
Why study Computer Vision?[S. Zhu, C.C Loy, D. Lin, R. Urtasun, S. Fidler. In submission.]Sanja FidlerIntro to Image Understanding33 / 63
Why study Computer Vision?[S. Zhu, C.C Loy, D. Lin, R. Urtasun, S. Fidler. In submission.]Sanja FidlerIntro to Image Understanding33 / 63
Why study Computer Vision?[S. Zhu, C.C Loy, D. Lin, R. Urtasun, S. Fidler. In submission.]Sanja FidlerIntro to Image Understanding33 / 63
Why study Computer Vision?[S. Zhu, C.C Loy, D. Lin, R. Urtasun, S. Fidler. In submission.]Sanja FidlerIntro to Image Understanding33 / 63
Why study Computer Vision?Figure: Modiface: Toronto-based startupSanja FidlerIntro to Image Understanding34 / 63
Why study Computer Vision?Play with facesSanja FidlerIntro to Image Understandinghttps://www.faceapp.com/(try it!)35 / 63
Why study Computer Vision?Play with facesSanja FidlerIntro to Image Understanding35 / 63
Why study Computer Vision?Play with facesSanja FidlerIntro to Image Understanding35 / 63
Why study Computer Vision?Play with facesSanja FidlerIntro to Image Understanding35 / 63
Why study Computer Vision?Generate new faceshttps://www.youtube.com/watch?v kSLJriaOumAStyleGAN, Tero Karras et al., https://github.com/NVlabs/stylegan]Sanja FidlerIntro to Image Understanding36 / 63
Why study Computer Vision?Generate image captions automatically[Source: L. Zitnick, NIPS’14 Workshop on Learning Semantics]Sanja FidlerIntro to Image Understanding37 / 63
Why study Computer Vision?Generate image captions automatically[Source: L. Zitnick, NIPS’14 Workshop on Learning Semantics]Sanja FidlerIntro to Image Understanding37 / 63
Why study Computer Vision?Generate image captions automatically[Source: L. Zitnick, NIPS’14 Workshop on Learning Semantics]Sanja FidlerIntro to Image Understanding37 / 63
Why study Computer Vision?Generate image captions automatically[Kiros, Salakhutdinov, Zemel. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. 2014]Sanja FidlerIntro to Image Understanding37 / 63
Why study Computer Vision?Have a computer do math for youFigure: Photomath: https://photomath.net/,Sanja Fidlerhttp://www.youtube.com/watch?v XlbVB50mIh4Intro to Image Understanding38 / 63
Why study Computer Vision?Fingerprint recognition[Source: S. Lazebnik]Sanja FidlerIntro to Image Understanding39 / 63
Why study Computer Vision?You can do some movie-like ForensicsFigure: Source: Nayar and Nishino, “Eyes for Relighting”[Source: N. Snavely]Sanja FidlerIntro to Image Understanding40 / 63
Why study Computer Vision?Source: Nayar and Nishino, “Eyes for Relighting”[Source: N. Snavely]Sanja FidlerIntro to Image Understanding41 / 63
Why study Computer Vision?Figure: Source: Nayar and Nishino, “Eyes for Relighting”[Source: N. Snavely]Sanja FidlerIntro to Image Understanding41 / 63
Why study Computer Vision?Some more CSICan you see something on the wall?Torralba & Freeman, CVPR’12Sanja FidlerIntro to Image Understanding42 / 63
Why study Computer Vision?Some more CSIFidlerTorralbaSanja& Freeman,CVPR’12 Intro to Image Understanding42 / 63
Why study Computer Vision?Object recognition (in mobile phones)[Source: S. Seitz]Sanja FidlerIntro to Image Understanding43 / 63
Why study Computer Vision?Games, games & games: 3D Pose Estimation with Depth Sensors[Source: Microsoft Kinect]Sanja FidlerIntro to Image Understanding44 / 63
How It All Began.Sanja FidlerIntro to Image Understanding45 / 63
How It All Began.[Slide credit: A. Torralba]Sanja FidlerIntro to Image Understanding46 / 63
50 years and thousands of PhDs later.Popular benchmarks: KITTI, PASCAL, Cityscapes, MS-COCOReasoning demo: http://vqa.cloudcv.org/Sanja FidlerIntro to Image Understanding47 / 63
Why is vision hard?Sanja FidlerIntro to Image Understanding48 / 63
Why is vision hard?Half of the cerebral cortex in primates is devoted to processing visualinformation. This is a lot. Means that vision has to be pretty hard!Sanja FidlerIntro to Image Understanding49 / 63
Why is vision hard?All this is dog.Sanja Fidler[slide adopted from: R. Urtasun]Intro to Image Understanding50 / 63
Why is vision hard?Biederman, 1987[slide credit: R. Urtasun]Sanja FidlerIntro to Image Understanding51 / 63
Why is vision hard?Lots of data to process:Thousands to millions of pixelsin an image100 hours of video added toYouTube per minute [source:YouTube]Over 6 billion hours of videoare watched each month onYouTube – almost an hour forevery person on Earth [source:YouTube]Sanja FidlerIntro to Image Understanding52 / 63
Why is vision hard?Lots of data to process: 5000 new tagged photos added to Flickr per minute (7M per day) 60M photos uploaded to Instagram every day [source: Instagram]Sanja FidlerIntro to Image Understanding53 / 63
Exploit so Much Data!Figure: Vemodalen: The Fear That Everything Has Already Been Done,https://www.youtube.com/watch?v 8ftDjebw8aA[Source: L. Zitnick, NIPS’14 Workshop on Learning Semantics]Sanja FidlerIntro to Image Understanding54 / 63
Why is vision hard?Human vision seems to work quite well.How well does it really work?Let’s play some games!Sanja FidlerIntro to Image Understanding55 / 63
How good are humans?Which square is lighter, A or B?[Slide credit: A. Torralba]Sanja FidlerIntro to Image Understanding56 / 63
How good are humans?Which square is lighter, A or B?[Slide credit: A. Torralba]Sanja FidlerIntro to Image Understanding56 / 63
How good are humans?Figure: 2006 Walt AnthonyWhich red line is longer?[Slide credit: A. Torralba]Sanja FidlerIntro to Image Understanding57 / 63
How good are humans?Figure: 2006 Walt AnthonyWhich red line is longer?[Slide credit: A. Torralba]Sanja FidlerIntro to Image Understanding57 / 63
How good are humans?Figure: Ames roomAssumptions can be wrong[Slide credit: A. Torralba]Sanja FidlerIntro to Image Understanding58 / 63
How good are humans?Figure: Chabris & Simons, https://www.youtube.com/watch?v vJG698U2MvoCount the number of times the white team pass the ballConcentrate, it’s difficult!https://www.youtube.com/watch?v vJG698U2MvoSanja FidlerIntro to Image Understanding59 / 63
How good are humans?Figure: Simons et al., http://www.perceptionweb.com/perception/perc1000/a d ex1.mov (more videoshere: http://www.perceptionweb.com/misc.cgi?id p3104)Is something happening in the picture?Sanja FidlerIntro to Image Understanding60 / 63
How good are humans?Figure: Torralba et .870/slides/blur.aviCan you describe what’s going on in the video?Sanja FidlerIntro to Image Understanding61 / 63
How good are humans?Figure: Torralba et .870/slides/highres.aviCan you describe what’s going on in the video?Sanja FidlerIntro to Image Understanding62 / 63
What do I need.What do I need to become a good Computer Vision researcher?Technical capabilitiesGood programming skillsImaginationEven better intuitionLots of persistenceSome luck always helpsSanja FidlerIntro to Image Understanding63 / 63
Scene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007 . Scene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007 Sanja Fidler Intro to Image Understanding 23/63. Why study Computer Vision? Allows you to manipulate your images Scene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007
Scene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007 . Scene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007 Sanja Fidler Intro to Image Understanding 23/60. Why study Computer Vision? Allows you to manipulate your images Scene Completion using Millions of Photographs, Hays & Efros, SIGGRAPH 2007
L2: x 0, image of L3: y 2, image of L4: y 3, image of L5: y x, image of L6: y x 1 b. image of L1: x 0, image of L2: x 0, image of L3: (0, 2), image of L4: (0, 3), image of L5: x 0, image of L6: x 0 c. image of L1– 6: y x 4. a. Q1 3, 1R b. ( 10, 0) c. (8, 6) 5. a x y b] a 21 50 ba x b a 2 1 b 4 2 O 46 2 4 2 2 4 y x A 1X2 A 1X1 A 1X 3 X1 X2 X3
Publication 1398-5.2 – PDF 1997 Table of Contents IntroTable of Contents Table of Contents Intro-3 List of Figures Intro-9 List of Tables Intro-13 Preface Intro-17 Who Should Use this Manual.Intro-17 ULTRA 100 Series Product Receiving and Storage Responsibility. .
ART V02A Intro to Hist of Western Art I 3 ARHS 200 Art of Western World I 3 EHAP, TCNA ART V02B Intro to Hist of West Art II 3 ARHS 2XXX Intro to Hist of West Art II 3 EHAP, EHAP ART V02C Intro to Non-Western Art 3 ARHS 2XXX Intro to Non-Western Art 3 ART V02D Art of Ancient Mediterranean 3
Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image 1. The Imperial – Mumbai 2. World Trade Center – Mumbai 3. Palace of the Sultan of Oman – Oman 4. Fairmont Bab Al Bahr – Abu Dhabi 5. Barakhamba Underground Metro Station – New Delhi 6. Cybercity – Gurugram 7.
work/products (Beading, Candles, Carving, Food Products, Soap, Weaving, etc.) ⃝I understand that if my work contains Indigenous visual representation that it is a reflection of the Indigenous culture of my native region. ⃝To the best of my knowledge, my work/products fall within Craft Council standards and expectations with respect to
facile. POCHOIR MONOCHROME SUR PHOTOSHOP Étape 1. Ouvrez l’image. Allez dans Image Image size (Image Taille de l’image), et assurez-vous que la résolution est bien de 300 dpi (ppp). Autre-ment l’image sera pixe-lisée quand vous allez l’éditer. Étape 2. Passez l’image en noir et blanc en choisissant Image Mode Grays-
April 23-25, 2018 ASTM International Headquarters West Conshohocken, Pennsylvania, USA October 14-17, 2018 The Pulitzer Hotel Amsterdam, The Netherlands Annual Business Meeting June 24, 2018 Sheraton San Diego Hotel & Marina San Diego, California, USA. 2018 Board of Directors www.astm.org 3 Chairman of the Board Dale F. Bohn Vice Chairmen of the Board Taco van der Maten Andrew G. Kireta Jr .
