OpenCV 3 - Silvio Savarese

InfillingOpenCV 3.0Uses in Robotics and ARGary BradskiVP Perception and Core Software,Magic LeapDirector: OpenCV Foundation1

OpenCV Thanks!For Key Support Intel– for getting it started and helping fundthe challenge Google– for growing support in Google Summerof Code over the last 5 years Nvidia– Supporting Cuda version with lots ofother help2

Outline: OPENCV 3.0 Intro– Learning OpenCV Version 2.0 coming by Aug– Announcing 50K Vision Challenge OpenCV BackgroundOpenCV 3.0 High LevelOpenCV 3.0 ModulesBrand New in OpenCVOpenCV Examples– Robotics– Augmented Reality3

Learning OpenCV V2.0 Out in Summer 2014!In C :V2.04

OpenCV 50K VisionChallenge5

OpenCV Background6

What is OpenCV Open Source Computer VisionLibrary Routines focused on real time imageprocessing and 2D 3D computervision.– On Linux, Windows, Mac, Androidand iOS– C , C, Java, Matlab and Pythoninterfaces Free for commercial or research usein whole or in part.7

OpenCV License Based on BSD license Free for commercial and researchuse Does not force your code to be open You need not contribute back–But you are very welcome to contributeback!8

RenewedIntel SupportOpenCVFoundationOpenCV HistoryNvidia SupportIntel SupportWillow SupportAlpha Release at CVPR 2000. Windows only.Release2.3.PythonGPU SupportRelease 3.0 RefactoreBeta 1. Linux supportRelease2.0.C OpenCV StartedRelease 1.1 Release 2.4BetaBeta2 Beta3 Beta45 Release 003200220012000209919Main Current Sponsors:Google Summer of Code9

Environments, Platforms Languages:– C , C#, Python, C, Java Platforms:10

OpenCV and HardwareAcceleration OpenCV was a central basis forOpenVX– a hardware abstraction layer– for embedded vision acceleration OpenVX Supporters:11

12

Where is OpenCV Used? Academic and Industry ResearchSecurity systemsGoogle Maps, StreetviewImage/video search and retrievalStructure from motion in moviesMachine vision factory production inspection systemsAutomatic Driver Assistance SystemsSafety monitoring (Dam sites, mines, swimming pools)Robotics – personal, industrial, hobbyCoin production in China2M downloads

PopularityOver 9M downloads!Ramping to 160K downloads/month

OpenCV Corporation Founded this July, 2012Documentation: http://opencv.org (user site)http://docs.opencv.org http://code.opencv.org (developer site) Contribute (via Credit, debit or paypal):– http://tinyurl.com/7eujyo2For corporatesupportAnd/or partnership, contactGarybradski@gmail.comI am looking forentrepreneurial people tostaff up OpenCV: VisionBusiness DevSoftwareHardware15

What’s In OpenCV High level16

Robot supporOpenCV Overview: 2500 algorithmseloper http://code.opencv.org; User: http://opencv.orgImage PyramidsGeneral Image Processing reo, 3DSegmentationFeaturesUtilities andData g Detection, RecognitionMatrix Math17

OpenCV Algorithm ModulesOverviewHighGUI:I/O, InterfaceImageProcessingTransformsFittingOptical pth, PoseObjectrecognitionMachinelearningComputational PhotographyNormals,Planes, 3DFeaturesCORE:Data structures, Matrix math, Exceptionsetc18

Machine Learning Library (MLL)CLASSIFICATION / REGRESSIONFast Approximate NN (FLANN)Extremely Random TreesCARTNaïve BayesMLP (Back propagation)Statistical Boosting, 4 flavorsRandom ForestsSVMFace Detector(Histogram bis ABBCBBCTUNING/VALIDATIONCross validationBootstrappingVariable importanceSampling methodshttp://opencv.org1919

20

21

OpenCV Architecture andDevelopment22

23

24

25

What’s In OpenCV 3.0 Modules26

CORE:OpenCV Modules: CoreData structures,Matrix math,Exceptions etcCore27

OpenCV Modules: HighGUIHighGUI:I/O, Interface28

OpenCV Modules: Image ProcessingImageProcessing29

OpenCV Modules: TransformsTransformsHoughLog-Polar30

OpenCV Modules: FittingFitting2D Rigid ObjectsEllipseDelaunay3DConvex Hull31

OpenCV Modules: Optic Flow, Trackhttp://www.youtube.com/watch?v bWyBGmzfP-g// opencv/samples/c/lkdemo.cOpticalFlowint main( ){Tracking CvCapture* capture ?cvCaptureFromCAM(camera id) :cvCaptureFromFile(path);lkdemo.c, 190 linesif( !capture ) return -1;(needs camera to run)for(;;) {I ( x dx, y dy, t dt ) I ( x, y , t );IplImage* frame cvQueryFrame(capture); I / t I / x ( dx / dt ) I / y ( dy / dt );if(!frame) break;// copy and process imageG X b,cvCalcOpticalFlowPyrLK( ) I x2 , I x I y I cvShowImage( “LkDemo”, result ); , b It x X ( x, y ), G I x I y , I y2 c cvWaitKey(30); // run at 20-30fps speed Iy if(c 0) {// process e.com/watch?v 1osj7kRgswk

OpenCV Modules: SegmentationGrab youtube.com/watch?v OxmDonZjahttp://www.youtube.com/watch?v Ktrjh5-KLKoWatershed33

CalibrationOpenCV Modules: CalibrationHomography3D view of checkerboardhttp://www.youtube.com/watch?v DrXIQfQHFv0Un-distorted image34http://www.youtube.com/watch?v PuWQ

OpenCV Modules: Features, VSLAMFeaturesVSLAMRead two input images:Mat img1 imread(argv[1], CV LOAD IMAGE GRAYSCALE);Detect keypoints in both images:// detecting keypointsFastFeatureDetector detector(15);vector KeyPoint keypoints1;detector.detect(img1, keypoints1);Compute descriptors for each of the keypoints:// computing descriptorsSurfDescriptorExtractor extractor;Mat descriptors1;extractor.compute(img1, keypoints1, descriptors1);Now, find the closest matches between descriptors from the first image to the second:// matching descriptorsBruteForceMatcher L2 float matcher; Change one or both of thesevector DMatch matches;to switch detector and/ormatcher.match(descriptors1, descriptors2, matches);linesdescriptor types3535

OpenCV Modules: Depth, PoseDepth, PoseNormals,Planes, 3DFeaturesLeft – right feature alignment:Some examples of3D stereo depth maps:36

OpenCV Modules: Obj Rec/MLObjectrecognitionMachine w.youtube.com/watch?v RF0VpR4xog37

OpenCV Modules: Comp PhotogComputationalPhotographyImage Stitching(Occipital Corp.)Tilt-shiftTexturalInpainting38

Brand New in OpenCV3.0User Contrib Module Thanks to Google Summer of Code!!– Supporting 15 interns!Accepted pull requests:xtended Python interface 8. New line segment detector9. Haze removal, depth estimationD object recognition and poseKAZE features10.GPU accelerated dense optical floar detection11.DTAM & pose estimationomputational photography 12.PNP pose detectionustom calibration and planar13.VisualARsaliency filtersense optical flow14.Text detection and reading in wild15.TLD tracker39

OpenCV Examples Industrial Perception Magic Leap40

Industrial Perception Sensor driven, Real time planning Applied to distribution41

Magic Leap Augmented Reality done right Lots of computer vision (We’re hiring) Gesture recognition demo42

Gesture was going to be livedemo43

Questions?44Photo: Gary Bradski

Language Modules GPU/CudaAndroidiOSPythonJava45

OpenCV GPU Module: Image processingbuilding ilteringmatchindetectorenginegs High-level algorithms:Stereo matchingFace detectionFeaturematching46

OpenCV GPU ModuleExampleMat frame;VideoCapture capture(camera);cv::HOGDescriptor hog;Mat frame;VideoCapture capture(camera);cv::gpu::HOGDescriptor ure frame;capture frame;GpuMat gpu frame;gpu frame.upload(frame);vector Rect found;hog.detectMultiScale(frame, found,1.4, Size(8, 8), Size(0, 0),1.05, 8);vector Rect found;hog.detectMultiScale(gpu frame,found,1.4, Size(8, 8), Size(0, 0),1.05, 8); Designed verysimilar!47

OpenCV GPU ModulePerformanceTesla C2050 (Fermi) vs. Core i5760 2.8GHz (4 cores, TBB, SSE)– Average speedup for primitives:33 For “good” data (large images arebetter) Without copying to GPUWhat can you get from yourcomputer?– opencv\samples\gpu\perfomance48

OpenCV Android ModuleOpenCV 2.4 for Android:Native Android Camera SupportMultithreadingJava API (soon)Tegra HW Optimizations (soon)Wiki with the latest mlSupport/discussion encv49

OpenCV iOS Module Full support50

OpenCV Python Module Full Python interface Example: Depth image from Kinect:51

OpenCV Java Module52

Book and Foundation53

Learning OpenCV V2.0 Out in Summer 2014!In C :V2.054

OpenCV Foundation Support55

Outline: OPENCV 3.0 Intro –Learning OpenCV Version 2.0 coming by Aug –Announcing 50K Vision Challenge OpenCV Background OpenCV 3.0 High Level OpenCV 3.0 Modules Brand New