Introduction To Computer Vision Using OpenCV

The most trusted source of analysis, advice, and engineeringfor embedded processing technology and applicationsIntroduction to Computer Visionusing OpenCVEric GregoriPresented at the2012 Embedded Systems Conference in San JoseBerkeley Design Technology, Inc.Oakland, California USA 1 (510) 451-1800info@BDTI.comhttp://www.BDTI.comCopyright 2012 Berkeley Design Technology, Inc.1

What is OpenCV?An open sourcelibrary of over500 functionsOver 2 dozenexamplesAn easy tool forexperimentingwith computervisionC/C r 3,000,000downloads 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY2

What Can OpenCV Do? 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY3

OpenCV in The Embedded SpaceOpenCV has always been available to theembedded space under Linux.The library has been ported to: PowerPC,MIPS, Blackfin, Xscale and ARM.If it can run Linux, it can run agle-has-2-eyes-opencv-stereo-on.htmlOn YouTube, you can find demonstrations of OpenCVrunning on the TI Beagleboard, Freescale i.MX53Quick Start Board, and various NVIDIA based tabletsand phones.In the mobile market you can find examples ofOpenCV running on the iPhone and of-course Androiddevices. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY4

OpenCV Licensing“OpenCV is released under a BSD license, it is free for both academicand commercial use.”“The BSD License allows proprietary use, and for the software releasedunder the license to be incorporated into proprietary products. Worksbased on the material may be released under a proprietary license oras closed source software. This is the reason for widespread use of theBSD code in proprietary products, ranging from JuniperNetworks routers to Mac OS X.” 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY5

Try OpenCV for Yourself BDTI has created some interactive examples to demonstrate just asmall part of what you can do with OpenCV. The build tools, OpenCV libraries, and examples are shipped as aVMware image. The BDTI OpenCV VMware image provides an easy to use pre-builtenvironment to get you up and running on OpenCV in minutes. Simply download the free VMware player and BDTI OpenCV VMwareimage and get started in developing with OpenCV now. You can download the BDTI OpenCV VMware image from theEmbedded Vision Alliance website at: eImage 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY6

GETTING STARTED WITH OPENCV THE EASY WAY 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY7

All The Installation and Configuration Has BeenDone for YouThe Ubuntu OSand GCC compilerruns in a VMwareimage.OpenCV is preinstalled andconfigured with allsource.Exampleapplications usethe Eclipsegraphicaldebuggingenvironment. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY8

The Basics—An “Image” and A “Frame” Both images and frames aremade up of individual pixelsorganized in a 2 dimensionalarray. For a color image, each pixelcan be anything from 8 to 32 bitswide. Most monochrome images use 8bits per pixel. A frame is a single image in avideo sequence. 2012 BDTIpixelYANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGYX9

“Feature”—A Fundamental Concept in ComputerVisionfeature (fchr)n.A prominent or distinctive aspect, quality, or characteristic: a feature of one’spersonality; a feature of the The concept of, “a feature of an object” is very important for most computervision algorithms.In a image or frame, a feature is a group of pixels with some uniqueattribute.corner 2012 BDTIpointsedgecontrastANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGYmotion10

Some Basic “Building Block” Algorithms Used inComputer Vision Detection Motion Detection—Finds groups of pixels (features) that are inmotion (change in position from one frame to the next). Line Detection—Finds groups of pixels (features) that areorganized in straight lines, along edges. Face Detection—Finds groups of pixels organized in a group thatfits the template of a face. Tracking Optical Flow based tracking—A combination of algorithms usedto track moving objects in a video using features. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY11

MOTION DETECTION 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY12

Motion Detection Motion detection in this context is doneusing frame subtraction, commonlyreferred to as background subtraction. The video is converted to monochromeand each pixel in the previous frame issubtracted from the current frame. If nothing changed between frames, theresult of all the pixel subtractions will be 0.previous frame – current frame 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY13

Demo 1—Motion Detection LearnRate—Regulates theupdate speed (how fast theaccumulator “forgets” aboutearlier images). Threshold—The minimum valuefor a pixel difference to beconsidered moving. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY14

LINE DETECTION 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY15

Line Detection Using EdgesIn a monochrome image, a line is defined as a group of pixelsorganized along a straight edge.An edge in a monochrome image is defined as a dark pixel next to alighter pixel.original image 2012 BDTIdetected edgeANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY16

Line Detection Using Edges After the edge detector finds the edges,another algorithm called the Houghtransform finds edge pixels that line-up instraight lines. Straight lines are a valuable feature in aimage. Straight lines define the boundaries ofobjects in an image and can be used fortracking purposes in a video stream. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY17

Demo 2—Line Detection Using Edges Threshold—Set the minimum differencebetween adjoining groups of pixels to beclassified as an edge. MinLength—The minimum number of“continuous” edge pixels required to beclassified as a straight line. MaxGap—The maximum number ofmissing edge pixels within a straight line,while still being considered “continuous”. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY18

FACE DETECTION 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY19

Face DetectionAs the name implies, face detection is used to find faces in an image. Theunderlying algorithm can actually be used to detect any object. This algorithmis trained to look for specific features, in a specific order.Training is done offline, and isaccomplished by “showing” thelearning algorithm both positiveand negative images (imageswith a face and without a face).The result of the training is a filethat describes the object todetect, using very specificfeatures. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY20

Face DetectionFace detection, as demonstrated in this context, uses a set of fourdistinct templates to define unique features. Templates are usedbecause they can be processed faster then other techniques. Thetemplate is laid over a portion of the image, and a weight is calculatedbased on the pixels under the template. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY21

Face DetectionHow does training work?A face of 24 24 pixels can have 45,396 possible combinations/scalesof the templates from the pervious slide.The purpose of training is to reduce the 45,396 possible combinationsdown to a minimum number and an ideal order.All Sub-WindowsFurther ProcessingT1F2TF3TFReject Sub-Window 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY22

Demo 3—Face Detection MinSize—The smallest face to detect.As a face gets further from the camera, itappears smaller. This parameter alsodefines the farthest a face can be fromthe camera and still get detected. MinN—The Minimum Neighborparameter groups faces that aredetected multiple times into onedetection. ScaleF—Scale Factor determines thenumber of times the face detector is runat each pixel location. The HaarCascade (xml file) that determines whatthe detector will detect, is designed foran object of only one size. In order todetect objects of various sizes (facesclose to the camera as well as far fromthe camera) the detector must be scaled. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY23

OPTICAL FLOW BASED TRACKING 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY24

Optical Flow Based TrackingOptical flow is the change in position of a group of pixels (feature) fromone image to the next. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY25

Optical FlowThe first step in optical flow is determining which features to use.The algorithm used in this example uses features like corners, edges,and points of brightness.These “good” features are found in each frame of a video stream.With this data, a tracking algorithm can then be applied to predictwhere the object will appear in the next frame. This is object tracking. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY26

Demo 4—Optical Flow Based Tracking MaxCount —The maximum number ofgood features to look for in a frame. qlevel —The quality of the features toaccept. A higher quality feature is morelikely to be unique, and be correctly foundin the next frame. A low quality featuremay get lost in the next frame, or worsebe confused with another point in theimage of the next frame. minDist —The minimum distancebetween features selected. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY27

Summary Computer vision represents the “software sensor” of the future. Computer vision trades unique hardware for software. In some instances, computer vision can be considered a “softwarescalable sensor”. As the available CPU horsepower increases, thecapabilities of the technology increase. This class used OpenCV to demonstrate just a few algorithmsavailable in the OpenCV library. OpenCV is a free computer vision library that has been downloadedover 3 million times. This presentation covered only 4 of the over 2,000 algorithmsavailable in OpenCV. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY28

To Probe FurtherVisit the Embedded Vision Alliance web siteat www.Embedded-Vision.comBDTI provides consulting services to companiesdeveloping and using vision technology: Technology selection Product development engineering services Competitive analysisVisit us at www.BDTI.com 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY29

RESOURCES 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY30

Selected Resources: The Embedded Vision AllianceThe VMware image used inthis presentation can bedownloaded y install the free VMwareplayer and download the BDTIOpenCV VMware image fromwww.embeddedvisionacademy.com/vmwareimage. 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY31

Selected Resources OpenCV: http://opencv.willowgarage.com/wiki/ Bradski and Kaehler, “Learning OpenCV: Computer Vision withthe OpenCV Library”, O’Reilly, 2008 Robert Laganière, “OpenCV 2 Computer Vision ApplicationProgramming Cookbook”, Packt, 2011 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY32

Selected ResourcesFree VMware player: http://www.vmware.com/products/player/Ubuntu 10.04 LTS: pse CDT: digosr1 2012 BDTIANALYSIS ADVICE ENGINEERINGFOR EMBEDDED PROCESSING TECHNOLOGY33