Real-time Computer Vision With OpenCV - NVIDIA

practicecomputer graphics problem. Whilegraphics transforms a scene or objectdescription to pixels, vision transformspixels to higher-level information.GPUs contain lots of similar processing units and are very efficient in executing simple, similar subtasks suchas rendering or filtering pixels. Suchtasks are often known as “embarrassingly parallel,” because they are so easyto parallelize efficiently on a GPU.Many tasks, however, do not parallelize easily, as they contain serial segments where the results of the laterstages depend on the results of earlierstages. These serial algorithms do notrun efficiently on GPUs and are mucheasier to program and often run fasteron CPUs. Many iterative numericaloptimization algorithms and stackbased tree-search algorithms belongto that class.Since many high-level tasks consistof both parallel and serial subtasks,the entire task can be accelerated byrunning some of its components onthe CPU and others on the GPU. Unfortunately, this introduces two sources of inefficiency. One is synchronization: when one subtask depends onthe results of another, the later stageneeds to wait until the previous stageis done. The other inefficiency is theoverhead of moving the data backand forth between the GPU and CPUmemories—and since computer-vision tasks need to process lots of pixels, it can mean moving massive datachunks back and forth. These are thekey challenges in accelerating computer-vision tasks on a system withboth a CPU and GPU.OpenCV LibraryThe open source computer vision library, OpenCV, began as a researchproject at Intel in 1998.5 It has beenavailable since 2000 under the BSDopen source license. OpenCV is aimedat providing the tools needed to solvecomputer-vision problems. It containsa mix of low-level image-processingfunctions and high-level algorithmssuch as face detection, pedestrian detection, feature matching, and tracking. The library has been downloadedmore than three million times.In 2010 a new module that provides GPU acceleration was added toOpenCV. The GPU module covers asignificant part of the library’s functionality and is still in active development. It is implemented using CUDAand therefore benefits from the CUDAecosystem, including libraries such asNVIDIA Performance Primitives (NPP).The GPU module allows users tobenefit from GPU acceleration withoutrequiring training in GPU programming. The module is consistent withthe CPU version of OpenCV, whichmakes adoption easy. There are differences, however, the most important ofwhich is the memory model. OpenCVimplements a container for imagescalled cv::Mat that exposes access toimage raw data. In the GPU module thecontainer cv::gpu::GpuMat storesthe image data in the GPU memory anddoes not provide direct access to thedata. If users want to modify the pixeldata in the main program running onthe GPU, they first need to copy thedata from GpuMat to Mat.#include opencv2/opencv.hpp #include opencv2/gpu/gpu.hpp using namespace cv;.Mat image imread("file.png");gpu::GpuMat image gpu;image gpu .upload(image);gpu::GpuMat result;gpu::threshold(image gpu,result, 128, CV THRESH , image);waitKey ();In this example, an image is readfrom a file and then uploaded to GPUmemory. The image is thresholdedthere, and the result is downloadedto CPU memory and displayed. In thissimple example only one operation isperformed on the image, but severalothers could be executed on the GPUwithout transferring images back andforth. The usage of the GPU module isstraightforward for someone who is already familiar with OpenCV.This design provides the user withexplicit control over how data is movedbetween CPU and GPU memory. Although the user must write some additional code to start using the GPU, thisapproach is flexible and allows more efficient computations. In general, it is agood idea to research, develop, and debug a computer-vision application us-ing the CPU part of OpenCV, and thenaccelerate it with the GPU module. Developers should try different combinations of CPU and GPU processing, measure their timing, and then choose thecombination that performs the best.Another piece of advice for developers is to use the asynchronous mechanisms provided by CUDA and the GPUmodule. This allows simultaneousexecution of data transfer, GPU processing, and CPU computations. Forexample, while one frame from thecamera is processed by the GPU, thenext frame is uploaded to it, minimizing data-transfer overheads and increasing overall performance.Performance ofOpenCV GPU ModuleOpenCV’s GPU module includes alarge number of functions, and manyof them have been implemented indifferent versions, such as the imagetypes (char, short, float), number ofchannels, and border extrapolationmodes. This makes it challenging toreport exact performance numbers. Anadded source of difficulty in distillingthe performance numbers down is theoverhead of synchronizing and transferring data. This means that best performance is obtained for large imageswhere a lot of processing can be donewhile the data resides on the GPU.To help the developer figure out thetrade-offs, OpenCV includes a performance benchmarking suite that runsGPU functions with different parameters and on different datasets. Thisprovides a detailed benchmark of howmuch different datasets are accelerated on the user’s hardware.Figure 2 is a benchmark demonstrating the advantage of theGPU module. The speedup is measured against the baseline of a heavily optimized CPU implementation ofOpenCV. OpenCV was compiled withIntel’s Streaming SIMD Extensionsn(SSE) and Threading Building Blocks(TBB) for multicore support, but notall algorithms use them. The primitiveimage-processing speedups have beenaveraged across roughly 30 functions.Speedups are also reported for severalhigh-level algorithms.It is quite normal for a GPU to show aspeedup of 30 times for low-level functions and up to 10 times for high-levelju n e 2 0 1 2 vo l. 55 n o. 6 c ommu n icat ion s of t he acm63

practicefunctions, which include more overhead and many steps that are not easyto parallelize with a GPU. For example,the granularity for color conversion isper-pixel, making it easy to parallelize. Pedestrian detection, on the otherhand, is performed in parallel for eachpossible pedestrian location, and parallelizing the processing of each window position is limited by the amountof on-chip GPU memory.As an example, we accelerated twopackages from Robot Operation System (ROS)8—stereo visual odometryand textured object detection—thatwere originally developed for the CPU.They contain many functional blocksand a class hierarchy.Wherever it made sense, we offloaded the computations to the GPU. Forexample, OpenCV GPU implementations performed Speeded-Up RobustFeature (SURF) key point detection,matching, and search of stereo correspondences (block matching) for stereo visual odometry. The acceleratedpackages were a mix of CPU/GPU implementations. As a result, the visualodometry pipeline was accelerated 2.7times, and textured object detectionwas accelerated from 1.5–4 times, asillustrated in Figure 3. Data-transferFigure 3. Textured object detection application: CPU and GPU.Figure 4. Stereo block matching pipeline.CPUGPUSpeckle filteringRectificationMatchingLow texture filteringFigure 5. RGB frame, depth frame, ray-casted frame, and point cloud.64communicatio ns o f th e acm j u ne 201 2 vo l . 5 5 no. 6Color and Showoverhead was not a significant part ofthe total algorithm time. This exampleshows that replacing only a few lines ofcode results in a considerable speedupof a high-level vision application.Stereo Correspondencewith GPU ModuleStereo correspondence search in ahigh-resolution video is a demandingapplication that demonstrates howCPU and GPU computations can beoverlapped. OpenCV’s GPU moduleincludes an implementation that canprocess full HD resolution stereo pairin real time (24 frames per second) onthe NVIDIA GTX580.In a stereo system, two cameras aremounted facing in the same direction.While faraway objects project to thesame image locations on each camera, nearby objects project to differentlocations. This is called disparity. Bylocating each pixel on the left cameraimage where the same surface pointprojects to the right image, you cancompute the distance to that surfacepoint from the disparity. Finding thesecorrespondences between pixels inthe stereo image pairs is the key challenge in stereo vision.This task is made easier by rectifyingthe images. Rectification warps the images to an ideal stereo pair where eachscene surface point projects to a matching image row. This way, only points onthe same scan line need to be searched.The quality of the match is evaluated bycomparing the similarity of a small window of pixels with the candidate-matching pixel. Then the pixel in the rightimage whose window best matches thewindow of the pixel on the left image isselected as the corresponding match.The computational requirementsobviously increase as the image sizeincreases, because there are more pixels to process. In a larger image therange of disparities measured in pixelsalso increases, which requires a largersearch radius. For small-resolution images the CPU may be sufficient to calculate the disparities; with full HD resolution images, however, only the GPUcan provide enough processing power.Figure 4 presents a block-matchingpipeline that produces a disparity image d(x,y) such that LeftImage(x,y)corresponds to RightImage(x-d(x,y),y).The pipeline first rectifies the images

practiceand then finds the best matches, aspreviously described. In areas wherethere is little texture—for example, ablank wall—the calculated matchesare unreliable, so all such areas aremarked to be ignored in later processing. As the disparity values are expected to change significantly near objectborders, the speckle-filtering stageeliminates speckle noise within largecontinuous regions of disparity image.Unfortunately, the speckle-filtering algorithm requires a stack-based depthfirst search difficult to parallelize, so itis run on the CPU. The results are visualized using a false-color image.All the steps except speckle filtering are implemented on the GPU. Themost compute-intensive step is blockmatching. NVIDIA GTX580 has accelerated it seven times faster than a CPUimplementation on a quad core Inteli5-760 2.8GHz processor with SSE andTBB optimizations. After this speedupthe speckle filtering becomes the bottleneck, consuming 50% of the frameprocessing time.An elegant parallel-processing solution is to run speckle filtering on theCPU in parallel with the GPU processing. While the GPU processes the nextframe, the CPU performs speckle filtering for the current frame. This canbe done using asynchronous OpenCVGPU and CUDA capabilities. The heterogeneous CPU/GPU system nowprovides a sevenfold speedup for thehigh-resolution stereo correspondenceproblem, allowing real-time (24fps)performance at full HD resolution.KinectFusionMicrosoft’s KinectFusion4 is an example of an application that previouslyrequired slow batch processing butnow, when powered by GPUs, can berun at interactive speeds. Kinect is acamera that produces color and depthimages. Just by aiming the Kinect device around, one can digitize the 3Dgeometry of indoor scenes at an amazing fidelity, as illustrated in Figure 5.An open source implementation ofsuch a scanning application is basedon the Point Cloud Library,6 a companion library to OpenCV that uses 3Dpoints and voxels instead of 2D pixelsas basic primitives.Implementing KinectFusion is nota simple task. Kinect does not returnrange measurements for all the pixels, and it works reliably only on continuous smooth matte surfaces. Therange measurements that it returnsare noisy, and depending on the surface shapes and reflectance properties,the noise can be significant. The noisealso increases with the distance to themeasured surface. Kinect generates anew depth frame 30 times in a second.If the user moves the Kinect devicetoo fast, the algorithm gets confusedand cannot track the motion using therange data. With a clever combinationof good algorithms and using the processing power provided by GPUs, however, KinectFusion works robustly.There are three key concepts thatmake a robust interactive implementation feasible. First, the tracking algorithm is able to process the new scandata so fast that the camera has time tomove very little between the consecutiveframes. This makes it feasible to trackthe camera position and orientation using just the range data.Second, fusion of depth data is doneusing a volumetric surface representation. The representation is a large voxelgrid that makes it easier to merge thedata from different scans in comparison with surface-based representations.To obtain high model quality, the gridresolution is chosen to be as dense aspossible (512 512 512), so it has to beprocessed by the GPU for real-time rates.Finally, the manner in which the newdata is merged with the old reduces thenoise and uncertainty as more data isgathered, and the accuracy of the model keeps improving. As the model getsbetter, tracking gets easier as well. Parallel ray casting through the volume isdone on the GPU to get depth information, which is used for camera trackingon the next frame. So frame-to-framemovement estimation is performedonly between the first and secondframes. All other movements are computed on model-to-frame data, whichmakes camera tracking very robust.All of these steps are computationally intensive. Volumetric integrationrequires the high memory bandwidththat only the GPU can deliver at a pricelow enough to be affordable by normalconsumers. Without GPUs this systemwould simply not be feasible. However, not every step of the computationis easy to do on a GPU. For example,tracking the camera position is doneon a CPU. Though the linear equationmatrix required for camera position estimation is fully computed on the GPU,computing the final solution does notparallelize well, so it is done on theCPU, which results in some downloadand API call overhead. Another problem is that the bottom-level image inthe hierarchical image processing approach is only 160 120, which is notlarge enough to fully load a GPU. Allthe other parts are ideal for GPU butlimited by the amount of available GPUmemory and computing resources.Further development requires evenmore GPU power. At the moment, thesize of the scene is limited by the volumetric representation. Using the samenumber of voxels but making them bigger would allow us to capture a largerscene but at a coarser resolution. Retaining the same resolution while scanning larger scenes would require morevoxels, but the number of voxels is limited by the amount of memory availableon GPU and by its computational power.Mobile DevicesWhile PCs are often built with a CPUand a GPU on separate chips, mobiledevices such as smartphones and tablets put all the computing elementson a single chip. Such an SoC (systemon chip) contains one or more CPUs, aGPU, as well as several signal processorsfor audio and video processing and datacommunication. All modern smartphones and some tablets also containone or more cameras, and OpenCV isavailable on both Android and iOS operating systems. With all these components, it is possible to create mobilevision applications. The following sections look at the mobile hardware inmore detail, using NVIDIA’s Tegra 2 andTegra 3 SoCs as examples, and then introduce several useful multimedia APIs.Finally, two mobile vision applicationsare presented: panorama creation andvideo stabilization.Tools for Mobile Computer VisionAt the core of any general-purposecomputer is the CPU. While Intel’sx86 instruction set rules on desktopcomputers, almost all mobile phonesand tablets are powered by CPUs fromARM. ARM processors follow the RISC(reduced instruction set computing)ju n e 2 0 1 2 vo l. 55 n o. 6 c ommu n icat ion s of t he acm65

practiceEnergy savings with GLSL on Tegra 3.OpenCV Function(10,000 iterations)median blurEnergySavings3.43planar warper6.25warpPerspective6.45cylindrical warper3.89blur3x33.60warpAffine15.38approach, as can be deduced fromARM’s original name, Advanced RiscMachines. While x86 processors weretraditionally designed for high computing power, ARM processors weredesigned primarily for low-power usage, which is a clear benefit for batterypowered devices. As Intel is reducingpower usage in its Atom family for mobile devices, and recent ARM designsare getting increasingly powerful, theymay in the future reach a similar design point, at least on the high end ofmobile computing devices. Both Tegra2 and Tegra 3 use ARM Cortex-A9 CPUs.Mobile phones used to have onlyone CPU, but modern mobile SoCs arebeginning to sport several, providingsymmetric multiprocessing. The reason is the potential for energy savings.One can reach roughly a similar levelof performance using two cores running at 1GHz each than with one corerunning at 2GHz. Since the power consumption increases super-linearly withthe clock speed, however, these twoslower cores together consume lesspower than the single faster core. Tegra2 provides two ARM cores, while Tegra3 provides four. Tegra 3 actually contains five (four plus one) cores, out ofwhich one, two, three, or four cores canbe active at the same time. One of thecores, known as the shadow or companion core, is designed to use particularly little energy but can run onlyat relatively slow speeds. That mode issufficient for standby, listening to music, voice calls, and other applicationsthat rely on dedicated hardware suchas the audio codec and require only afew CPU cycles. When more processingpower is needed (for example, reading email), the slower core is replacedby one of the faster cores, and for increased performance (browsing, gaming) additional cores kick in.66communicatio ns o f th e acmSIMD (single instruction, multipledata) processing is particularly usefulfor pixel data, as the same instruction can be used on multiple pixelssimultaneously. SSE is Intel’s SIMDtechnology, which exists on all modern x86 chips. ARM has a similar technology called NEON, which is an optional coprocessor in the Cortex A9.The NEON can process up to eight,and sometimes even 16 pixels at thesame time, while the CPU can process only one element at a time. Thisis very attractive for computer-visiondevelopers, as it is often easy to obtain three to four times performancespeedup—and with careful optimization even more than six times. Tegra 2did not include the NEON extension,but each of Tegra 3’s ARM cores has aNEON coprocessor.All modern smart phones includea GPU. The first generation of mobileGPUs implemented the fixed-functionality graphics pipeline of OpenGL ES1.0 and 1.1. Even though the GPUs weredesigned for 3D graphics, they couldbe used for a limited class of imageprocessing operations such as warping and blending. The current mobileGPUs are much more flexible and support OpenGL shading language (GLSL)programming with the OpenGL ES 2.0API, allowing programmers to run fairly complicated shaders at each pixel.Thus, many old-school GPGPU tricksdeveloped for desktop GPUs about 10years ago can now be reused on mobiledevices. The more flexible GPU computing languages such as CUDA andOpenCL will replace those tricks in thecoming years but are not available yet.Consumption and creation of audioand video content is an important usecase on modern mobile devices. To support them, smartphones contain dedicated hardware encoders and decodersboth for audio and video. Additionally,many devices have a special ISP (imagesignal processor) that processes thepixels streaming out from the camera.These media accelerators are not as easily accessible and useful for computervision processing, but the OpenMAXstandard helps.1 OpenMAX definesthree different layers: AL (application),IL (integration), and DL (development).The lowest, DL, specifies a set of primitive functions from five domains: audio/video/image coding and image/ j u ne 201 2 vo l . 5 5 no. 6signal processing. Some of them are ofpotential interest for computer-visiondevelopers, especially video coding andimage processing, because they providea number of simple filters, color spaceconversions, and arithmetic operations. IL is meant for system programmers for implementing the multimediaframework and provides tools such asfor camera control. AL is meant for application developers and provides highlevel abstractions and objects suchas Camera, Media Player, and MediaRecorder. The OpenMAX APIs are useful for passing image data efficientlybetween the various accelerators andother APIs such as OpenGL ES.Sensors provide another interestingopportunity for computer-vision developers. Many devices contain sensorssuch as an accelerometers, gyroscopes,compasses, and GPSs. They are notable to perform calculations but canbe useful if the application needs toreconstruct the camera orientation or3D trajectory. The problem of extracting the camera motion from a set offrames is challenging, both in terms ofperformance and accuracy. Simultaneous localization and mapping (SLAM),structure from motion (SfM), and other approaches can compute both thecamera position and even the shapesof the objects the camera sees, butthese methods are not easy to implement, calibrate, and optimize, and theyrequire a lot of processing power. Thesensors can nonetheless deliver a fairlyaccurate estimate of the device orientation at a fraction of the cost of relyingonly on visual processing. For accurateresults the sensor input should be usedonly as a starting point, to be refinedusing computer-vision techniques.OpenCV On TegraA major design and implementationgoal for OpenCV has always been highperformance. Porting both OpenCVand applications to mobile devicesrequires care, however, to retain a sufficient level of performance. OpenCVhas been available on Android sincethe Google Summer of Code 2010 whenit was first built and run on GoogleNexus One. Several demo applicationsillustrated almost real-time behavior,but it was obvious that OpenCV needed optimization and fine-tuning formobile hardware.

practicebined. If the algorithm is constrainedby the speed of memory access, however, multithreading may not providethe expected performance improvement. For example, the NEON versionof cv::resize does not gain fromadding new threads, because a singlethread already fully consumes thememory-bus capacity.The final method applied duringthe optimization of the OpenCV library for the Tegra platform is GPGPUwith GLSL shaders. Though the mobile GPU has limitations as discussedpreviously, on certain classes of algorithms the GPU is able to show animpressive performance boost whileconsuming very little energy. On mobile SoCs it is possible to share memory between CPU and GPU, whichallows interleaving C and GLSLprocessing of the same image buffer.Figure 7 shows example speedups ofsome filters and geometric transformations from the OpenCV library.An additional benefit of using theGPU is that at full speed it runs at alower average power than the CPU.On mobile devices this is especiallyimportant, since one of the main usability factors for consumers is howlong the battery lasts on a charge. Wemeasured the average power and timeelapsed to perform 10,000 iterationsof some optimized C functions,compared with the same functionswritten in GLSL. Since these functions are both faster on the GPU, andthe GPU runs at lower peak power. Wemeasured the result is significant energy savings (see the accompanyingtable). We measured energy savings of3–15 times when porting these functions to GPU.Figure 6. Performance improvement with NEON on Tegra 3.Tegra CPUTegra NEON3002501.6xTime Down2.6xSobelMorphologyC

openCV Library The open source computer vision li-brary, OpenCV, began as a research project at Intel in 1998.5 It has been available since 2000 under the BSD open source license. OpenCV is aimed at providing the tools needed to solve computer-vision problems. It contains a mix of low-level image-processing functions and high-level algorithmsFile Size: 2MB