OpenCV Introduction And New Features

OpenCV Introduction and NewFeaturesOpenCV China Team

Outline General overviewWhat’s new in OpenCV 4.xOpenCV DNN module: overview & new featuresHow to make OpenCV run fast1

OpenCV isThe most popular “traditional” computer vision library with growing DL capabilities:http://opencv.orgLicenseBSD (free for non-commercial and commercial use)Supported LanguagesC/C , Java, Python, JavascriptSize 1M lines of codeSourceForge statistics20M downloads (not include github traffic)Github statistics 10K clones per week, 9000 patches merged since 2012 ( 5 patches per day)Accelerated withSIMD: SSE, AVX2, AVX512, NEON, many core CPUs: parallel for, GPU: OpenCL, CUDA,Vulkan, S/W: IPP, MKL, Intel DLDTThe actual versions3.4.9, 4.2.0 (Dec. 2019)2

More than just OpenCVopencvThe main OpenCV repository, essential, stable modulesopencv contribExperimental or obsolete OpenCV functionalitycvat (Computer VisionAnnotation Tool)Tool for annotation of datasets; reworked version of VATICdldt (Deep LearningDeployment Toolkit)Very fast Deep Learning Inference Engine and ModelOptimizer/Converter tool; for Intel/AMD platforms only.open model zooHigh-quality CV deep learning models by Intelopencv training extensionsScripts for TensorFlow, PyTorch etc. to retrain some of themodels from open model zoo, quantize networks etc.http://github.com/opencv3

OpenCV History2007. OpenCV Symposium in Beijing, organized by Shiqi Yu, VadimPisarevsky and Shinn-Horng LEE.2006. OpenCV China web site2006. OpenCV 1.0 gold is finally out. 5 modules (core, cv, cvaux, ml,highgui), 200K lines of code, 500 functions & classes (ml), HTMLdocs, no IPL dependency, uses IPP for acceleration, Win & Linux, x86& x64 support, includes Python interface2000. OpenCV 1.0 alpha announced at CVPR. Win32 only; C API;includes image processing, contours, LK optical flow, uses IPL ascomplimentary library2000. Vadim Pisarevsky joined OpenCV Development Team as the team leader.1998. OpenCV project started at Intel under name CVL by Gary Bradski4

2015. OpenCV 3.0. 30 30 modules (with opencv contrib), 1M lines ofcode, T-API, NEON & AVX accelerations, free IPP, GSoC 2013 & 2014results2012. OpenCV 2.4.0-2.4.3. 22 modules, 600K lines of code,automatically generated Python & Java interfaces, CUDA & OpenCLacceleration, lot’s of parallel code, GSoC 2011&2012 results, Win,Linux, Android, iOS, x86, x64, ARM support. Migrated to githubOpenCV: Itseez!2009. OpenCV 2.0. Rewritten in C , uses CMake, lots of native SSE2accelerations. LBP face detection, HOG pedestrian detector, Farnebackdense optical flow, FAST corner detector, MSER region detector.5

2018. OpenCV 4.0. Major cleanup. Emphasis on DL. Halide integration.Intelintegration.2016. OpenCV 3.2. Deep Learning module. OpenVX support. GSoC2015 & 2016 results: state-of-art traditional optical flow, stereo, objectdetection, text detection, feature detection, tracking, calibration,comp. photography algorithms.OpenCV: Back to Intel!6

Plan to2025 OSVFOpenCVChinaTeams inRussiaOSVF, OpenCV China Foundation (to be founded) support OpenCVOpenCV Courses (online offline) in multiple languages,especially in ChineseBetter support for 3D sensors (lidar, ToF camera)Better support on ARM and specialized DL acclerators (NPU)OpenCV Distributed Team (?)2020 OpenCV 20-year anniversary and OpenCV 5 Release2019- OpenCV 4.x Release7

Developers and ContributorsUSA, Bay AreaOpen Source VisionFoundation (OSVF);OpenCV.orgRussia, Ni.NoChina, ShenzhenOpenCV.org.cn(since 2019 Dec)Community2-3 patches per dayvia github.comCore OpenCV team@ Intel Russia8

Major New Features of OpenCV 4.x C 11 library! Emphasis on Deep Learning (see further) Significantly extended and accelerated OpenCV DNNmoduleStarted replacing some traditional algorithms in OpenCVwith deep nets (e.g. face, object, text detection) Introduced graph API (G-API) for efficient imageprocessing pipelines Smaller and faster AVX2 & AVX512 acceleration; NEON acceleration for 32-bitand 64-bit ARM CPUs; 10-30% acceleration using AVX2!Lower footprint. OpenCV 4.0 is 20% smaller than OpenCV3.x.9

FP16 support (especially useful for efficient DeepLearning inference): cv::Mat fp16 tensor({32,32,16},CV FP16); Video I/O: Hardware-accelerated videodecoding/encoding on Windows (WMF) and Linux(GStreamer), new Android backend QR code detector and decoder Results from GSoC 2017 and GSoC 201910

Graph API (G-API) Overview A new separate module opencv gapi (not a complete libraryrewrite): https://github.com/opencv/opencv/wiki/Graph-API Provides alternative “lazy” image processing functions, e.g.cv::Sobel cv::gapi::Sobel‐ Instead of immediate evaluation gapi:: functions constructexpression subgraphs (GMat) and then you get a completegraph (GComputation) The produced graph is compiled (once) and then can beprocessed more efficiently than a sequence of direct functioncalls CPU and GPU backends are ready; more backends are inprogress11

12

G-API: Print imaging benchmarkWe can see the “Graph effect” : Memory consumption – process big images by tiles w/o storingintermediate results explicitly Cache efficiency better efficiency Code compactness – better performance with no need to write custom“fused” loops [to be added soon] automatic offloading to GPUMemory consumption*Performance* (based on cache efficiency)* – all measurements are taken on Intel Core -i5 6600 CPU, single thread13

DNN module overview Compact self-contained implementation in C ; inferenceonly! 5 importers (Caffe 1, TensorFlow, Torch, Darknet, ONNX) 40 layers, 100 unit tests, 20 samples Supports many popular topologies: image classification,object & text detection, semantic segmentation, instancesegmentation, pose estimation, face recognition, styletransfer, tracking, etc.14

Easy-to-use C , Python, Java and Javascript interface Sophisticated layer fusion mechanism & memory manager toimprove efficiency and decrease memory footprint Many different execution backends with graceful fallback tothe default C implementation:DNN BACKEND OPENCVDNN BACKEND INFERENCE ENGINEDNN BACKEND HALIDE (Deprecated)DNN BACKEND VKCOM (Vulkan) – –– – ?– ––– ––DNN BACKEND TENGINE (soon)15

Supported Topologies ClassificationCaffe: AlexNet, GoogLeNet, VGG, ResNet, SqueezeNet, DenseNet, ShuffleNetTensorFlow: Inception, MobileNetDarknet (https://pjreddie.com/darknet/imagenet/), ONNX (https://github.com/onnx/models) Object detectionCaffe: VGG-SSD, Mobilenet-SSD, Faster-RCNN, R-FCNTensorFlow: SSD, Faster-RCNN, Mask-RCNN (TF OD API), EAST text detectionYOLOv2, TinyYOLO, YOLOv3 (all Darknet), TinyYOLOv2 (ONNX) Semantic segmentationFCN (Caffe), ENet (Torch), ResNet101 DUC HDC (ONNX) OtherOpenPose body and hands pose estimation (Caffe), Colorization (Caffe), Fast-Neural-Style(Torch), OpenFace face recognition (Torch)Refer to g-in-OpenCV for details16

SSD-like network versus Haar CascadesHaar CascadeDLSize on disk528KB10MB (fp32), 5MB (fp16)Efficiency @ 300x300**30 ms9.34 msPerformanceAP @ IoU 0.5*0.609(FDDB) network versus Haar 0.797(FDDB)SSD-likeCascades0.149 (WIDER FACE, val.)0.173 (WIDER FACE, val.)*PASCAL VOC metric using COCO evaluation tool, http://cocodataset.org/#detections-eval**Intel Core i5-4460 CPU @ 3.20GHz x 417

18

New Features in OpenCV DNN (4.x) Vulkan-based backend (for Android)CUDA-based backend (GSoC ntel NCS and NCS2 support via Intel Inference EngineONNX importer added in 4.0, extended in 4.1.xMask-RCNN topology support mask rcnn.py sample3D CNNs support. New Action Recognition sample: action recognition.pyNew high-level API for detection, semantic segmentationAsynchronous inferenceDeep learning networks visualization: cv::dnn::dumpToFile(dot file);Improvements of ONNX and TensorFlow importers18% speedup of YOLOv3 on NCS2FPS53.2async face detectionperformance onIntel NCS248.639.526.419.9async01234dump of opencv face detect model(shows fused layers)19

Acceleration on Different PlatformsTools/libsApplicable forcv::parallel formany-core CPUsIntel & AMD x86/x64: SSE2-4, AVX2, AVX512wide universal intrinsicsCPUs with SIMD (vector) instructions.Dynamic dispatching is used sinceOpenCV 4.0ARM v7 and v8 (aarch64): NEONPPC64: VSXMIPS: MSA (PR submitted)OpenCL (OpenCV T-API)Intel iGPU, AMD GPU, Nvidia GPUCUDANVidia GPU (deprecated, except for DNN)VulkanDNN Inference on GPU (mostly for Android)IPP, MKL, OpenBLASCPU (traditional vision; image processing & linear algebra)Intel DLDTDNN Inference on Intel CPUs, GPUs, VPUsTengineIn progress: DNN Inference on ARM20

write once, run fast everywhere 400 intrinsics for each platform – lightweight, 5% overhead500 optimized SIMD loops 300 OpenCL kernels 25000 Unit testsCPUparallel for()OpenCVkernel 1kernel 2 kernel NGPU T-API(High-levelOpenCL-based API)UniversalIntrinsics APIOpenMPintrin sse.hppC 11 threadsUnix pthreadsMicrosoftConcurrencyApple GrandCentral DispatchIntel TBBintrin avx2.hpp,intrin avx512.hppARM MALI GPUintrin msa.hppMIPS P5600 (MSA)AMD GPUintrin riscv.hppintrin neon.hppARMv7, ARMv8(NEON)Intel Gen GPUPowerPC64 (VSX)Old Intel CPUs(SSE2 . SSE4)New Intel CPUs(AVX2, AVX512)Dynamically loaded& configuredintrin vsx.hppNVidia GPURISC-V ?Any GPU w.OpenCL support2121

Accelerating OpenCV on CPUs: cv::parallel for cv::parallel for – cross-platform implementation ofparallel loop concept (uses Win32 threads,std::threads, GDC, pthreads, OpenMP etc.underneath). cv::Mutex – cross-platform implementation of threadsynchronization objectYmax image height-win size 1y: 0Thr 0forparallel forThr 0y3y2y10Thr 1Thr 3y3Thr 2y4Thr 1y5Thr 0y6Thr 2YmaxThr 322

Accelerating OpenCV on CPUs: wide universalintrinsics23

The Future OpenCV on Edge – much better ARM support:‐ more optimizations of traditional vision algorithms‐ DNN inference optimization (probably, using Tengine by OpenAI) Support for specialized H/W DNN accelerators‐ extensive testing (Continuous Integration) of OpenCV on ARM Improved documentation, tutorials, courses (online & offline) Slow but steady refinement of traditional CV functionality (cameracalibration, 3D vision, image processing ) OpenCV 5 24

Thank You !25

3 opencv The main OpenCV repository, essential, stable modules opencv_contrib Experimental or obsolete OpenCV functionality cvat (Computer Vision Annotation Tool) Tool for annotation of datasets; reworked version of VATIC dldt (Deep Learning Deployment To