Exposing The Android Camera Stack
Exposing the Android Camera Stack Balwinder Kaur, Principal Software Architect Joe Rickson, Principal Software Engineer Android Builder’s Summit 2012 2.14.2012 2012 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners. 1 2012 Aptina Imaging Corporation
Agenda Hardware Independent Section Overview of android.hardware.Camera Prominent Camera Use Cases High Level Architecture JNI Layer Native Camera service Media Subsystem Interactions Hardware Dependent Modules Camera Hardware Abstraction Layer Camera Device Driver Camera Hardware Architecture Future Trends Q&A 2 2012 Aptina Imaging Corporation
Section I Hardware Independent Camera Stack 3 2012 Aptina Imaging Corporation
Overview of android.hardware.Camera 6 Classes Camera Camera.CameraInfo Camera.Parameters Camera.Size Camera.Face Camera.Area 4 2012 Aptina Imaging Corporation 7 Callback Interfaces Camera.AutoFocusCallback Camera.ErrorCallback Camera.FaceDetectionListener Camera.OnZoomChangeListener Camera.PictureCallback Camera.PreviewCallback Camera.ShutterCallback
Handling Camera Hardware Fragmentation Camera.Parameters class provides a “dumb” pipe to the hardware Hardware capabilities can be queried for capabilities. As an example, for Video Stabilization Feature isVideoStabilizationSupported() setVideoStabilization(boolean) getVideoStabilization() 5 2012 Aptina Imaging Corporation
Android 4.0 Camera Features Feature Platform Feature with API In-built Camera Proprietary Solution Application Code Face Detection 14 Face Recognition Panoramic Stitch 14 14 Video Snapshot 14 AE & AWB Lock 14 Continuous Focus Mode 14 Region Of Interest (AE, AWB and AF) Zero Shutter Lag* 14 Video Stabilization 15 Live Effects on Images / Video** 14 14 * There is no API for ZSL. It is a hardware dependent feature. ** android.media.Effect AE : Auto Exposure 6 API Level AWB : Auto White Balance 2012 Aptina Imaging Corporation AF : Auto Focus
Prominent Camera Use Cases Main Use Cases Live Preview of Camera Stream Live Preview copy of the Frame returned to the application Capture a frame Video Recording of a Camera Stream Secondary Use Cases Configuring the Camera Receiving more than an image back . e.g. face detection data Event Callbacks: Shutter Clicked, AutoFocus Achieved 7 2012 Aptina Imaging Corporation
High Level Architecture 8 2012 Aptina Imaging Corporation
Android High Level Architecture Source: Android Anatomy and Physiology, Google IO 2008 9 2012 Aptina Imaging Corporation
Hardware Abstraction Layer Camera Source: Android Anatomy and Physiology, Google IO 2008 10 2012 Aptina Imaging Corporation
Camera Subsystem Application Application framework Hardware Independent Camera Service Camera HAL Implementation Camera Device Driver Camera Hardware HAL Hardware Abstraction Layer 11 2012 Aptina Imaging Corporation Hardware Dependent
Process View App App App Binder IPC ICamera Camera Service Binder IPC Surfaceflinger ISurface Back Facing Camera Hardware Object System Call Kernel Driver 12 2012 Aptina Imaging Corporation Front Facing Camera Hardware Object Media server System Call Kernel Driver
Inside the Camera App Application Code android framework code Camera.java JNI android hardware C amera.cpp Camera Service libcamera.so Camera HAL implementation Media server 13 2012 Aptina Imaging Corporation
JNI Layer 14 2012 Aptina Imaging Corporation
JNI Layer Application Code android framework code Camera.java JNI android hardware C amera.cpp Camera Service libcamera.so Camera HAL implementation Media server 15 2012 Aptina Imaging Corporation
android hardware Camera Creates a persistent context for callbacks from native code to Java (JNICameraContext) Holds references to the Java Camera, Face and Area objects. If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here. Allocates Memory from the Java memory heap for JPEG images. 16 2012 Aptina Imaging Corporation
Camera Service 17 2012 Aptina Imaging Corporation
Camera Service Application Code android framework code Camera.java JNI android hardware C amera.cpp Camera Service libcamera.so Camera HAL implementation 18 2012 Aptina Imaging Corporation Media server
Camera Service Resource Manager for the Camera Hardware Asset Runs in the media server process It is a shared library libcameraservice.so Main Functions: 19 Permission check android.permission.CAMERA Ensures only one Client connects to a Camera Hardware Object Ensures each Process connects to a single Camera Hardware Object Redirects callbacks back to the app layer Accessed over IBinder Interface Number of Cameras Available CameraInfo Details 2012 Aptina Imaging Corporation
Camera Service (contd.) Android.mk file frameworks/base/media/mediaserver/Android.mk LOCAL SHARED LIBRARIES : \ libaudioflinger \ libcameraservice \ libmediaplayerservice \ libutils \ libbinder Gets instantiated as along with other components of the media server AudioFlinger::instantiate(); MediaPlayerService::instantiate(); CameraService::instantiate(); AudioPolicyService::instantiate(); 20 2012 Aptina Imaging Corporation
Interaction with the Media Subsystem ICameraRecordingProxy and ICameraRecordingProxyListener were introduced in Android 4.0 Allow apps to use the camera subsystem while the MediaRecorder is recording the video frames. ICameraRecordingProxy is a proxy of Icamera startRecording stopRecording releaseRecordingFrame ICameraRecordingProxyListener is an interface that allows the recorder to receive video frames during recording. dataCallbackTimestamp 21 2012 Aptina Imaging Corporation
Android Open Source Project (AOSP) Structure Android Framework Java: frameworks/base/core/java/android/hardware JNI: frameworks/base/core/jni Camera Service frameworks/base/services/camera/libcameraservice/ IBinder Interfaces frameworks/base/libs/camera/ICamera.h Camera HAL Interface frameworks/ base/services/camera/libcameraservice Camera HAL hardware/ vendor /camera (typically) 22 2012 Aptina Imaging Corporation
Section II Hardware-Dependent Camera Stack 23 2012 Aptina Imaging Corporation
Camera Hardware Abstraction Layer Review of a Typical Implementation 24 2012 Aptina Imaging Corporation
Camera Stack – Camera HAL Upper Camera Stack Camera Hardware Abstraction Layer (HAL) SurfaceFlinger /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Image Sensor 25 2012 Aptina Imaging Corporation Image Sensor Processer Kernel Hardware
Android CameraHAL Library The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform Written by hardware vendors (Qualcomm, TI, others) CameraHAL maps Android Camera Service calls to driver functions Android Froyo uses CameraHardwareInterface.h wrapper Ice Cream Sandwich (ICS) and above use camera.h CameraHAL low level interface communicates with the kernel level driver It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX) Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc) 26 2012 Aptina Imaging Corporation
Sample CameraHAL Functional Diagram Camera Service I/F Memory Manager Display Surface Manager CameraHAL Camera Manager Event Notification Manager User Source: TI OMAP4 git.omapzoom.org 27 2012 Aptina Imaging Corporation Camera Driver /dev/videoX Kernel
CameraHAL Block Diagram Discussion (1) Parts of the previous block diagram are hardware vendor specific May be different for each vendor and target platform CameraHAL Initialization – initialize the CameraHAL block and the target device driver Camera Services interface – Handle each Camera Service request, dispatch requests to the appropriate functional block Camera State machine – maintain the camera state through different API calls (e.g., preview, capture, recording, focus enable, etc). Memory Manager Cameras are memory intensive devices On request, allocate buffers for preview, capture and other functions Display Surface Manager Controls preview and video displaying - helps to coordinate with the camera manager block 28 2012 Aptina Imaging Corporation
CameraHAL Block Diagram Discussion (2) Display Surface Manager (cont) Communicates to the display when a frame is ready for preview Signals to the Camera Manager when the image buffer can be re-queued Event Notification Manager Supported callbacks include notify, data and timestamp Notify – call on camera error, shutter, focus, zoom events or raw image notify event Timestamp – call on video frame event Data – call on preview, postview, compressed image, and other capture events Call backs types are separated at the Camera Service level Camera Manager Handle camera activities Setting parameters Preview and snapshot callback 29 2012 Aptina Imaging Corporation
CameraHAL Preview Discussion The following slides discuss the preview use case Preview – displaying the camera image on the device display in real time The startPreview application call initiates image preview A single application level call results in a chain of CameraHAL and driver events Preview continues until the stopPreview() application call During preview, no application interaction unless a preview callback is registered 30 2012 Aptina Imaging Corporation
Preview Start Up Sequence Diagram (V4L2) Application Camera Server start preview Kernel Driver /dev/videoX CameraHAL start preview VIDIOC S FMT VIDIOC REQBUFS VIDIOC QUERYBUFS For each buffer For each buffer return 31 2012 Aptina Imaging Corporation MMAP VIDIOC QBUF VIDIOC STREAMON
Preview Operation Sequence Diagram (V4L2) Camera Server Application start preview Kernel Driver /dev/videoX CameraHAL start preview Preview image received signal VIDIOC DQBUF Send image to surface/Display Preview Notify VIDIOC QBUF If preview callback enabled, copy image and notify Camera Server 32 2012 Aptina Imaging Corporation
Camera Preview Interaction with the Display Subsystem Matching the timing of 2 events Preview frames arrive asynchronously from the camera The display subsystem refreshes the display at regular intervals Potential mismatch between these 2 system Sending the preview image to the display subsystem The preview frame is removed from the V4L2 queue of buffers The frame is sent to the display subsystem The frame memory is shared by the display subsystem Or the frame is copied to a buffer for display subsystem use The preview frame may be copied to a user space buffer if preview callback is enabled The frame is returned to the V4L2 queue of buffers when done 33 2012 Aptina Imaging Corporation
Camera Device Driver 34 2012 Aptina Imaging Corporation
Camera Stack – Camera Driver Upper Camera Stack Camera Hardware Abstraction Layer (HAL) SurfaceFlinger /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Image Sensor 35 2012 Aptina Imaging Corporation Image Sensor Processer Kernel Hardware
Android Kernel Camera Driver The kernel driver presents a standard interface for different types of camera hardware Camera hardware specific attributes are (usually) handled by the low level kernel driver Image Sensor Processor (ISP) vs. smart image sensor - differences are handled at the driver level For Android, Video for Linux 2 (V4L2) is used in many implementations V4L2 has been in existence for many years OpenMax (OMX) is also used for a low level driver interface by some vendors. 36 2012 Aptina Imaging Corporation
V4L2 Kernel Driver Block Diagram V4L2 Driver Interface IOCTL support/dispatch V4L2 driver infrastructure - - - 37 Controlling Interface Support for different device configurations Control device flow Buffer/Memory Management Memory allocation (if needed) Buffer management Buffer queue/de-queue 2012 Aptina Imaging Corporation - Camera HW Management One of these blocks for each camera type Device discovery Device initialization Power management Set/get device specific parameter Enable/disable image streaming
Android Linux Kernel Functionality Support for multiple camera types Camera specific code is localized to one file (the subdev device) Compile time option to add other cameras (one driver can support many different camera hardwares) More cameras mean a longer start up time since the driver is searching for each device The driver manages the underlying hardware topology (e.g., ISP sensor, smart sensor) For two or more cameras, the V4L2 driver creates additional device nodes Devices show up as /dev/video0 (primary), /dev/video1 (secondary), 38 2012 Aptina Imaging Corporation
V4L2 Kernel Driver Resources Memory Memory can be either driver-allocated or user-provided Moving image from the camera to memory should be done through hardware DMA (Direct Memory Access) Hardware memory management required to avoid contiguous memory requirement Interrupts Camera ports support for interrupts on events such as frame start, finish, focus events, etc. Camera Control: I2C/SPI Communications with the camera is usually done with I2C by either writing or reading sensor registers. I2C is somewhat slow, this limits the number of register accesses during a frame. SPI (Serial Peripheral Interface) is an alternative to I2C. Control Signals/Power/GPIO All controlled by the low level driver Power Sensor power management is critical to embedded device operation Save power by disabling the sensor and sensor processor when not used 39 2012 Aptina Imaging Corporation
V4L2 Driver Buffer Management One or more buffers are supported User buffers or kernel-allocated buffers are supported Buffers are treated the same for preview, capture, video (output resolution does not matter) Buffers are queued to a circular list Buffer filling starts when the V4L2 Stream On command is executed Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re-queues the buffer The Stream Off command causes all buffer to be released 40 2012 Aptina Imaging Corporation
Typical Android V4L2 Start up Sequence The V4L2 call for the preview are given below V4L2 Call Comments V4L2 Open() Open a V4L2 Device VIDIOC S FMT IOCTL: Set format Set both resolution and output pixel format VIDIOC G PARM IOCTL: Get camera parameter Get the camera frame rate VIDIOC S PARM IOCTL: Set camera parameter Set the camera frame rate VIDIOC CROPCAP IOCTL: Get the camera cropping capabilities Get the current crop rectangle VIDIOC S CROP IOCTL: Set the cropping rectangle Set the desired cropping rectangle VIDIOC REQBUFS IOCTL: Request camera buffers Request buffer support from the driver (user vs. kernel) Loop: VIDIOC QUERYBUF IOCTL: Return buffer address information Used for mapping buffers to user space Memory map buffers to user space Make buffers visible to user applications V4L2 MMAP 41 Title 2012 Aptina Imaging Corporation
Typical V4L2 Start up Sequence V4L2 driver start up sequence (cont) V4L2 Call Loop: VIDIOC QBUF Title IOCTL: Add buffer to queue Typical Android V4L driver shut-down sequence VIDIOC STREAM ON IOCTL: Start image streaming V4L2 Call 42 Title Comments Queue of buffers the kernel manages Camera/Driver starts filling the queued buffers Comments VIDIOC STREAM OFF IOCTL: Stop image streaming Stop camera/driver streaming V4L2 Close() Close the camera device Disable camera operations, free resources 2012 Aptina Imaging Corporation
V4L2 Driver Directions Other Topics V4L2 Media Controller Architecture Exposing the hardware image processor to the calling application Allows for greater programmer control Supported only on open source architectures Proprietary ISP software moves to user space Many ISP providers wish to hide their hardware Moving ISP code to user space handles this (avoid kernel open source issues) Driver source code location: {kernel sources}/drivers/media/video 43 2012 Aptina Imaging Corporation
Camera Hardware Overview 44 2012 Aptina Imaging Corporation
Camera Stack – Camera Hardware Upper Camera Stack Camera Hardware Abstraction Layer (HAL) SurfaceFlinger /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Image Sensor 45 2012 Aptina Imaging Corporation Image Sensor Processer Kernel Hardware
Camera Hardware Introduction Types of Sensor Hardware Raw or Bayer Sensor Outputs a Bayer (unprocessed) image Bayer Pattern Used with internal or external Image Sensor Processor (ISP) Internal ISP – System Processor and ISP bundled together External ISP – External companion chip Controls include exposure time and analog/digital gains Smart or System On a Chip (SOC) Sensor ISP built into the sensor YCrCb Pattern RGB565 Pattern Outputs processed YUV/RGB/Other formats Controls include exposure, white balance, gamma correction, and many others 46 2012 Aptina Imaging Corporation
Bayer Sensor Block Diagram Example - MT9M032 – 1.6MP Image Sensor 47 2012 Aptina Imaging Corporation
SOC Sensor Block Diagram Example - MT9M131 – 1.3 MP Image Sensor 48 2012 Aptina Imaging Corporation
Camera Hardware Inputs/Outputs Controlled by the sensor driver Inputs: Power/Ground (analog, digital power/grounds) Control Signals Reset – reset the camera to a default state Standby – place the camera in low power standby mode GPIO, others – control camera peripherals such as autofocus, flash, etc. Clock In (system clock in) Register control through I2C, SPI, or others Output Data Output Parallel (8, 10, 12, 14 bits) Serial (MIPI) Control Signals (frame/line valid) Clock Out (pixel clock out) 49 2012 Aptina Imaging Corporation
A Peek into the Future 50 2012 Aptina Imaging Corporation
Camera Application Trends Android Applications – memory limitation 16MB 24MB Higher pixel sizes and Bursty modes put a strain on the system Computer Vision Applications go mainstream APIs on Object Tracking, Gesture Recognition become more common place Computation Photography application Developers get fine grained control of flash and camera 51 2012 Aptina Imaging Corporation
Camera Hardware Trends Back Side Illumination(BSI) vs. Front Side Illumination(FSI) BSI can add up to 30% more light gathering capability Smaller Pixels Constant push to reduce pixel and sensor package sizes Faster data output rates, higher clock speeds 1080p30, 1080p60 Serial data interfaces enable increased sensor output speeds High Dynamic Range Ability to capture larger exposure range 3D Imaging Use of 2 cameras to generate a 3D image 52 2012 Aptina Imaging Corporation
Q&A 53 2012 Aptina Imaging Corporation
References http://developer.android.com/ http://www.codeaurora.org http://omappedia.org http://source.android.com 54 2012 Aptina Imaging Corporation
Communications with the camera is usually done with I2C by either writing or reading sensor registers. I2C is somewhat slow, this limits the number of register accesses during a frame. SPI (Serial Peripheral Interface) is an alternative to I2C. Control Signals/Power/GPIO All controlled by the low level driver Power
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
Chính Văn.- Còn đức Thế tôn thì tuệ giác cực kỳ trong sạch 8: hiện hành bất nhị 9, đạt đến vô tướng 10, đứng vào chỗ đứng của các đức Thế tôn 11, thể hiện tính bình đẳng của các Ngài, đến chỗ không còn chướng ngại 12, giáo pháp không thể khuynh đảo, tâm thức không bị cản trở, cái được
2-9V in unit & 2 AA in camera. Match polarities ( ) and ( ). Set camera date back, close camera lens and connect plug to camera port. 2 3 Secure camera, open camera shutter, and slide unit power switch to (ON) and back to (OFF), then push camera test button. Close camera Shutter, remove camera & load film, connect plug to camera, close cover. 4
Android Studio IDE Android SDK tool Latest Android API Platform - Android 6.0 (Marshmallow) Latest Android API emulator system image - Android 6.0 Android Studio is multi-platform Windows, MAC, Linux Advanced GUI preview panel See what your app looks like in different devices Development environment Android Studio 9
