CMOS Active Pixel Image Sensors: Past, Present, And Future
CMOS Active Pixel Image Sensors:Past, Present, and FutureDr. Eric R. FossumJanuary 2008 2008 E R Fossum
2008 E R Fossum
2008 E R Fossum12/31/2007 D. Snyder – night handheld no flash
Early History of Image Sensors 1963 Morrison - Honeywell – Light spot position “computational”sensor – “Passive pixel” photodiode array(“reticon”)– 100 x 100 element array1964 Horton, et al. - IBM– The “scanistor” 1967 Weckler - Fairchild– Charge integration on a floating pnjunction 1967 Weimer, et al. - RCA– 180x180 TFT element self-scannedsensor– Battery powered, wireless camera 2008 E R Fossum1968 Noble - Plessey–––––1966 Schuster & Strull Westinghouse– 50 x 50 element phototransistor array 1968 Dyck & Weckler - Fairchild Passive pixel photodiode arrayOn-chip charge integrating amplifierBuried photodiode structureSource-follower buffer in pixelLots of problems – Vt instability, FPN1970 Boyle, Smith, Amelio, TompsettAT&T Bell Labs– Charge-coupled semiconductordevices (CCDs)
CCD Operation Charge-coupled devices shift charge onestep at a time to a common output amplifierV1V220% fill factorboosted to 60%with microlensV3V1Parallel vertical shift registersPDHorizontal shift register 2008 E R FossumAmp
Charge-Coupled Devices CCDs were better– Smaller pixel sizes (3electrodes/pixel)– Lower readout noise– No fixed pattern noise– Low on-chip power dissipation– Interesting device physics CCDs have limitations– Requires high charge transferefficiency Special fabrication process Large voltage swings,different voltage levels Radiation “soft” in spaceenvironment Limits readout speed– Difficult to integrate on-chiptiming, control, drive andsignal chain electronics– Serial access to image data– System power in 1-10 Wattrange 2008 E R Fossum
anceOperabilityCMOS PPSEarlyMOSSporadic activityCost & FunctionalityCMOS APSPerformance,Cost & Functionality 2008 E R Fossum
111 Mpix Charge-Coupled DeviceR. Bredthauer et al.2007 IISW 2008 E R Fossum
Passive Pixel MOS Image Sensor Like DRAM Single switch for resetand readout Charge read out at arraycorner or amplified atbottom of column. Approach taken initiallyby Hitachi, VVL (ST),Omnivision and others Too much noise and FPNto compete with CCD Allows integration of othercircuits on same chip 2008 E R FossumTXCOL BUS
Active Pixel Image Sensors Amplifier inside pixel––––Olympus Charge Modulated Device (CMD)TI Bulk Charge Modulated Device (BCMD)Canon Base-Stored Image Sensor (BASIS)Olympus SIT Image Sensor All “tricky” devices to make and require highlyspecialized fab process.– Hard to make, hard to manufacture– Difficult to evolve to smaller process features– No compelling reason for industry to switch 2008 E R Fossum
Active Pixel Image Sensors NHK Amplified MOS Image Sensor– 3T PD type– Almost got it right but readout chain sensitiveto MOSFET “on resistance” and hence largeFPN 2008 E R Fossum
Driving Forces CCD development in the 1980’s driven bycamcorder market– Solid-state much better than tubes forconsumer CMOS image sensor development in thelater 1990’s, 2000’s driven by cameraphone market– Lower power, higher integration, small formfactor, lower cost for same functions. 2008 E R Fossum
CMOS Active Pixel Image Sensor Invented at NASA/JPL 1992 (patents owned by Caltech) Used vanilla CMOS process available at many foundries Single-stage “CCD” in each pixel to allow completecharge transfer In-pixel source-follower amplifier for charge gain Allows low noise CDS operation In-column FPN reduction Permitted high performance camera-on-a-chip Basis of all modern CMOS image sensors 2008 E R Fossum
Advantages ofCMOS Image Sensors CMOS Camera-on-a-chip technology is better thanCCDs because:–––––––––Much lower power - important for portable applicationsSystem-on-a-chip integration allows smaller camerasLower cost of sensor chip and fewer components in cameraEasy digital interface for faster camera design & time to marketLess image artifacts - no blooming or smear, with samesensitivityHigher dynamic range for security and auto applicationsDigital output for faster readout speeds and frame ratesDirect addressing of pixels allows electronic pan/tilt/zoomFaster design cycles means faster evolution path 2008 E R Fossum
CMOS APS OPERATION (1)VDDPGTXRSCRCOL BUS1. Signal is integratedunder PGCSRS 2008 E R Fossum
CMOS APS OPERATION (2)VDDPGTXRSCRCOL BUS2. Pixel is selected andfloating diffusion is reset.After reset, FD voltagesampled onto capacitorCR at bottom of columnbus. (All pixels in samerow processedsimultaneously)CSRS 2008 E R Fossum
CMOS APS OPERATION (3)VDDPGTXCOL BUSRSCR3. Charge under PG istransferred to floatingdiffusion by pulsing PG.(All pixels in same rowprocessedsimultaneously)CSRS 2008 E R Fossum
CMOS APS OPERATION (4)VDDPGTXRSCRCOL BUS4. New voltage on floatingdiffusion sampled ontocapacitor CS. Correlatedsample exists on CR andCS so kTC, 1/f and Vtvariations suppressed.CSRS 2008 E R Fossum
CMOS APS OPERATION (5)VDDPGTXRSCRCOL BUS5. After row has beenprocessed, pixel datais selected and drivenonto differentialhorizontal bus using2nd stage buffers.Data driven off-chipusing 3rd stagebuffers.CSRS 2008 E R Fossum
Pinned PD Photogate Better to replace polyMOSFET photogatewith JFET photogate(a.k.a. pinned photodiode or buriedphotodiode) First demonstrated byJPL/Kodakcollaboration in 1995Lee, Gee, Guidash, Lee and Fossum1995 IEEE CCD AIS Workshop 2008 E R Fossum
2D Potential Simulation 2008 E R Fossum
Color Filter Arraysand MicrolensesMicrolens layerColor filter layerMetal opaque layerPhotodiodeSilicon substrate 2008 E R Fossum
Camera-on-a-chip Pixel arraySignal chainADCDigital logic––––I/O interfaceTiming and controlExposure controlColor processing Ancillary circuits 2008 E R FossumPain et al.2007 IISW
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)ResetAgain 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLResetAgainReadout(& Reset) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLResetAgainReadout(& Reset) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLResetAgainReadout(& Reset) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLResetAgainReadout(& Reset) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLResetAgainReadout(& Reset) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLResetAgainReadout(& Reset) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLReadout(& Reset)IntegrationResetAgain Sometimes it is a problem that not all pixels integratefor the same period of time - moving object distortion 2008 E R FossumTime
PICTURES FROM PC-TYPE CAMERAWITH ELECTRONIC ROLLING SHUTTERCIF ResolutionVGA Resolution(352 x 288)(640 x 480) 2008 E R Fossum
ROLLING SHUTTEREXPOSURE CONTROLIllustration of distortionfor moving objects Note that closer objects (objects passing faster in field of view)are more “tilted” compared to more distant objects. Not important for most tethered PC applications 2008 E R Fossum
First JPL CMOS APS Chip 4/93 JPL: Mendis & Fossum 2008 E R Fossum28 x 28 element array2 μm CMOS40 μm x 40 μm pixelsNo on-chip timing or control
JPL Multiresolution ny, Panicacci, Pain, Matthies, Fossum 1995 2008 E R FossumHP 1.2 umn-well CMOSPixel pitch: 24 umNo. pixels: 128 x 128Pwr suppy: 5 voltsSaturation: 1200 mVConv. gain: 8 uV/eNoise:116 uV rms15 e- rmsDynamic80 dBRange:FPN: 3 mV p-p 2.5 %Power: 5 mW at 30HzFull resolution image4x4 Averaged image (left)1/4 Subsampled image (right)
JPL/AT&T Digital CMOS APS 176x144 elements20 μm pixel pitchSingle-slope ADC per column176 ADCs per chip8 bit resolution35 mW at 30 Hz3.5 volt supply 2008 E R FossumMendis, Inglis, Dickinson, and Fossum 1995
PB PrometheusPanicacci 1996 2008 E R Fossum 576 (H) x 432 (V)20.9 μm PD pixel pitch1.4 μm process576 single slope ADCsdigital & analog outputports
PB Very Low light sensorlow light image ( 0.004 lux) 176 (H) x 144 (V) elements 30 μm E-PG pixel pitch 1.2 μm process 20 fps analog outputXue 1996 2008 E R Fossum
PB 300 for Multimedia 640x487 (640x480 effective) pixels320 column-parallel 8b ADCs7.9 μm x 7.9 μm (1/3” optical format)Progressive scan, window readout15,000 gate on-chip logic 30 frames per sec (adj.)5 V operation 300 mW total powerBuilt-in autoexposure controlOn-chip biasingOn-chip CFA 20 dB SNR @ 1 lux, 30 fpsPanicacci, Cho 1999 2008 E R Fossum
PBL080 Low Light Sensor 342 x 258 pixels336 x 242 optical pixels20 x 20 μm pixel pitch64x on-chip gainOn-chip gamma correction8b digital output10,000 gates on-chip logicEIA/NTSC timing60 Hz progressive scanAutoexposure control--low light to sunlight @ F/1.4150 mWBarna, Ang, O’Conner, Wang 1999 2008 E R Fossum200 lux room light, 60 Hz, F/1.41000x less light: 200 millilux, 60 Hz, F/1.4
Ultra High Dynamic (HiDy) RangeImaging SystemDualSensitivityPixels(Linear Response)ASPADCsFUSION/DRCPROCESSORASICHigh Sensitivity/High Gain 8-bitHigh Sensitivity/Low Gain 8-bitLow Sensitivity/High Gain 8-bitLow Sensitivity/Low Gain 8-bitFused Output 8-bit Linear outputs preserve contrast unlike logarithmicsensors Low susceptibility to FPN compared to logarithmicsensors Higherframe rate and less motion blur with2008 E R Fossumconcurrent capturingWang 2000
Pill-Camera Pixel Format:Pixel Size:Frame Rate:ADC:Power Supply:Power: 2008 E R Fossum256 X 25610 µm X 10 µm2 fpsOn-Chip, 8 bits2.8 V3 mWKrymski
PB Dental Xray- Sensorx-ray Total chip size: 37 x 27 mm900 (V) x 675 (H) elements40 μm pixel pitch2 μm CMOS processDifferential analog outputOn-chip timing and controlEvent detector self-triggeredreadout World’s largest CMOS chip inproductionNixon1996 2008 E R Fossumvisibleflash
PB Buttable X-ray Sensor Xue 1997466 x 466 elements30 μm pixel pitch0.8 μm process2 column loss 2008 E R Fossum 3-sides buttableP-channel PD-type pixelDifferential analog outputSome on-chip circuits
PB Optical Memory ReadoutPanicacci 1997 2008 E R Fossum 816 (V) x 616 (H) elements 17 μm PD pixel pitch 0.8 μm process special column-parallel comparator circuit
PB1024 High Speed Sensor 1024x1024 elements10 μm x 10 μm pixel pitch0.5 μm CMOS1024 on-chip 8b ADCs8 digital output ports (64 pins)Open architecturePower: 95 mW @ 60 fpsPower: 370 mW @ 574 fpsBy far, then world’s pixel raterecord of ALL image sensors(CCDs and CMOS)Krymski, Van Blerkom, Bock, Anderson 1998 2008 E R Fossum
PBMV40 2352x1728 (4.1Mpix)200fps ERS 4.1 Mpixel sensor 7 μm x 7 μm pixel pitch 16x10b digital output 200 fps ERS 960 Mbytes/sec at 66 MHz 4000 bits/lx-sec 3.3 volt operationKrymski 2001 2008 E R Fossum
Shutter EfficiencyShutter closedShutter open1 frame period (Tperiod)TopentimeIdeal: Signal Topen / Tperiod Signal0Actual: Signal Topen / Tperiod Signal0 (1-ε) Signal0where ε is the shutter efficiency 2008 E R Fossum
CMOS SNAP Proprietary Photobittechnology, 5-T pixelNormal photodiodeSignal transferred tostorage node in NwellStorage node readout in usual wayDiffusingphotoelectrons donot affect storageVery high shutterefficiency 99.9% 2008 E R FossumVddColVRn RPTXN-wellRGVRRS
PB MV13 1280x1024 (1.3Mpix)SNAP 1000 fps 1.3 Mpixel sensor12 μm x 12 μm pixel pitch0.35 μm CMOSHigh efficiency ( 99.9%)freeze frame shutter Shutter speeds from 1/30th to1/100,000th sec 10x10b digital output (100pins) Open architecture 3.3 volt operation Power: 500 mW @500 fps 1000 bits/lx-sTu,Blerkom, Barna, Fossum 2001 2008E RVanFossum Krymsi,ColorormonochromeColor image withrotating fan1/605 sec 1652 usecRolling shutter1/33,000 sec 30 usecFreeze frame shutter
High Speed Linear Sensorfor Inspection Strand, Iverson2001 2008E R Fossum4096 x 1 linear array7 um pixel pitch (28 mm long)Curtain-style readout4 analog output ports, each 60 MHz240 Mpix/sec total output rate35,000 lines/sec5 V operationSubwindowing allowed, from center
CMOS Image Sensors in 2008 Basic operation still the same. Sharing of active transistor has improvedfill factor ADC performance improved On-chip color processing improved Pixel pitch shrunk with smaller designrules 2008 E R Fossum
3.2 um 2T pixel 0.18 um process8832 x 5748 52 Mpix @ 160 Mp/s 2008 E R FossumIwane et al.2007 IISW
UDTV or Super Hi-Vision Sensor 2008 E R Fossum7680 x 432033 Mpixels60 fps2 Gp/s
To The Near Future Main application of CMOS image sensors will becamera phonesArea Image Sensor MarketCamera Phone 2003-2011 0CCD600,000CMOS400,000200,0002003 2008 E R Fossum200420052006F2007F2008F2009F2010F2011F
Most of MarketControlled by 7 PlayersMarket Share CCD and CMOS Sensors 20072%1%1%Micron0%Omnivision2%5%SamsungST Micro5%32%12%ToshibaMagna ChipSonySharpAVAGO (Agilent)9%Sharp (pixelplus)17%14%PanasonicOthers 2008 E R Fossum
Usual technology drivers willcontinue for next few years Smaller pixels (sub-diffraction limit or SDL)– Sensitivity– Full well– SNR Larger array sizes (up to 8-10 Mpixels)– Smaller pixels– Improved optics SoC will continue to demand premium– Despite trend to commoditize sensors thru 2-chipsolutions.– Off-chip system integration too much work for cameraphone OEMs. 2008 E R Fossum
Existing Solutions will beTried and Tried Again Brute force shrink of pixel– Path of least resistance Stacked structures– Lag, noise and stability need to be overcome Backside illumination– Process flow development for high volumemanufacturing Improved on-chip optics– Multi layer optics, optical funnels Improved dynamic range– Numerous adequate sensor solutions exist 2008 E R Fossum
Longer Term R&D Thoughts Stacked structures look interesting forcapacitance improvement and opticalimprovement. Organic polymers for wavelengthselectivity and lower dark current Jot/Digital Film paradigm shift Etc. 2008 E R Fossum
Gigapixel Digital Film Sensor(DFS) 2008 E R Fossum
State of the Art Pixel counts for consumers are in the 8-12Mpixel range. Pixel counts for professional cameras are inthe 20-40 Mpixel range Pixel counts for aerospace application areapproaching 100 Mpixels. Pixel size is 2.2 um or smaller for commonconsumer applications. 2008 E R Fossum
Diffraction Limit201816Cheap Lens Resolution(30 lp/mm)LENSSize (microns)14F/1112Airy Disk DiameterD 2.44 λ F#108F/2.86420400 2008 E R FossumHigh Performance Lens Resolution(120 lp/mm)BBGG500RR600700800Wavelength (nm)90010001100
SDL Pixels are Coming Today you can make a 6T SRAM cell in lessthan 0.7 um2 using 65 nm technology. CMOS active pixels are typically under 4T usingshared readouts so 0.5 um pixel size (0.25 um2)seems around the corner. Never underestimate the force of marketing.Marketing Engineering If you can’t fix it, feature it. (purpose of this talk) 2008 E R Fossum
Consider 0.5 um pixel4 um0.5 umAiry Disk3.7 umAt 550 nmAbout 40 0.5 um SDL pixels fit inside the Airy Disk 2008 E R Fossum
The Joy of SDL-Pixels(as seen by marketing)For 0.5 um pixel: 3,456 x 2,304 (8 Mpixel) 1.73 mm x 1.15 mm 2 mm diagonal 1/9 inch format 2008 E R Fossum 38,750 x 25,833 (1 Gpixel) 19.5 mm x 13 mm 23 mm diagonal APS-C is 22.5mm x 15 mm (30% larger)
Small engineering problems Reduced full-well capacity (Q CV)– 0.5 um pixel has 1% of area of 5.6 um pixel– Reduced operating voltage Reduced SNR and DR– Fewer photoelectrons for same exposure Increased optical and carrier cross-talk Increased dark current due to higher doping andsharper corners Non-uniformity 2008 E R Fossum
SDL Pixels can oversamplespatial dimensions No color aliasingImproved spatial resolutionFPN reductionWill require more digitalsignal processing Doesn’t really address fullwell and SNR issues 2008 E R Fossum
ProposedDigital Film Sensor (DFS) 2008 E R Fossum
Silver Halide FilmDensity D exposure H 2008 E R Fossum AgX grains inemulsion exposedto light Grains hit byphoton(s) are“tagged” latentimage Developer convertswhole grain orwashes it away
D-Log(H) Characteristic1Probability of hitting those lastgrains only approaches unity.Density of Exposed Grains(normalized)0.90.80.70.60.50.40.30.2Linear regimeProbability of grain being hit light exposure0.101 2008 E R Fossum10Log Exposure H (arbitrary units)100
Grain size – Speed trade off Larger grains have greater probability of beingtagged for same exposure. Since entire grain becomes silver (black) uponexposure, larger grain size film seems moresensitive to light Trades spatial resolution for sensitivity 2008 E R Fossum
Translate from Film toSolid-State Specialized deep SDL pixel called “jot”Sensitive to single photoelectronIf hit, jot is logic “1” upon or before readoutElse, jot is logic “0” Could be a very-high-conversion-gain CMOS APS pixelStill sensitive to dark currentLess sensitive to lag (stacked structure?)Eliminates full well and uniformity concerns 2008 E R Fossum
Digital Development in twosteps First, area amplification (converting awhole grain to “1’s”) Second, converting binary data to graytones. 2008 E R Fossum
1st Step of Digital DevelopmentDigital development of grainsmade of 4x4 neighborhoodsJots with registered photon hitsRegion-growing approach fordigital development (3x3) 2008 E R Fossum
2nd Step of Digital Development Select kernel of certain areaand weighting fnc. Convolve kernel with grain(or jot) pattern to measuredensity Gray-scale valueproportional (or not) todensity. Can resample convolutionresult at arbitrary resolution 2008 E R Fossum
Some interesting opportunities Can dynamically adjust grain size to tradespatial resolution for light sensitivity Can still do color using color filter arrays – justtreat each color plane independently Can scan out jot pattern multiple times perexposure since readout rates of binary imagecan be quite high (e.g. 50 nsec/row) makinggrains spatial and temporal Could apply neural nets to digital development 2008 E R Fossum
Some interesting questions What happens to photon shot noise in this DFSimaging paradigm. Same, or shaped? Will the D-logH characteristics of the DFS be moreappealing to photography and cinematography? Does the DFS have scientific applications, esp.using the multi-scan mode (basically, a photoncounting sensor)? Can you use this in a cell-phone for video? What are dynamic range and SNR limitations forthis device, and how does one optimally tradeagainst spatial resolution (grain size)? 2008 E R Fossum
Conclusions CMOS image sensors are about as old asCCDs were in 1986 Most improvements will be incremental Still accessible for start-ups thanks tofoundry support Next generation image sensors will requiresome sort of paradigm shift and will needto be market driven. Jot-based sensors may be interesting 2008 E R Fossum
Used vanilla CMOS process available at many foundries Single-stage "CCD" in each pixel to allow complete charge transfer In-pixel source-follower amplifier for charge gain Allows low noise CDS operation In-column FPN reduction Permitted high performance camera-on-a-chip Basis of all modern CMOS image sensors
CMOS APS, or Monolithic Active Pixel Sensors, MAPS 1, have several advantages. 1 The terms MAPS is used to distinguish CMOS APS from hybrid detectors (also called Hybrid Active Pixel Sensors or MAPS are
Keyyg ggy( p) to Integrated Imaging System (camera on a chip) is making a good pixel in a CMOS platform process. Prior to CMOS active pixel was CMOS passive pixel. Simple and DRAM-like Not such a good pixel dthih d idue to high read noise Gets worse with small pixels, large arrays and M1 RS Vpd Cpd Ipd pixels, large arrays and
CMOS APS: The CMOS pixel sensors use a photogate or photodiode to capture the photon flux and convert it into corresponding electrical charge. The main photodiode type MOS pixel sensors which use the photon flux integration mode can be divided into 2 basic approaches, i.e. of passivepixels (PPS) and active pi
the various types of image sensors, complementary metal oxide semiconductor (CMOS) based active pixel sensors (APS), which are characterized by reduced pixel size, give fast readouts and reduced noise. APS are used in many applications such as mobile cameras, digital cameras, Webcams, and many consumer, commercial and scientific applications.Cited by: 2Publish Year: 2007Au
Full-frame vs. APS-C, some other considerations 10 IV. THE ECONOMICS OF IMAGE SENSORS 11 Wafers and sensors 11 V. WHY CMOS? 13 CCD 13 CMOS 14 Power consumption issues 15 Speed issues 16 Noise issues 17 VI. CANON’S UNIQUE R&D SYNERGY 20 Other differences between CMOS and CCD image sensors
i dc 5fA 2%i dc pixel/offset A D 50µm 2 0. 4%A D pixel/gain C D 20fF 0. 4%C D pixel/offset,gain v TR 1.1V 0. 2%v TR pixel/offset C R 0.4fF 0. 4%C R pixel/pffset v TF 0.9V 0. 2%v TF pixel/offset W F L F 4 2 0. 2% W F L F pixel/offset i s 1. 88µA 1%i s column/offset k e 7-21
CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode . Scientific CMOS (sCMOS) 95% QE #electrons #photons Meynants et al. IISW 2013 QE of a 24MP CMOS full-frame sensor . Color Archi
awakening joy and beauty. On the Equinox, the Vase will be planted in Crete’s fertile soil to seed a new story for these times and connect with a global mandala of healing, protection and renewal for the Earth. The Practice of the Earth Treasure Vases Almost 30 years ago, on a life-changing pilgrimage to meet a 106-year-old lama living in a remote cave in Nepal, Cynthia Jurs met the great .
