Pyramid Blending, Templates, NL Filters

Pyramid Blending, Templates, NL FiltersCS194: Intro to Comp. Vision and Comp. PhotoAlexei Efros, UC Berkeley, Fall 2021

Gaussian is not perfectGaussian

But better than box filter!Box Filter

Low-pass, Band-pass, High-pass filterslow-pass:High-pass / band-pass:

Edges in images

Low Pass vs. High Pass filteringImageSmoothedDetails

Filtering – SharpeningImageDetails α“Sharpened” α 1

Filtering – SharpeningImageDetails α“Sharpened” α 0

Filtering – SharpeningImageDetails α“Sharpened” α 2

Filtering – SharpeningImageDetails α“Sharpened” α 0

Filtering – Extreme SharpeningImageDetails α“Sharpened” α 10

Unsharp mask filterf ( f f g ) (1 ) f f g f ((1 )e g )imageunit impulseunit impulse(identity)blurredimageGaussianLaplacian of Gaussian

application: Hybrid ImagesAude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006

Application: Hybrid ImagesGaussian FilterA. Oliva, A. Torralba, P.G. Schyns,“Hybrid Images,” SIGGRAPH 2006Laplacian Filterunit impulseGaussian Laplacian of Gaussian

Yestaryear’s homework(CS194-26: Riyaz Faizullabhoy)Prof. Jitendros Papadimalik

Band-pass filtering in spatial domainGaussian Pyramid (low-pass images)Laplacian Pyramid (subband images)Created from Gaussian pyramid by subtraction

Laplacian PyramidNeed this!OriginalimageHow can we reconstruct (collapse) thispyramid into the original image?

Da Vinci and The Laplacian Pyramid

Da Vinci and The Laplacian PyramidLeonardo playing with peripheral visionLivingstone, Vision and Art: The Biology of Seeing

Blending

Alpha Blending / Feathering 1010Iblend Ileft (1- )Iright

Affect of Window Size1left1right00

Affect of Window Size1100

Good Window Size10“Optimal” Window: smooth but not ghosted

What is the Optimal Window?To avoid seams window size of largest prominent featureTo avoid ghosting window 2*size of smallest prominent featureNatural to cast this in the Fourier domain largest frequency 2*size of smallest frequency image frequency content should occupy one “octave” (power of two)FFT

What if the Frequency Spread is WideFFTIdea (Burt and Adelson) Compute Fleft FFT(Ileft), Fright FFT(Iright) Decompose Fourier image into octaves (bands)– Fleft Fleft1 Fleft2 Feather corresponding octaves Flefti with Frighti– Can compute inverse FFT and feather in spatial domain Sum feathered octave images in frequency domainBetter implemented in spatial domain

Octaves in the Spatial DomainLowpass ImagesBandpass Images

Pyramid Blending101010Left pyramidblendRight pyramid

Pyramid Blending

laplacianlevel4laplacianlevel2laplacianlevel0left pyramidright pyramidblended pyramid

Blending Regions

Laplacian Pyramid: BlendingGeneral Approach:1. Build Laplacian pyramids LA and LB from images A and B2. Build a Gaussian pyramid GR from selected region R3. Form a combined pyramid LS from LA and LB using nodesof GR as weights: LS(i,j) GR(I,j,)*LA(I,j) (1-GR(I,j))*LB(I,j)4. Collapse the LS pyramid to get the final blended image

Horror Photo david dmartin (Boston College)

Results from this class (fall 2005) Chris Cameron

Simplification: Two-band BlendingBrown & Lowe, 2003 Only use two bands -- high freq. and low freq. – without downsampling Blends low freq. smoothly Blend high freq. with no smoothing: use binary alpha

2-band “Laplacian Stack” BlendingLow frequency (l 2 pixels)High frequency (l 2 pixels)

Linear Blending

2-band Blending

Side note: Image Compression89k

Lossless Compression (e.g. Huffman coding)Input image:Pixel histogram:Pixel code:Compressed image:0 110 110 0 00 10 110 111 0 g-jpeg

Lossless Compression not enough

Lossy Image Compression (JPEG)Block-based Discrete Cosine Transform (DCT)

Using DCT in JPEGThe first coefficient B(0,0) is the DC component,the average intensityThe top-left coeffs represent low frequencies,the bottom right – high frequencies

Image compression using DCTQuantize More coarsely for high frequencies (which also tend to have smallervalues) Many quantized high frequency values will be zeroEncode Can decode with inverse dctFilter responsesQuantization tableQuantized values

JPEG Compression SummarySubsample color by factor of 2 People have bad resolution for colorSplit into blocks (8x8, typically), subtract 128For each blocka. Compute DCT coefficientsb. Coarsely quantize–Many high frequency components will become zeroc. Encode (e.g., with Huffman .wikipedia.org/wiki/JPEG

Block size in JPEGBlock size small block– faster– correlation exists between neighboring pixels large block– better compression in smooth regions It’s 8x8 in standard JPEG

JPEG compression comparison89k12k

Review: Smoothing vs. derivative filtersSmoothing filters Gaussian: remove “high-frequency” components;“low-pass” filter Can the values of a smoothing filter be negative? What should the values sum to?– One: constant regions are not affected by the filterDerivative filters Derivatives of Gaussian Can the values of a derivative filter be negative? What should the values sum to?– Zero: no response in constant regions High absolute value at points of high contrast

Template matchingGoal: findin imageMain challenge: What is agood similarity ordistance measurebetween two patches? CorrelationZero-mean correlationSum Square DifferenceNormalized Cross CorrelationSide by Derek Hoiem

Matching with filtersGoal: findin imageMethod 0: filter the image with eye patchh[ m, n] g[ k , l ] f [ m k , n l ]k ,lf imageg filterWhat went wrong?InputFiltered ImageSide by Derek Hoiem

Matching with filtersGoal: findin imageMethod 1: filter the image with zero-mean eyeh[ m, n] ( f [ k , l ] f ) ( g[ m k , n l ] )mean of fk ,lTrue detectionsFalsedetectionsInputFiltered Image (scaled)Thresholded Image

Matching with filtersGoal: findin imageMethod 2: SSD (L2)h[ m, n] ( g[ k , l ] f [ m k , n l ] )2k ,lTrue detectionsInput1- sqrt(SSD)Thresholded Image

Matching with filtersCan SSD be implemented with linear filters?h[ m, n] ( g[ k , l ] f [ m k , n l ] )2k ,lSide by Derek Hoiem

Matching with filtersGoal: findin imageMethod 2: SSDWhat’s the potentialdownside of SSD?h[ m, n] ( g[ k , l ] f [ m k , n l ] )2k ,lInput1- sqrt(SSD)Side by Derek Hoiem

Matching with filtersGoal: findin imageMethod 3: Normalized cross-correlationmean templateh[ m, n] mean image patch ( g[k , l ] g )( f [m k , n l ] fm ,n)k ,l 22 ( g[ k , l ] g ) ( f [ m k , n l ] f m,n ) k ,l k ,l 0.5Side by Derek Hoiem

Matching with filtersGoal: findin imageMethod 3: Normalized cross-correlationTrue detectionsInputNormalized X-CorrelationThresholded Image

Matching with filtersGoal: findin imageMethod 3: Normalized cross-correlationTrue detectionsInputNormalized X-CorrelationThresholded Image

Q: What is the best method to use?A: DependsZero-mean filter: fastest but not a greatmatcherSSD: next fastest, sensitive to overallintensityNormalized cross-correlation: slowest,invariant to local average intensity andcontrastSide by Derek Hoiem

DenoisingGaussianFilterAdditive Gaussian Noise

Reducing Gaussian noiseSmoothing with larger standard deviations suppresses noise,but also blurs the imageSource: S. Lazebnik

Reducing salt-and-pepper noise by Gaussian smoothing3x35x57x7

Alternative idea: Median filteringA median filter operates over a window byselecting the median intensity in the window Is median filtering linear?Source: K. Grauman

Median filterWhat advantage does median filteringhave over Gaussian filtering? Robustness to outliersSource: K. Grauman

Median filterSalt-and-peppernoiseMedian filteredMATLAB: medfilt2(image, [h w])Source: M. Hebert

Median vs. Gaussian filtering3x3GaussianMedian5x57x7

A Gentle Introductionto Bilateral Filteringand its Applications“Fixing the Gaussian Blur”:the Bilateral FilterSylvain Paris – MIT CSAIL

Blur Comes fromAveraging across Edgesinput*output**Same Gaussian kernel everywhere.

Bilateral Filter [Aurich 95, Smith 97, Tomasi 98]No Averaging across Edgesinput*output**The kernel shape depends on the image content.

Bilateral Filter Definition:an Additional Edge TermSame idea: weighted average of pixels.new1BF [ I ]p Wpnot newnew G ( p q ) G ( Iq Snormalizationfactorsspace weightrp I q ) I qrange weightI

Illustration a 1D Image 1D image line of pixels Better visualized as a plotpixelintensitypixel position

Gaussian Blur and Bilateral FilterGaussian blurpGB [ I ]p G ( p q ) I qqq SspacespaceBilateral filter[Aurich 95, Smith 97, Tomasi 98]prangeqBF [ I ]p 1Wp G ( p q ) G ( Iq Snormalizationspacesrspacep I q ) I qrange

Bilateral Filter on a Height FieldBF [ I ]p 1Wp G ( p q )q SsG r ( I p I q ) I qpqoutputinputreproducedfrom [Durand 02]

Space and Range Parameters1BF [ I ]p Wp G ( p q ) G ( Iq Ssrp I q ) I q space s : spatial extent of the kernel, size ofthe considered neighborhood. range r : “minimum” amplitude of an edge

Influence of PixelsOnly pixels close in space and in range are considered.spacerangep

Exploring the Parameter Space r 0.1input s 2 s 6 s 18 r 0.25 r (Gaussian blur)

Varying the Range Parameter r 0.1input s 2 s 6 s 18 r 0.25 r (Gaussian blur)

input

r 0.1

r 0.25

r (Gaussian blur)

Varying the Space Parameter r 0.1input s 2 s 6 s 18 r 0.25 r (Gaussian blur)

input

s 2

s 6

s 18

Laplacian Pyramid: Blending General Approach: 1. Build Laplacian pyramids LA and LB from images A and B 2. Build a Gaussian pyramid GR from selected region R 3. Form a combined pyramid LS from LA and LB using nodes of GR as weights: LS(i,j) GR(I,j,)*LA(I,j) (1-GR(I,j))*LB(I,j) 4. Collapse the LS pyramid to get the final blended image