Shape from Silhouettes IGuido GerigCS 6320, Spring 2015Credits: Marc Pollefeys, UNC Chapel Hill, some of the figuresand slides are also adapted from J.S. Franco, J. Matusik’spresentations, and referenced papers)
Shape from silhouettesSlides fromLazebnik,MatusikYerexand othersAutomatic 3D Model Construction for Turn-Table Sequences,A.W. Fitzgibbon, G. Cross, and A. Zisserman, SMILE 1998
Big Picture Multi-cameraenvironments Dynamic sceneOutdoor data capturing with 9 video cameras behind the Ackland Museum,UNC-Chapel Hill, 2006/8/24. Pictured by Jae Hak Kim. N cameras observe the scene and produce Nvideo streams What can we do with this data?
Motivation: MoviesSinha Sudipta, UNC PhD 2008
Motivation: 3D from MoviesSinha Sudipta, UNC PhD 2008
Motivation: 3D from Movies:Replay from arbitrary viewpointsSinha Sudipta, UNC PhD 2008
What can we do with this data? Reconstruct scene objects:–shape from silhouettes–photo-consistencyCalibrate cameras– recover epipolar geometryFit specific models (articulated models)
Outline Silhouettes––––––basic conceptsextract silhouettesfundamentals about using silhouettesreconstruct shapes from silhouettesuse uncertain silhouettescalibrate from silhouettes Perspectives and interesting ideas
Silhouettes of objects of interest Silhouettes are the regions where objects ofinterest project in images Silhouettes can generally be obtained usinglow level information (fast) They give information about the global shapeof scene objects
How to extract silhouettes? Sometimes done manually (for offlineapplications, ground truth and verifications) Region based-extraction (automatic)– silhouette extraction is a 2-region imagesegmentation problem, w/ specificsolutions: chroma keying (blue, green background) background subtraction (pre-observed static ordynamic background)(refer to segmentation course)
How to extract silhouettes? Contour-based extraction focus on silhouette outline instead of regionitself–snakes, active contours: fitting of a curve to highgradients in image, local optimizationYilmaz&Shah ACCV04
How to extract silhouettes? (cont.) Background subtractionSimple thresholdingTrain an appearance model foreach pixel, from a set ofbackground images RGB 3D-Gaussian modelHSV modelGMM modelNon-parametric model(histogram/kernel densityfunction) Apply the pixel color to the model,then classify it to beforeground/background We will talk about this in moredetail later
Why use a Visual Hull? Good shape representationCan be computed efficientlyNo photo-consistency requiredAs bootstrap of many fancyrefinement background foregroundbackground-foreground
Outline Silhouettes––––––basic conceptsextract silhouettesfundamentals about using silhouettesreconstruct shapes from silhouettesuse uncertain silhouettescalibrate from silhouettes Perspectives and cool ideas
What is shape fromsilhouette? The silhouette, or occluding contourof an object in an image containssome information about the 3Dshape of the object. Given a single silhouette image ofan object, we know that the 3Dobject lies inside the volumegenerated by back-projecting thesilhouette area using the cameraparameters.
What is shape fromsilhouette? With multiple views of thesame object, we canintersect the generalizedcones generated by eachimage, to build a volumewhich is guaranteed tocontain the object. The limiting smallestvolume obtainable in thisway is known as the visualhull of the object.
Literature Theory– Laurentini ’94, Petitjean ’98, Laurentini ’99 Solid cone intersection:– Baumgart ’74 (polyhedra), Szeliski ’93 (octrees) Image-based visual hulls– Matusik et al. ’00, Matusik et al. ’01 Advanced modeling– Sullivan & Ponce ’98, Cross & Zisserman ’00,Matusik et al. ’02 Applications– Leibe et al. ’00, Lok ’01, Shlyakhter et al. ’01
One-view silhouette geometry
Multi-view silhouette geometry:the Visual HullVisual hull Maximal volume consistentwith silhouettes[Laurentini94] [Baumgart74] Viewing cone Can be seen as theintersection of viewing conesProperties: Containment property: contains real scene objectsConverges towards the shape of scene objects minusconcavities as N increasesProjective structure: simple management of visibilityproblems
Visual Hull: A 3D Example
Convex Hull: ComputationalGeometry ProblemIn mathematics, the convex hull orconvex envelope for a set of points Xin a real vector space V is theminimal convex set containing X.Convex hull: Elasticband analogy:Concave parts ofobject not part ofhull.In computational geometry, it iscommon to use the term "convexhull" for the boundary of the minimalconvex set containing a given nonempty finite set of points in the plane.Unless the points are collinear, theconvex hull in this sense is a simpleclosed polygonal chain.
Convex Hull: ComputationalGeometry ProblemHint: Calculate the convex hull based on the Delauneytriangulation and its dual, the Voronoi diagram.
Outline Silhouettes––––––basic conceptsextract silhouettesfundamentals about using silhouettesreconstruct shapes from silhouettesuse uncertain silhouettescalibrate from silhouettes Perspectives and cool ideas
What representation for scene objects?Voxel gridSurfaceVolumetric approachesSurface approachesImage-basedapproachesPolyhedron meshA priori knowledgeex: articulated model
General idea and assumptions 2 main families of approaches for VH:– focus on visual hull as volume: locate portions ofspace that don't project in silhouettes (carving) use 2D silhouette regions in images– focus on visual hull as surface: locate theboundary surface of the visual hull use 2D silhouette contours in images General assumptions:– very good silhouettes are extracted– views are calibrated parameters and positions are known
Computational complexity Intersection of many volumes can be slow Simple polyhedron-polyhedron intersectionalgorithms are inefficient To improve performance, most methods:– Quantize volumesand/or– Perform Intersection computations in 2D not 3D
Algorithms Standard voxel basedmethodMarching Intersections Exact polyhedral methods Image-based visual hulls
Voxel based– First the object space is split up into a 3Dgrid of voxels.– Each voxel is intersected with eachsilhouette volume.– Only voxels that lie inside all silhouettevolumes remain part of the final shape.
Visual hull as voxel grid Identify 3D region using voxel carving– does a given voxel project inside all silhouettes? pros: simplicity cons: bad precision/computation timetradeoff
Classical voxel grid improvement:octrees Same principle, but refinement throughspace subdivision[Szeliski TR 90’]
Marching intersectionsTarini et al., 2002– The object space is again split up into a 3D grid.– The grid used is made of 3 sets of rays, rather thanvoxels.– Rays are aligned with the 3 axes, and store points ofentry/exit into the volume– Each silhouette cone can be converted to themarching intersections data structure.– Then merging them is reduced to 1D intersectionsalong each ray.M. Tarini et al, Marching intersections, Anefficient Approach to Shape from Silhouette
Marching intersections example
Marching intersections example
Marching intersections -ConceptM. Tarini et al,Marchingintersections, Anefficient Approachto Shape fromSilhouette Given a curve Select reference grid Intersections between curve and horizontal andvertical lines: MI Create look-up-table for each non-empty box
Marching intersections Silhouettes Convert conoid structures to MI datastructure Intersection tested in 2D image: purely 2Doperation Intersection of conoids: AND operations on MIdatastructures
Marching intersections SilhouettesFinal step: Convert MI datastructure representingall intersections to triangular mesh
Marching intersections Silhouettes
Example: Student Project Compute visual hull with silhouette images from multiplecalibrated camerasCompute Silhouette ImageVolumetric visual hull computationDisplay the result
Algorithms Standard voxel basedmethod Exact polyhedral methods Image-based visual hulls
Exact Polyhedral MethodsWojciech Matusik et al.– First, silhouette images are converted topolygons. (convex or non-convex, withholes allowed)– Each edge is back projected to form a 3dpolygon.– Then each polygon is projected onto eachimage, and intersected with each silhouettein 2D.– The resulting polygons are assembled toform the polyhedral visual hullWojciech Matusik, An Efficient Visual Hull Computation Algorithm
Exact Polyhedral MethodsWojciech Matusik, An Efficient Visual Hull Computation Algorithm
Exact Polyhedral - example
Wojciech Matusik et al.
Wojciech Matusik et al.
Metric Cameras and Visual-HullReconstruction from 4 viewsFinal calibration quality comparable to explicit calibration procedure
IBVH Results Approximately constant computation perpixel per camera Parallelizes Consistent with input silhouetteshttp://www.youtube.com/watch?v Lw9aFaHobao
IBVH Results
Image Based Visual Hullshttp://www.youtube.com/watch?v Lw9aFaHobaoSee also: http://www.youtube.com/watch?v UdmBW4kDcok
Video ShadingMoviehttp://www.google.com/url?q uRpSG9iYW8%26image-based-visualhulls%3D&ei Zc8ES8wggaKyA73OrcIK&sa X&oi video result&resnum 5&ct thumbnail&ved 0CCQQuAIwBA&usg AFQjCNETWA Eqgy8 10mJUmc540zpx8T6A
Non-parametric model (histogram/kernel density function) Apply the pixel color to the model, . - use uncertain silhouettes - calibrate from silhouettes Perspectives and cool ideas. What is shape from . Advanced modeling - Sullivan & Ponce '98, Cross & Zisserman '00, Matusik et al. '02
Texts of Wow Rosh Hashana II 5780 - Congregation Shearith Israel, Atlanta Georgia Wow ׳ג ׳א:׳א תישארב (א) ׃ץרֶָֽאָּהָּ תאֵֵ֥וְּ םִימִַׁ֖שַָּה תאֵֵ֥ םיקִִ֑לֹאֱ ארָָּ֣ Îָּ תישִִׁ֖ארֵ Îְּ(ב) חַורְָּ֣ו ם
National Silhouettes of Kappa Alpha Psi Fraternity, Inc Policies and Procedures Manual 1. Membership A. Conditions/Requirements for membership Membership is restricted to the wife or widow of a financial member of Kappa Alpha Psi Fraternity, Inc. Sweethearts and lady friends of Kappa
: Show 3D Text.: Show 2D Text with a back shape.: Show 3D Text with a back shape.: Show a text hole in shape.: Show 3D Text in the border of shape. One Click add a object: Click the item to add a object quickly. Change Shape or Text Style: Double click the style icon can change node's shape or
Step 7 - Adding Shape # 2 Duplicate the shape from previous step by first selecting the arrow (A). While holding down the alt key click on the shape and drag it below the yellow rectangle. This will duplicate the shape. Now hold down the command key (apple) and drag the four points to transform your shape properly. Step 8 - Duplicating Shape #2
Shape Analysis & Measurement The extraction of quantitative feature information from images is the objective of image analysis. The objective may be: – shape quantification – count the number of structures – characterize the shape of structures. 3 Shape Measures
Sevtsuk, A. “”How we shape our cities, and then they shape us”, MAJA: the Estonian Architectural Review, 2-2012 (72), pp. 10-15, 2012. !! 1! How we shape our cities, and then they shape us. The quality of the built envi
For 6 BP the Molecular Shape is: Octahedral. Try These! Determine the geometric and molecular shape for the following: BrCl 4-CH 2F 2 HCN AsCL 5 SO 2 NH 4 SO 3 BF 4-SO 4 2-Shape and Polarity Shape and Polarity are directly related . Dipole u In diatomic molecules bond polarity applies to the overall molecule
GRADE: K . Strand: READING STANDARDS FOR LITERATURE Cluster 1: Key Ideas and Details STANDARD CODE STANDARD LAFS.K.RL.1.1 With prompting and support, ask and answer questions about key details in a text. Cognitive Complexity: Level 2: Basic Application of Skills & Concepts LAFS.K.RL.1.2 With prompting and support, retell familiar stories, including key details. Cognitive Complexity: Level 2 .