The In And Out: Making Games Play Right With Stereoscopic .

The In and Out:Making Games Play Right withStereoscopic 3D TechnologiesSamuel Gateau

OutlineNVIDIA 3D VisionStereoscopic driver & HW display solutionsStereoscopic basicsDefinitions and equationsRendering techniquesPractical recommendationsAdvanced stereo effectsSelection Marquee3D Video

How does it work ?NVIDIA 3D VISION

NVIDIA 3D VisionStereoscopic 3D DriverDirectXStereoscopic 3D DriverMicropol DisplayAny monitor120 Hz LCDsDLPs3D ProjectorPassive Polarized GlassesAnaglyph Red/blueAnaglyph GlassesShutter GlassesShutter GlassesShutter Glasses

How It Works3D game data is sent to stereoscopic driverThe driver takes the 3D game data and renderseach scene twice – once for the left eye and oncefor the right eye.Left Eye viewA Stereoscopic display then shows the left eyeview for even frames (0, 2, 4, etc) and the righteye view for odd frames (1, 3, 5, etc).Right Eye view

How It WorksLeft eye view on,right lens blockedRight eye view on,left lens blockedIn this example active shutter glasses black-out theright lens when the left eye view is shown on thedisplay and black-out the left lens when the righteye view is shown on the display.This means that the refresh rate of the display iseffectively cut in half for each eye. (e.g. a displayrunning at 120 Hz is 60 Hz per eye)onLeft lensThe resulting image for the end user is acombined image that appears to have depth infront of and behind the stereoscopic 3D Display.offRight lensoffLeft lensonRight lens

NVIDIA 3D VisionNVAPI Stereoscopic ModuleNVAPI is NVIDIA's core software development kit thatallows direct access to NVIDIA GPUs and driversNVAPI now expose a Stereoscopic Module providingaccess to developer to the Stereoscopic driversettingsDetect if the system is 3D Vision capableManage the stereo profile settings for the gameControl dynamically the stereo parameters from within thegame engine for a better stereo experienceFor download and api.html

Under the hoodSTEREOSCOPIC BASICS

Stereo BasicsStandard MonoScene is viewed from one mono eye and projected on Near Clipping plane inViewportNear clipping planeYZXEye spaceViewport

Stereoscopic BasicsTwo eyes, one screen, two imagesLeft and Right eyesLeft FrustumRight FrustumShifting the mono eye alongthe X axisScreenLeft ImageRight ImageOne “virtual” screenWhere the left and rightfrustums convergeNear clipping planeLeft RightImage ImageTwo imagesYZX2 images are generated atthe near clipping plane ineach viewsEye spacethen presented independentlyto each eyes of the user on thereal screen

Stereoscopic BasicsInteraxialDistance between the 2 virtual eyes in eye spaceThe mono, left & right eyes directions are all parallelsScreenEye spaceYZXLeft EyeMono EyeRight EyeInteraxial

Stereoscopic BasicsScreen DepthAlso called “Convergence”Screen‘s virtual depth in eye spacePlane where Left and Right Frustums intersectScreenEye spaceYZXLeft FrustumLeft EyeMono EyeRight EyeRight FrustumScreen Depth

Stereoscopic BasicsLeft / Right ProjectionProjection matrix for each eyes is a horizontallymodified version of regular mono projection matrixShifting X coordinate left or rightScreenEye spaceYZXLeft FrustumLeft EyeMono EyeMono FrustumRight EyeRight Frustum

Stereoscopic BasicsParallaxSigned Distance on the screen between the projected positionsof one vertex in left and right imageParallax is function of the depth of the vertex in eye spaceClip spaceYScreenZParallaxXLeft EyeMono EyeRight EyeScreen DepthVertex depth

Stereoscopic BasicsIn / Out of the ScreenParallax creates the depth perception relative to the screenWhen Parallax is negative, vertex appears Out of the screenOut of the ScreenLeft EyeMono EyeRight EyeEye spaceYZXScreenIn the Screen

Stereoscopic BasicsParallax in equationIn normalized window space, after perspective division by WDepth FactorDepth Factor is the maximumParallax for object at infinityVertex Depth (W)Screen DepthParallax in window spaceParallax DepthFactor * ( 1 – ScreenDepth / W )Parallax is 0 at screen depthParallax diverges quickly to negative infinityfor object closer to the eye

Stereoscopic BasicsLeft / Right surfacesThe back buffer is duplicatedIntermediate full screen render targets usedto process the final image should also beduplicatedHigh dynamic range, Blur, BloomScreen Space Ambient OcclusionSome render targets DON’T need to beduplicatedShadow mapview independent render targets

Stereoscopic BasicsStereoscopic Surfaces & SeparationStereoscopic rendering is the result of2 independent actionsStereoscopic surfacesRendering surfaces are duplicatedStereoscopic separationParallax shift is applied to the geometryvertices

Stereoscopic BasicsStereoscopic Surfaces in NVIDIA driverAutomatic duplication is based on thesurface sizeSurfaces equal or larger than back buffer sizeare duplicatedSquare surfaces are NOT duplicatedSmall surfaces are NOT duplicatedHeuristic defined by driver profile settingConsult documentation for fine tuningExplicit duplication will be availableprogrammatically soon

Stereoscopic BasicsStereoscopic Separation in NVIIDAdriverEvery Draw callIssued twice in left & right surfacesParallax shift is applied in the vertexshader to the vertex’s clip position outputWhen separation is not required, rendergeometry at Screen depthFull screen quad to do image filteringNo separation if the surface is mono

Recommendations RENDERING TECHNIQUES

Rendering Techniques3D ObjectsAll the 3D objects should be rendered usinga unique Perspective Projection in a givenframeThe sky box should be drawn with a validdepth further than the regular sceneBest is at the Far distanceAll the 3D objects must have a coherentdepth relative to the scene

Rendering TechniquesParallax BudgetDefine Screen Depth for best usage of the parallaxbudget in screen and out of the screenJust like when dealing with Near and FarFarDepth FactorVertex Depth (W)Screen DepthParallax in window spaceNear

Rendering TechniquesDefining Screen DepthMaximizing the usage of the depth range to see thevariation of separation along the depthScene Depth should range between Screen Depth and100*Screen DepthWith NVIDIA 3D Vision, Screen depth can be definedfrom NVAPI

Rendering Techniques3D Objects CullingCulling should be extended horizontally to see extraleft and right frustum space after screen depthScreenLeft FrustumLeft EyeMono EyeMono FrustumRight EyeRight Frustum

Rendering Techniques2D Objects2D Overlay elements (defined in window space) mustbe drawn at a valid DepthAt the screen depth to look monoHead Up Display interfaceUI elementsAt the correct depth when interacting with the 3D sceneMouse Cursor at the pointed object’s depthCan not use the HW cursorCrosshairLabels or billboards in the sceneThe depth is provided by the game engineNeeds to modify the projection to take intoaccount depth

Rendering Techniques2D Objects hybrid projectionProposed vertex shaderfloat4 2DObjects VertexShader(in float2 posClip : POSITION, // Input position in clip spaceuniform float depth// Depth where to draw the 2D object) : POSITION// Output the position in clip space{return float4(posClip.xy * depth,// Simply scale the posClip by the depth// to compensate for the division by W// performed before rasterization0,// Z is not used if the depth buffer is not used// If needed Z ( depth * f – nf )/(f – n);depth ); // W is the Z in eye space}

Stereoscopic BasicsOut of the screen objectsThe user’s brain is fighting against the perception ofhovering objects out of the screenExtra care must be taken to achieve a convincing effectMake sure object is not clipped by the edges of thewindowBe aware of the extra guard bandsMove object slowly from inside the screen to theoutside area to give eyes time to adaptMake smooth visibility transitionsNo blinkingRealistic rendering helps

The cool partADVANCED STEREOSCOPICEFFECTS

Advanced Stereoscopic EffectsSelection marqueeSomething is brokenIn mono, the selection marquee is naturally defined in the windowspace as a 2D shapeIt can be simply drawn as a rectangle in 2D in window spaceIn stereo, the same solution does not workEach view defines its own selection rectangle in its clipping spaceThe vertical edges of the rectangles don’t matchArea in the rectanglein right imageArea in the rectanglein left imageSelection rectangle

Advanced Stereoscopic EffectsSelection marqueeIn fact it’s a volumeThe selection marquee seen in mono is in fact a 2D shape projectedon the 3D scene and intersecting with the scene surfacethe 2D shape defined in near clip plane is generating a selectionvolumeNear planeSelection marqueeprojected on 3D scene

Advanced Stereoscopic EffectsSelection marqueeRepresenting the selection surfaceIn stereo, we want to represent the selection marquee as thefragments of the 3D scene inside the selection volumeJust like shadow volumeRender the selection volume with z testagainst the depth bufferand just incrementing the stencil bufferRender the selection volumedisplaying only the fragmentswith a odd stencil (inside the volume)Selection marqueefor right view

Advanced Stereoscopic EffectsSelection marqueeRepresenting the selection surfaceRender the selectionvolume with z testagainst the depth bufferand just incrementingthe stencil bufferRender the selectionvolumedisplaying only thefragmentswith a odd stencil(inside the volume)Left ViewportYZXNearRight Viewport

Advanced Stereoscopic EffectsSelection marqueeGoing back to the 2D spaceFor shading purpose, we need to know the 2Dcoordinate of the fragment expressed in the original2D selection spaceThe fragment is on the selection volumeAt that fragment, fetch the view space position of the 3Dscene from a previously rendered position bufferTransform back that position to the mono 2D windowspaceYZXNear

Advanced Stereoscopic EffectsSelection marqueeConclusionIt works with any 2D selection marquee shape, justneed to generate the corresponding selectionvolumeLasso !An accurate solution to represent the selectionmarquee in stereoRequires an extra buffer containing the position inview space of 3D scene fragments

Advanced Stereoscopic Effects3D VideoDisplaying a stereo frameHow to display existing stereo contentin Stereo ?Replay 3D videoDisplay 3D photosPrerecorded cut scenes

Advanced Stereoscopic Effects3D VideoPresenting a stereo frame from 2 imagesOn a 3D vision driver enabledplatform under DirectX9Upload new frame (left andright) in a video surface2 * WidthHeight 1Left image on the left, Rightimage on the rightEdit stereo tag in the last extrarawStretchRect the video surfaceinto the back bufferPresenting the back buffer willdisplay the stereo frameLeftw*hRightw*hUploadVideo stereoLeftRightsurfaceStereo tagStretchRectBackBufferPresentRightLeft

Advanced Stereoscopic Effects3D VideoPresenting a stereo frame: Creating the source imageIDirect3DSurface9* gImageSrcLeft; // Left Source image surface in video memoryIDirect3DSurface9* gImageSrcRight; // Right Source image Surface in video memoryint gImageWidth 1680;int gImageHeight 1050;// Source image width// Source image heightIDirect3DSurface9* gImageSrc NULL; // Source stereo image beeing createdD3DDev- CreateOffscreenPlainSurface(gImageWidth * 2,// Stereo width is twice the source widthgImageHeight 1,// Stereo height add one raw to encode signatureD3DFMT A8R8G8B8, D3DPOOL DEFAULT, // Surface is in video memory&gImageSrc, NULL);// Blit left src image to left side of stereoRECT srcRect { 0, 0, gImageWidth, gImageHeight };RECT dstRect { 0, 0, gImageWidth, gImageHeight };D3DDev- StretchRect(gImageSrcLeft, &srcRect, gImageSrc, &destRect, D3DTEXF LINEAR);// Blit left src image to left side of stereodstRect {gImageWidth, 0, 2*gImageWidth, gImageHeight };D3DDev- StretchRect(gImageSrcRight, &srcRect, gImageSrc, &destRect, D3DTEXF LINEAR);