Spatial Augmented Reality
Spatial Augmented RealityMerging Real and Virtual WorldsOliver Bimber, Ramesh RaskarThis copy was downloaded fromSpatialAR.com for individual use.Save20% on the printed copyof this book when you order onlineat www.akpeters.com. Use discountcode spatialar.Copyright 2005 by A K Peters, Ltd.All rights reserved. No part of the material protectedby this copyright notice may be reproduced or utilizedin any form, electronic or mechanical, including photocopying, recording, or by any information storage orretrieval system, without written permission from thecopyright owner.
iiiiSpatial Augmented Realityiiii
iiiiiiii
iiiiSpatial Augmented RealityMerging Real and Virtual WorldsOliver BimberBauhaus-University, WeimarRamesh RaskarMitsubishi Electric Research Laboratory,Cambridge, MAA K PetersWellesley, Massachusettsiiii
iiiiEditorial, Sales, and Customer Service OfficeA K Peters, Ltd.888 Worcester Street, Suite 230Wellesley, MA 02482www.akpeters.comCopyright 2005 by A K Peters, Ltd.All rights reserved. No part of the material protected by this copyrightnotice may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage andretrieval system, without written permission from the copyright owner.Library of Congress Cataloging-in-Publication DataBimber, Oliver, 1973–Spatial augmented reality : merging real and virtual worlds / OliverBimber, Ramesh Raskar.p. cm.Includes bibliographical references and index.ISBN 1-56881-230-21. Computer graphics- 2. Virtual reality. I. Raskar, Ramesh II. Title.T385.B5533006.8–dc2220042005043110Printed in the United States of America09 08 07 06 0510 9 8 7 6 5 4 3 2 1iiii
iiiiTo Mel—O. B.To my parents—R. R.iiii
iiiiiiii
iiiiContentsPrefacexi1 A Brief Introduction to Augmented Reality1.11.21.31.4What is Augmented Reality . .Today’s Challenges . . . . . . . .Spatial Augmented Reality . . .Outline of the Book . . . . . . .1.2 Fundamentals: From Photons to Pixels2.12.22.32.42.5Light in a Nutshell . . . .Geometric Optics . . . .Visual Depth Perception .Rendering Pipelines . . .Summary and Discussion13.3 Augmented Reality Displays3.13.23.33.4Head-Attached Displays .Hand-Held Displays . . .Spatial Displays . . . . . .Summary and DiscussionGeometric Model . . . . .Rendering Framework . .Calibration Goals . . . . .Display Environments andSummary and Discussion1417304466714 Geometric Projection Concepts4.14.24.34.44.51478. . . . . . . . . . . . . . . . . . . . . .Applications. . . . . . . .7279839093. 93. 98. 104. 108. 108viiiiii
iiiiviiiContents5 Creating Images with Spatial Projection Displays5.15.25.35.45.55.6Planar Displays . . . . . . . . . . . .Non-Planar Display . . . . . . . . .Projector Overlap Intensity BlendingQuadric Curved Displays . . . . . .Illuminating Objects . . . . . . . . .Summary and Discussion . . . . . .111.Transparent Screens . . . . . . . . . . . . .Mirror Beam Combiners . . . . . . . . . . .Planar Mirror Beam Combiners . . . . . . .Screen Transformation and Curved ScreensMoving Components . . . . . . . . . . . . .Multi-Plane Beam Combiners . . . . . . . .Curved Mirror Beam Combiners . . . . . .Summary and Discussion . . . . . . . . . .Image-Based Illumination: Changing Surface AppearanceCreating Consistent Occlusion . . . . . . . . . . . . . . . .Creating Consistent Illumination . . . . . . . . . . . . . .Augmenting Optical Holograms . . . . . . . . . . . . . . .Augmenting Flat and Textured Surfaces . . . . . . . . . .Augmenting Geometrically Non-Trivial Textured SurfacesSummary and Discussion . . . . . . . . . . . . . . . . . .6 Generating Optical Overlays6.16.26.36.46.56.66.76.81497 Projector-Based Illumination and Augmentation7.17.27.37.47.57.67.7Shader Lamps . . . . . . . .Being There . . . . . . . . .iLamps: Mobile Projectors .The Extended Virtual TableThe Virtual Showcase . . .The HoloStation . . . . . .Augmented Paintings . . . .Smart Projectors . . . . . .Summary and Discussion .9 The Future9.19.29.3.1501521521631661661742062138 Examples of Spatial AR 14320321Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321Supporting Elements . . . . . . . . . . . . . . . . . . . . . . 326Summary and Discussion . . . . . . . . . . . . . . . . . . . 329iiii
iiiiContentsixA Calibration of a Projector (or a Camera)331A.1 Source code for Calibration of a Projector (or a camera) . . 331A.2 Quadric Image Transfer . . . . . . . . . . . . . . . . . . . . 334B OpenGL’s Transformation Pipeline Partially Re-Implemented 337B.1B.2B.3B.4General Definitions . . . .Projection Functions . . .Transformation FunctionsAdditional Functions . . .337337338340Bibliography343Index363iiii
iiiiPrefaceSpatial Augmented Reality is a rapidly emerging field which concerns everyone working in digital art and media who uses any aspects of augmentedreality and is interested in cutting-edge technology of display technologiesand the impact of computer graphics. We believe that a rich pallet of different display technologies, mobile and non-mobile, must be considered andadapted to fit a given application so that one can choose the most efficienttechnology. While this broader view is common in the very establishedarea of virtual reality, it is becoming more accepted in augmented realitywhich has been dominated by research involving mobile devices.This book reflects our research efforts over several years and the materialhas been refined in several courses that we taught at the invitation ofEurographics and ACM SIGGRAPH.Who Should Read This BookIn order for a broad spectrum of readers—system designers, programmers,artists, etc— to profit from the book, we require no particular programmingexperience or mathematical background. However, a general knowledge ofbasic computer graphics techniques, 3D tools, and optics will be useful.The reader will learn about techniques involving both hardware andsoftware to implement spatial augmented reality installations. Many Cgand OpenGL code fragments, together with algorithms, formulas, drawings,and photographs will guide the interested readers who want to experimentwith their own spatial augmented reality installations.By including a number of exemplary displays examples from differentenvironments, such as museums, edutainment settings, research projects,and industrial settings, we want to stimulate our readers to imagine novelxiiiii
iiiixiiPrefaceAR installations and to implement them. Supplementary material can befound at http://www.spatialar.com/.About the CoverThe images at the top of the front cover show a rainbow hologram of adinosaur (Deinonychus) skull (found in North America). It has been augmented with reconstructed soft tissue and artificial shading and occlusioneffects. The soft tissue data, provided by Lawrence M. Witmer of OhioUniversity, were rendered autostereoscopically. A replica of the skull washolographed by Tim Frieb at the Holowood holographic studio in Bamberg,Germany. The hologram was reconstructed by projected digital light thatcould be controlled and synchronized to the rendered graphics. This enabled a seamless integration of interactive graphical elements into opticalholograms.The image at the bottom show an example of Shader Lamps: an augmentation of a white wooden model of the Taj Mahal with two projectors. The wooden model was built by Linda Welch, George Spindler andMarty Spindler in the late 1970s. In 1999, at the University of North Carolina, Greg Welch spray painted the wooden model white and Kok-LimLow scanned it with a robotic arm to create a 3D model. The woodenmodel is shown illuminated with images rendered with real time animationof a sunrise.The art work on the back cover was created by Matthias Hanzlik. Thesketches show early concepts of SAR prototypes. They have all been realized and are described in the book.AcknowledgementsThe material presented in this book would not have been realized withoutthe help and dedication of many students and colleagues. We want tothank them first of all: Gordon Wetzstein, Anselm Grundhöfer, SebastianKnödel, Franz Coriand, Alexander Kleppe, Erich Bruns, Stefanie Zollmann,Tobias Langlotz, Mathias Möhring, Christian Lessig, Sebastian Derkau,Tim Gollub, Andreas Emmerling, Thomas Klemmer, Uwe Hahne, PaulFöckler, Christian Nitschke, Brian Ng, Kok-Lim Low, Wei-Chao Chen,Michael Brown, Aditi Majumder, Matt Cutts, Deepak Bandyopadhyay,Thomas Willwacher, Srinivas Rao, Yao Wang, and Johnny Lee.iiii
iiiiPrefacexiiiWe also want to thank all colleagues in the field who provided additional image material: Vincent Lepetit, Pascal Fua, Simon Gibson, KiyoshiKiyokawa, Hong Hua, Susumu Tachi, Daniel Wagner, Dieter Schmalstieg,George Stetten, Tetsuro Ogi, Barney Kaelin, Manfred Bogen, Kok-LimLow, Claudio Pinhanez, Masahiko Inami, Eric Foxlin, Michael Schnaider,Bernd Schwald, and Chad Dyner.Special thanks go to those institutions and people who made this research possible: Bauhaus-University Weimar, Mitsubishi Electric ResearchLaboratories, Jeroen van Baar, Paul Beardsley, Paul Dietz, Cliff Forlines,Joe Marks, Darren Leigh, Bill Yerazunis, Remo Ziegler, Yoshihiro Ashizaki,Masatoshi Kameyama, Keiichi Shiotani, Fraunhofer Institute for ComputerGraphics, José L. Encarnação, Bodo Urban, Erhard Berndt, University ofNorth Carolina at Chapel Hill, Henry Fuchs, Greg Welch, Herman Towles,Fraunhofer Center for Research in Computer Graphics, Miguel Encarnação,Bert Herzog, David Zeltzer, Deutsche Forschungsgemeinschaft, and European Union.We thank the Institution of Electronics & Electrical Engineers (IEEE),the Association for Computing Machinery (ACM), Elsevier, MIT Press,Springer-Verlag, and Blackwell Publishing for granting permission to republish some of the material used in this book.Many thanks to Thomas Zeidler and to Alice and Klaus Peters forhelping us manage the process of writing this book alongside our everydayduties.Oliver BimberWeimar April 2005Ramesh RaskarCambridge, MA April 2005iiii
iiii1A Brief Introduction toAugmented RealityLike Virtual Reality (VR), Augmented Reality (AR) is becoming an emerging edutainment platform for museums. Many artists have started usingthis technology in semi-permanent exhibitions. Industrial use of augmentedreality is also on the rise. Some of these efforts are, however, limited to using off-the-shelf head-worn displays. New, application-specific alternativedisplay approaches pave the way towards flexibility, higher efficiency, andnew applications for augmented reality in many non-mobile applicationdomains. Novel approaches have taken augmented reality beyond traditional eye-worn or hand-held displays, enabling new application areas formuseums, edutainment, research, industry, and the art community. Thisbook discusses spatial augmented reality (SAR) approaches that exploitlarge optical elements and video-projectors, as well as interactive rendering algorithms, calibration techniques, and display examples. It provides acomprehensive overview with detailed mathematics equations and formulas, code fragments, and implementation instructions that enable interestedreaders to realize spatial AR displays by themselves.This chapter will give a brief and general introduction into augmentedreality and its current research challenges. It also outlines the remainingchapters of the book.1.1What is Augmented RealityThe terms virtual reality and cyberspace have become very popular outsidethe research community within the last two decades. Science fiction movies,such as Star Trek, have not only brought this concept to the public, but1iiii
iiii21. A Brief Introduction to Augmented Realityhave also influenced the research community more than they are willing toadmit. Most of us associate these terms with the technological possibilityto dive into a completely synthetic, computer-generated world—sometimesreferred to as a virtual environment. In a virtual environment our senses,such as vision, hearing, haptics, smell, etc., are controlled by a computerwhile our actions influence the produced stimuli. Star Trek ’s Holodeck isprobably one of the most popular examples. Although some bits and piecesof the Holodeck have been realized today, most of it is still science fiction.So what is augmented reality then? As is the case for virtual reality,several formal definitions and classifications for augmented reality exist(e.g., [109, 110]). Some define AR as a special case of VR; others arguethat AR is a more general concept and see VR as a special case of AR.We do not want to make a formal definition here, but rather leave it tothe reader to philosophize on their own. The fact is that in contrast totraditional VR, in AR the real environment is not completely suppressed;instead it plays a dominant role. Rather than immersing a person into acompletely synthetic world, AR attempts to embed synthetic supplementsinto the real environment (or into a live video of the real environment).This leads to a fundamental problem: a real environment is much moredifficult to control than a completely synthetic one. Figure 1.1 shows someexamples of augmented reality applications.As stated previously, augmented reality means to integrate synthetic information into the real environment. With this statement in mind, woulda TV screen playing a cartoon movie, or a radio playing music, then bean AR display? Most of us would say no—but why not? Obviously, thereis more to it. The augmented information has to have a much strongerlink to the real environment. This link is mostly a spatial relation betweenthe augmentations and the real environment. We call this link registration. R2-D2’s spatial projection of Princess Leia in Star Wars would bea popular science fiction example for augmented reality. Some technological approaches that mimic a holographic-like spatial projection, like theHolodeck, do exist today. But once again, the technical implementation asshown in Star Wars still remains a Hollywood illusion.Some say that Ivan Sutherland established the theoretical foundationsof virtual reality in 1965, describing what in his opinion would be theultimate display [182]:The ultimate display would, of course, be a room within whichthe computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Hand-iiii
iiii1.1. What is Augmented Reality3(a)(b)(c)(d)Figure 1.1. Example of augmented reality applications. The glasses, mustache,dragons, and fighter figure are synthetic: (a) and (b) augmenting a video of thereal environment; (c) and (d) augmenting the real environment optically. (Images: (a) courtesy of Vincent Lepetit, EPFL [87]; (b) courtesy of Simon Gibsonc University of Manchester 2005; (c) and (d)[55], Advanced Interfaces Group prototypes implemented by the Barhaus-University Weimar.)cuffs displayed in such a room would be confining, and a bulletdisplayed in such a room would be fatal. With appropriateprogramming, such a display could literally be the Wonderlandinto which Alice walked.However, technical virtual reality display solutions were proposed muchearlier. In the late 1950s, for instance, a young cinematographer namedMort Heilig invented the Sensorama simulator, which was a one-persondemo unit that combined 3D movies, stereo sound, mechanical vibrations,fan-blown air, and aromas. Stereoscopy even dates back to 1832 whenCharles Wheatstone invented the stereoscopic viewer .Then why did Sutherland’s suggestions lay the foundation for virtualreality? In contrast to existing systems, he stressed that the user of such aniiii
iiii41. A Brief Introduction to Augmented Realityultimate display should be able to interact with the virtual environment.This led him to the development of the first functioning Head-MountedDisplay (HMD) [183], which was also the birth of augmented reality. Heused half-silvered mirrors as optical combiners that allowed the user tosee both the computer-generated images reflected from cathode ray tubes(CRTs) and objects in the room, simultaneously. In addition, he usedmechanical and ultrasonic head position sensors to measure the position ofthe user’s head. This ensured a correct registration of the real environmentand the graphical overlays.The interested reader is referred to several surveys [4, 5] and Web sites[3, 193] of augmented reality projects and achievements. Section 1.2 givesa brief overview of today’s technical challenges for augmented reality. It isbeyond the scope of this book to discuss these challenges in great detail.1.2 Today’s ChallengesAs mentioned previously, a correct and consistent registration between synthetic augmentations (usually three-dimensional graphical elements) andthe real environment is one of the most important tasks for augmentedreality. For example, to achieve this for a moving user requires the systemto continuously determine the user’s position within the environment.Thus the tracking and registration problem is one of the most fundamental challenges in AR research today. The precise, fast, and robusttracking of the observer, as well as the real and virtual objects within theenvironment, is critical for convincing AR applications. In general, we candifferentiate between outside-in and inside-out tracking if absolute tracking within a global coordinate system has to be achieved. The first type,outside-in, refers to systems that apply fixed sensors within the environment that track emitters on moving targets. The second type, inside-out,uses sensors that are attached to moving targets. These sensors are able todetermine their positions relative to fixed mounted emitters in the environment. Usually these two tracking types are employed to classify camerabased approaches only—but they are well suited to describe other trackingtechnologies as well.After mechanical and electromagnetic tracking, optical tracking becamevery popular. While infrared solutions can achieve a high precision anda high tracking speed, marker-based tracking, using conventional cameras,represent a low-cost option. Tracking solutions that do not require artificialmarkers, called markerless tracking, remains the most challenging, and atiiii
iiii1.2. Today’s Challenges5the same time, the most promising tracking solution for future augmentedreality applications. Figure 1.1(a) shows an example of a markerless facetracker.Much research effort is spent to improve performance, precision, robustness, and affordability of tracking systems. High-quality tracking withinlarge environments, such as the outdoors, is still very difficult to achieveeven with today’s technology, such as a Global Positioning System (GPS) incombination with relative measuring devices like gyroscopes and accelerometers. A general survey on different tracking technology [164] can be usedfor additional reading.Besides tracking, display technology is another basic building block foraugmented reality. As mentioned previously, head-mounted displays arethe dominant display technology for AR applications today. However, theystill suffer from optical (e.g., limited field of view and fixed focus), technical(e.g., limited resolution and unstable image registration relative to eyes)and human-factor (e.g., weight and size) limitations. The reas
Augmented Reality Like Virtual Reality (VR), Augmented Reality (AR) is becoming an emerg-ing edutainment platform for museums. Many artists have started using thistechnologyinsemi-permanentexhibitions. Industrialuseofaugmented reality is also on the rise. Some of these efforts are, however, limited to us-ing off-the-shelf head-worn displays.
pembelajaran augmented reality dan kelompok siswa yang diajar dengan menggunakan media pembelajaran virtual reality sebesar 5.71. Penelitian ini menunjukkan bahwa ada peningkatan hasil belajar dengan menggunakan media virtual reality dan augmented reality, serta terdapat perbedaan efektivitas antara media virtual reality dan augmented reality.
Spatial augmented reality (SAR) is an emerging paradigm that dif-fers from its origin, the traditional augmented reality (AR), in many regards. While traditional AR is a well-studied field of research, the characteristic features of SAR and their implications on the percep-tion of spatial augmented environments have not been analyzed so far.
Keywords: spatial knowledge; augmented reality; driving; head-up display . Effects of Augmented Reality Head-up Display Graphics' Perceptual Form on Driver Spatial Knowledge Acquisition NAYARA DE OLIVEIRA FARIA ABSTRACT In this study, we investigated whether modifying augmented reality head-up display (AR HUD) graphics' perceptual form .
virtual reality reality augmented reality augmented virtuality mixed reality real environment virtual environment alex olwal course notes mixed reality 9 augmented reality: definition [Azuma 1997; Azuma, Baillot, Behringer, Feiner, Julier & MacIntyre 2001] 1) real virtual objects in real environment 2) runs interactively and in realtime
alternative reality market. The Alternative Reality Landscape Virtual Reality Augmented Reality Mixed Reality What it Does Changes reality by placing the user in a 360-degree imaginary world. Visible world is overlaid with digital content. Like AR, but virtual objects are integrated into and respond to visible surroundings. Where it Stands
Augmented Sound Reality (2002) Wearable interface for placing spatial audio cue Virtual icons representing audio cues 3D stylus for direct manipulation of sound sources Viewing on stereo video see-through HMD Spatial audio playback Dobler, D., Haller, M., & Stampfl, P. (2002). ASR: augmented sound reality. In ACM SIGGRAPH
Figure 1: Photos of the three spatial augmented reality demonstrations. ABSTRACT In this demonstration we showcase a new spatial augmented reality device (interactive projection) with three applications: education and experimentation of color models, map exploration for visually impaired people and scientific vulgarization of machine learning.
Accounting information from several branches can be merged, making decision-making easy and fast. End of Chapter Questions 1 Anti-virus software, complicated passwords. 2 Email, cloud. 3 You can save your work, easy to send to other people, calculations and templates are already there for you to use. 4 Hacking, failure in technology – power cut, some software is expensive. Exam Practice 1B .
