Scaling Wearable Cognitive Assistance
Scaling Wearable Cognitive AssistanceJunjue WangCMU-CS-20-107May 2020School of Computer ScienceCarnegie Mellon UniversityPittsburgh, PA 15213Thesis Committee:Mahadev Satyanarayanan (Satya) (Chair), Carnegie Mellon UniversityDaniel Siewiorek, Carnegie Mellon UniversityMartial Hebert, Carnegie Mellon UniversityRoberta Klatzky, Carnegie Mellon UniversityPadmanabhan Pillai, Intel LabsSubmitted in partial fulfillment of the requirementsfor the degree of Doctor of Philosophy.Copyright c 2020 Junjue WangThis research was supported by the National Science Foundation (NSF) under grant number CNS-1518865. Additional support was provided by Intel, Vodafone, Deutsche Telekom, Verizon, Crown Castle, Seagate, VMware,MobiledgeX, InterDigital, and the Conklin Kistler family fund.
Keywords: Wearable Cognitive Assistance, Scalability, Adaptation, Wearable Computing,Augmented Reality, Edge Computing, Cloudlet
To the future of machines.
iv
AbstractIt has been a long endeavour to augment human cognition with machine intelligence. Recently, a new genre of applications, named Wearable Cognitive Assistance,has advanced the boundaries of augmented cognition. These applications continuously process data from body-worn sensors and provide just-in-time guidance tohelp a user complete a specific task. While previous research has demonstrated thetechnical feasibility of wearable cognitive assistants, this dissertation addresses theproblem of scalability. We identify two critical challenges to the widespread deployment of these applications to be 1) the need to operate cloudlets and wirelessnetwork at low utilization to achieve acceptable end-to-end latency 2) the level ofspecialized skills and the long development time needed to create new applications.To address these challenges, we first design and evaluate adaptation-centric optimizations that reduce resource consumption and improve resource management incontentious systems while maintaining acceptable end-to-end latency. We then propose and implement a new prototyping methodology and a suite of developmenttools to lower the barrier of application development.
vi
AcknowledgmentsI have been extremely fortunate to be inspired and supported by a group of talented and caring individuals throughout my PhD research. This dissertation wouldnot be possible without them.I am deeply indebted to my advisor Mahadev Satyanarayanan (Satya), who playeda paramount role in guiding me through this dissertation research. Satya taught mehow to design, implement, and evaluate mobile and distributed systems. His knowledge and encouragement have been a constant source of inspiration for me to workharder and produce more impactful system research. In addition to an excellentacademic advisor, Satya has also been a visionary mentor whom I look up to. Ithas been invaluable for me to observe and learn from his leadership in shaping newtechnologies and bringing people together for a shared vision. Another person whohas inspired and helped me tremendously is Padmanabhan (Babu) Pillai. He is anexemplary researcher with broad and deep knowledge, kindness, and patience. Inaddition, I would like to thank my thesis committee members, Daniel Siewiorek,Martial Hebert, and Roberta (Bobby) Klatzky. As this dissertation is at the intersection of multiple fields, they have given me many insightful feedback and guidance onhuman-computer interaction and computer vision. Their comments from our weeklymeetings have been vital for me to stay on the correct course.Brilliant and kind individuals in Satya’s research group have provided many rapport and joy during my PhD study. In addition, many ideas and experimental resultspresented in this dissertation come from close collaborative work with them. Theyhave also provided many system infrastructure support to perform the experiments.I cannot dream of a better team and environment to be in for a PhD student. My dayto-day interactions with them helped me grow as a better researcher and a criticalthinker. Especially, I would like to give my appreciation to, in an alphabetical order, Yoshihisa Abe, Brandon Amos, Jim Blakley, Zhuo Chen, Tom Eiszler, Ziqiang(Edmond) Feng, Shilpa George, Kiryong Ha, Jan Harkes, Wenlu Hu, Roger Iyengar,Natalie Janosik, and Haithem Turki. I also want to thank Chase Klingensmith for hisoutstanding administrative assistance for the group.My PhD research has also benefited significantly from the larger research community. I was fortunate enough to collaborate with many exceptional academic researchers and industry professionals in the past few years. Thank you all for theinspiration you have given me. In particular, thank you, Pauline Anthonysamy,Mihir Bala, Richard Buchler, Kevin Christensen, Anupam Das, Nigel Davies, Debidatta Dwibedi, Khalid Elgazzar, Wei Gao, Ying Gao, James Gross, Alex Guerrero,Guenter Klas, Zico Kolter, Michael Kozuch, Hongkun Leng, Grace Lewis, Tan Li,
Haodong Liu, Yuqi Liu, Lin Ma, Mateusz Mikusz, Manuel Olguín, Sai Teja Peddinti, Truong An Pham, Norman Sadeh, Rolf Schuster, Asim Smailagic, Nihar B.Shah, Nina Taft, Joseph Wang, Guanhang Wu, Yu Xiao, Shao-Wen Yang, Canbo(Albert) Ye, and Siyan Zhao.My beloved family and friends have provided enormous understanding, sympathy, and support throughout my PhD study. I could not have finished this thesisresearch without knowing they are on my back. In particular, I have always felt extremely lucky and gracious to be born to the most attentive, considerate, and wiseparents in the world. They instilled the value of perseverance and compassion earlyin my life through examples. Without their trust and encouragement, I would neverdream of completing a doctorate degree halfway around the globe at one of the finestcomputer science institutions in the world. Furthermore, as lucky as a man could be,I met my dearest partner, Han, while pursuing my PhD. Since then, she has beenmy closest friend and most cherished counselor who has provided an immeasurableamount of love, strength, and comfort. I am able to become a better version of myself because of her. As much as this dissertation is written for the future of machines,it is the unwritten words about people that will be treasured for the rest of my life.viii
Contents12Introduction11.1Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31.2Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3Background52.1Edge Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.2Gabriel Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.3Example Gabriel Applications . . . . . . . . . . . . . . . . . . . . . . . . . . .92.432.3.1RibLoc Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3.2DiskTray Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Application Latency Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Application-Agnostic Techniques to Reduce Network Transmission3.13.23.315Video Processing on Mobile Devices . . . . . . . . . . . . . . . . . . . . . . . . 163.1.1Computation Power on Tier-3 Devices . . . . . . . . . . . . . . . . . . . 163.1.2Result Latency, Offloading and Scalability . . . . . . . . . . . . . . . . . 18Baseline Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2.1Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2.2Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2.3Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21EarlyDiscard Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3.1Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3.2Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.3.3Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24ix
3.443.3.4Use of Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.3.5Effects of Video Encoding . . . . . . . . . . . . . . . . . . . . . . . . . 29Just-In-Time-Learning (JITL) Strategy To Improve Early Discard . . . . . . . . . 303.4.1JITL Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 333.4.2Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.5Applying EarlyDiscard and JITL to Wearable Cognitive Assistants . . . . . . . . 333.6Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.7Chapter Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 37Application-Aware Techniques to Reduce Offered Load4.1Adaptation Architecture and Strategy . . . . . . . . . . . . . . . . . . . . . . . 404.2System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.3Adaptation Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.4Leveraging Application Characteristics . . . . . . . . . . . . . . . . . . . . . . . 424.4.1539Adaptation-Relevant Taxonomy . . . . . . . . . . . . . . . . . . . . . . 444.5Adaptive Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.6IMU-based Approaches: Passive Phase Suppression . . . . . . . . . . . . . . . . 474.7Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.8Chapter Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 50Cloudlet Resource Management for Graceful Degradation of Service5.15.2System Model and Application Profiles . . . . . . . . . . . . . . . . . . . . . . 515.1.1System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525.1.2Application Utility and Profiles . . . . . . . . . . . . . . . . . . . . . . 53Profiling-based Resource Allocation . . . . . . . . . . . . . . . . . . . . . . . . 555.2.15.351Maximizing Overall System Utility . . . . . . . . . . . . . . . . . . . . 55Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.3.1Effectiveness of Workload Reduction . . . . . . . . . . . . . . . . . . . 575.3.2Effectiveness of Resource Allocation . . . . . . . . . . . . . . . . . . . 575.3.3Effects on Guidance Latency . . . . . . . . . . . . . . . . . . . . . . . . 595.4Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.5Chapter Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 64x
6Wearable Cognitive Assistance Development Tools6.1Ad Hoc WCA Development Process . . . . . . . . . . . . . . . . . . . . . . . . 666.2A Fast Prototyping Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 676.36.47656.2.1Objects as the Universal Building Blocks . . . . . . . . . . . . . . . . . 676.2.2Finite State Machine (FSM) as Application Representation . . . . . . . . 70OpenTPOD: Open Toolkit for Painless Object Detection . . . . . . . . . . . . . 726.3.1Creating a Object Detector with OpenTPOD . . . . . . . . . . . . . . . 726.3.2OpenTPOD Case Study With the COOKING Application . . . . . . . . 756.3.3OpenTPOD Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 75OpenWorkflow: FSM Workflow Authoring Tools . . . . . . . . . . . . . . . . . 776.4.1OpenWorkflow Web GUI . . . . . . . . . . . . . . . . . . . . . . . . . . 776.4.2OpenWorkflow Python Library . . . . . . . . . . . . . . . . . . . . . . . 786.4.3OpenWorkflow Binary Format . . . . . . . . . . . . . . . . . . . . . . . 786.5Lessons for Practitioners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786.6Chapter Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 80Conclusion and Future Work817.1Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817.2Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 827.2.1Advanced Computer Vision For Wearable Cognitive Assistance . . . . . 827.2.2Fine-grained Online Resource Management . . . . . . . . . . . . . . . . 827.2.3WCA Synthesis from Example Videos . . . . . . . . . . . . . . . . . . . 82Bibliography83xi
xii
List of Figures2.1Tiered Model of Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . .62.2CDF of pinging RTTs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72.3FACE Response Time over LTE . . . . . . . . . . . . . . . . . . . . . . . . . .72.4Gabriel Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.5RibLoc Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92.6RibLoc Assistant Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.7Assembled DiskTray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.8Small Parts in DiskTray WCA . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.1Early Discard on Tier-3 Devices . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2Tiling and DNN Fine Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3Speed-Accuracy Trade-off of Tiling . . . . . . . . . . . . . . . . . . . . . . . . 243.4Bandwidth Breakdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.5Event Recall at Different Sampling Intervals . . . . . . . . . . . . . . . . . . . . 273.6Sample with Early Discard. Note the log scale on y-axis. . . . . . . . . . . . . . 283.7JITL Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.8JITL Fraction of Frames under Different Event Recall . . . . . . . . . . . . . . . 313.9Example Images from a Lego Assembly Video . . . . . . . . . . . . . . . . . . 343.10 Example Images from LEGO Dataset . . . . . . . . . . . . . . . . . . . . . . . 343.11 EarlyDiscard Filter Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . 353.12 JITL Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.1Adaptation Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.2Design Space of WCA Applications . . . . . . . . . . . . . . . . . . . . . . . . 454.3Dynamic Sampling Rate for LEGO . . . . . . . . . . . . . . . . . . . . . . . . . 46xiii
4.4Adaptive Sampling Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.5Accuracy of IMU-based Frame Suppression . . . . . . . . . . . . . . . . . . . . 495.1LEGO Processing DAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525.2Resource Allocation System Model . . . . . . . . . . . . . . . . . . . . . . . . 535.3FACE Application Utility and Profile . . . . . . . . . . . . . . . . . . . . . . . . 545.4POOL Application Utility and Profile . . . . . . . . . . . . . . . . . . . . . . . 545.5Iterative Allocation Algorithm to Maximize Overall System Utility . . . . . . . . 565.6Effects of Workload Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.7Total Utility with Increasing Contention . . . . . . . . . . . . . . . . . . . . . . 595.8Normalized 90%-tile Response Latency . . . . . . . . . . . . . . . . . . . . . . 605.9Average Processed Frames Per Second Per Client . . . . . . . . . . . . . . . . . 615.10 Normalized 90%-tile Response Latency . . . . . . . . . . . . . . . . . . . . . . 625.11 Processed Frames Per Second Per Application . . . . . . . . . . . . . . . . . . . 625.12 Fraction of Cloudlet Processing Allocated . . . . . . . . . . . . . . . . . . . . . 635.13 Guidance Latency Compared to Loose Latency Bound . . . . . . . . . . . . . . 636.1Ad Hoc Development Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . 666.2A Frame Seen by the PING PONG Assistant . . . . . . . . . . . . . . . . . . . . 686.3A Frame Seen by the LEGO Assistant . . . . . . . . . . . . . . . . . . . . . . . 696.4Example WCA FSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716.5OpenTPOD Training Images Examples . . . . . . . . . . . . . . . . . . . . . . 726.6OpenTPOD Video Management GUI . . . . . . . . . . . . . . . . . . . . . . . . 736.7OpenTPOD Integration of an External Labeling Tool . . . . . . . . . . . . . . . 746.8OpenTPOD Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.9OpenTPOD Provider Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.10 OpenWorkflow Web GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.11 OpenWorkflow FSM Binary Format . . . . . . . . . . . . . . . . . . . . . . . . 79xiv
List of Tables2.1Prototype Wearable Cognitive Assistance Applications . . . . . . . . . . . . . . 102.2Application Latency Bounds (in milliseconds) . . . . . . . . . . . . . . . . . . . 133.1Deep Neural Network Inference Speed on Tier-3 Devices . . . . . . . . . . . . . 173.2Benchmark Suite of Drone Video Traces . . . . . . . . . . . . . . . . . . . . . . 203.3Baseline Object Detection Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 203.4Recall, Event Latency and Bandwidth at Cutoff Threshold 0.5 . . . . . . . . . . 263.5Test Dataset Size With Different Encoding Settings . . . . . . . . . . . . . . . . 284.1Application characteristics and corresponding applicable techniques to reduce load 434.2Adaptive Sampling Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.3Effectiveness of IMU-based Frame Suppression . . . . . . . . . . . . . . . . . . 505.1Resource Allocation Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . 58xv
xvi
Chapter 1IntroductionIt has been a long endeavour to augment human cognition with machine intelligence. As earlyas in 1945, Vannevar Bush envisioned a machine Memex that provides "enlarged intimate supplement to one’s memory" and can be "consulted with exceeding speed and flexibility" in theseminal article As We May Think [14]. This vision has been brought closer to reality by years ofresearch in computing hardware, artificial intelligence, and human-computer interaction. In late90s to early 2000s, Smailagic et al. [102, 103, 104] created prototypes of wearable computersto assist cognitive tasks. For example, they displayed inspection manuals in a head-up screen tofacilitate aircraft maintenance. Around the same time, Loomis et al. [63, 64] explored using computers carried in a backpack to provide auditory cues in order to help the blind navigate. Davis etal. [18, 23] developed a context-sensitive intelligent visitor guide leveraging hand-portable multimedia systems. While these research works pioneered cognitive assistance and its related fields,their robustness and functionality were limited by the supporting technologies of their time.More recently, as the underlying technologies experience significant advancement, a newgenre of applications, Wearable Cognitive Assistance (WCA) [16, 35], has emerged that pushesthe boundaries of augmented cognition. WCA applications continuously process data from bodyworn sensors and provide just-in-time guidance to help a user complete a specific task. For example, an IKEA Lamp assistant [16] has been built to assist the assembly of a table lamp. To usethe application, a user wears a head-mounted smart glass that continuously captures her actionsand surroundings from a first-person viewpoint. In real-time, the camera stream is analyzed toidentify the state of the assembly. Audiovisual instructions are generated based on the detectedstate. The instructions either demonstrate a subsequent procedure or alert and correct a mistake.Since its conceptualization in 2004 [92], WCA has attracted much research interest from bothacademia and industry. The building blocks for its vision came into place by 2014, enabling thefirst implementation of this concept in Gabriel [35]. In 2017, Chen et al [17] described a numberof applications of this genre, quantified their latency requirements, and profiled the end-to-endlatencies of their implementations. In late 2017, SEMATECH and DARPA jointly funded 27.5million of research
human-computer interaction and computer vision. Their comments from our weekly meetings have been vital for me to stay on the correct course. Brilliant and kind individuals in Satya’s research group have
Wearable Technology Definition Wearable technology (also called wearable gadgets) is a category of technology devices that can be worn by a consumer and often include tracking information related to health and fitness. Other wearable tech gadgets include devices that have small motion sensors to take photos and sync with your mobile devices.
Measurement and Scaling Techniques Measurement In Research In our daily life we are said to measure when we use some yardstick to determine weight, height, or some other feature of a physical object. We also measure when we judge how well we like a song, a File Size: 216KBPage Count: 23Explore further(PDF) Measurement and Scaling Techniques in Research .www.researchgate.netMeasurement & Scaling Techniques PDF Level Of .www.scribd.comMeasurement and Scaling Techniqueswww.slideshare.netMeasurement and scaling techniques - SlideSharewww.slideshare.netMeasurement & scaling ,Research methodologywww.slideshare.netRecommended to you b
AWS Auto Scaling lets you use scaling plans to configure a set of instructions for scaling your resources. If you work with AWS CloudFormation or add tags to scalable resources, you can set up scaling plans for different sets of resources, per application. The AWS Auto Scaling console provides recommendations for
Memory Scaling is Dead, Long Live Memory Scaling Le Memoire Scaling est mort, vive le Memoire Scaling! . The Gap in Memory Hierarchy Main memory system must scale to maintain performance growth 21 3 227 11 13 2215 219 23 Typical access latency in processor cycles (@ 4 GHz) L1(SRAM) EDRAM DRAM HDD 25 29 217 221 Flash
strategy. It provides an overview of scaling frameworks and models, together with a set of case studies of scaling strategies applied by organisations within and outside the YBI network. Different models for scaling and replication are introduced by means of frameworks developed by innovation and scaling experts Nesta and Spring
The scaling plans included growth goals, plans for achieving the goals, resources to be invested in scaling, planned actions to achieve goals, plans for . Scaling Programs and Growing Impact with the Social Innovation Fund Issue Brief #7: Scaling Programs and Growing Impact with the Social Innovation Fund .
Gustafson’s law [5], and Sun-Ni’s law [6], are no longer ad- . over whether scaling-out is indeed better than scaling-up or not [15]. Second, as the existing scaling laws are increasingly . non-linear, monotonic or peaked) major scaling properties
The Allen Cognitive Level Screen (ACLS) can help you identify the Allen Cognitive Levels of clients with Alzheimer’s disease, dementia, and other cognitive disabilities. Also referred to as the leather lacing tool, this cognitive assessment tool measures global cognitive processing capaci
