Proposal(Updated) For AGL HMI-Framework

Proposal(Updated) for AGL HMI-FrameworkAGL All-Member Meeting @ TOKYOFebruary 2018Toshikazu Oiwatoshikazu ohiwa@mail.toyota.co.jpTOYOTA MOTOR CORPORATION TOYOTA MOTOR CORPORATION All Rights Reserved.1

Who is Oiwa?Ø Software engineer, expert in in-vehicle infotainment.Ø Have been developing software for in-vehicle infotainmentsystem such as apps, services since 1994.Ø In charge of HMI-Framework. TOYOTA MOTOR CORPORATION All Rights Reserved.2

2017 AGL Development Plan of TOYOTAALS JunHMI-FWWindow Manager Proto.Sound Manager Proto.@ UI & Graphics EGAMM OctPresentationwith DEMODRESDENVehicleI/FReferenceHardwareALS DEMOJanHMI-FWon EEGERMANYCAN SDK / Ethernet Proto.@ Navigation EG@ Reference Hardware System Architecture EG TOYOTA MOTOR CORPORATION All Rights Reserved.CESVehicle I/Fon EERequirementSpec3

Table of ContentsØ Background of HMI-FrameworkØ Proposal for HMI-Framework OEM can choose GUI OEM can choose HMI-Manager HMI-Framework can arbitrate HMI ResourcesØ DemonstrationØ ScheduleØ Conclusion TOYOTA MOTOR CORPORATION All Rights Reserved.4

Backgroundof HMI-Framework TOYOTA MOTOR CORPORATION All Rights Reserved.5

Background for in-vehicle HMI #1Ø Needs for HMI has increased dramaticallyExpectation to HMI is getting higher and higherbecause of better user experiences(UX) by smartphone,especially for richer graphics and easier interaction by using touch-panel. Users expect HMI at the same level as smartphoneto other consumer products.Such expectation shall be the same for in-vehicle system.However,HMI of present in-vehicle systems is behind. TOYOTA MOTOR CORPORATION All Rights Reserved.6

Background for in-vehicle HMI #2Ø Apps compete to output information in in-vehicle system HMIRunning at the same time in in-vehicle system.Compete for getting resources(screen,speaker) to output information.Wants to display mapWants to display control of HVACcan’t appropriately place on screen Lack of proper management results in driver distraction.In addition, resources management would be differentby system configuration such as low-end systems or high-end systems. TOYOTA MOTOR CORPORATION All Rights Reserved.7

Requirements for HMI-Framework①OEM can choose GUI②OEM can choose HMI-Manager③HMI-FW can arbitrate HMI resourcesWhy requirement①？Ø Comfortable GUI ”GUI for better UX” (background #1) Intuitively easy to understand (３D) Animation (Graphic Effects)Wants to be able to use the most suitable GUI.Why requirement②③？Ø Safe HMI(background #2) Manage resources(Window-Resources/Sound-Resources) Arbitration of Window-Resources/Sound-ResourcesWants to be able to use the most suitable software to system.Wants to be able to arbitrate resources. TOYOTA MOTOR CORPORATION All Rights Reserved.8

Proposal of HMI-Framework・OEM can choose GUI・OEM can choose HMI-Manager・HMI-FW can arbitrate HMI resources TOYOTA MOTOR CORPORATION All Rights Reserved.9

Requirement① OEM can choose GUIlegendNo change in API which Apps refer toApps can use API of each GUI-lib.Proposal pointEach GUI-lib has layer to adapt to PFCan have many different GUI-libs without change of PF.App LayerHMI AppsGUI-libraryproprietarysoftwareGUIHMI-FWApp FW LayerGUI-lib PAL(*)For AGL PFHMI-Manager(*)Service Layer TOYOTA MOTOR CORPORATION All Rights Reserved.*PF Adaptation Layer* Generic name ofcomponent to manage HMIHMI-Server (e.g. Weston, OpenGL/EG, ALSA)10

Requirement② OEM can choose HMI-Manager(*)* Generic name of component to manage HMI* Decides the optimum layout and controls(e.g. screen), based request from apps. No change in API which Apps and GUI-libs refer toApps and GUI-libs are available to use unique API of HMI-Manager.legend. Each HMI-Manager has interface to adapt to service layerCan have different HMI-Manager without change in PF.App LayerProposal pointHMI AppsGUI-libGUI-lib PALFor AGL PFHMI-FWApp FW LayerWindowManagerService Layer TOYOTA MOTOR CORPORATION All Rights Reserved.HMI-ManagerSoundManagerOthers SoundManagerHMI-Server (e.g. Weston, OpenGL/EG, ALSA)11

legendComponents for Requirement①②AppLayerProposal pointHMI AppsQt APIHTML APIEB-Guide APIProprietary GUI APIproprietary GUIApp FWLayerGUI-lib COREGUI-lib COREGUI-lib COREGUI-lib COREGUI-lib PALFor AGL PFGUI-lib PALFor AGL PFGUI-lib PALFor AGL PFGUI-lib PALFor AGL r API(common)WindowManagerSoundManagerOther SoundManagerHMIManagere.g. AAAAI/FFor AGL PFI/FFor AGL PFServiceLayerWeston TOYOTA MOTOR CORPORATION All Rights Reserved.OpenGL/EGLI/FFor AGL PFALSA12

Requirement③ HMI-FW can arbitrate HMI resources１．Judges use of resources for request from Apps(Policy Manager)Upon request (Event) from Apps,Policy Manager decides which App can allocate resources.Has a rule of arbitration as Policy DBOEM can easily change Policy DB without changing HMI Manager.Policy DB is described in a format such as XML.HMI AppsRequirementJudgment resultDesigned by developeror generated using toolHMI-ManagerPolicyManager1Policy DB“A”2Policy DB“B” TOYOTA MOTOR CORPORATION All Rights Reserved.State transition tableis made by HMI specific designerat the design process.13

Structure of HMI-Manager including Policy ManagerDecides the optimum screen layout or speaker layoutand controls screen or speaker, based on request from Apps.Window Manager/Sound Manager in HMI manager are the same in structure.[Main components] Policy Manager Resource ManagerPolicyManagerResourcesLayout ManagerManages layout accordingto judgment result byPolicy Manager.MessageSignalingClient ManagerResourceManagerManages resource informationsuch as display, speaker. MessageSignalingServerHMI-ManagerClient ManagerPolicy ux.org/hmiframework General Information “HMI-Framework Architecture” TOYOTA MOTOR CORPORATION All Rights Reserved.14

Demonstration TOYOTA MOTOR CORPORATION All Rights Reserved.15

Structure of demonstration Built software for AGL 5.0 on M3Demonstration’s Apps are based on Apps of CES2018HMI-Framework supports AFBChoose Qt for Req.①Choose Audio Manager(Genivi) for Req.②, Window Manager and Sound Managerare updated Window Policy DB is ZIPC format, Sound Policy DB is Genivi format for Req.③HomeScreenNavi*Native Appassuming navi①Window ManagerPolicyManagerZIPCMedia PlayerPhoneSDL App“Glympse”Qt (GUI-lib)HMI-ManagerSound ManagerPolicyManager(Genivi)Policy DBZIPCHMI-FWPolicy DBGeniviAGL for ver5.0R-Car M3 TOYOTA MOTOR CORPORATION All Rights Reserved.16

Demo① Changing the behavior of HMIØ Use case:Changes the behavior of HMI according to destination or grade.Policy AScreen layout (1 App: 1 screen)PushHVAC buttonPolicy BPushMedia PlayerbuttonScreen layout (2 App: 1 screen)Step1:screen changesaccording to Policy A.Step2:Changed from Policy A to Policy B.And restart the system.PushHVAC buttonPushMedia PlayerbuttonStep3:screen changesaccording to Policy B.In this demonstration, “simple-egl” is used instead of “map” of native app. TOYOTA MOTOR CORPORATION All Rights Reserved.17

Demo① Changing the behavior of HMIØ Use case:Changes the behavior of HMI according to destination or grade.Policy AScreen layout (1 App: 1 screen)PushHVAC buttonPolicy BPushMedia PlayerbuttonScreen layout (2 App: 1 screen)Step1:screen changesaccording to Policy A.Step2:Changed from Policy A to Policy B.And restart the system.PushHVAC buttonPushMedia PlayerbuttonStep3:screen changesaccording to Policy B.In this demonstration, “simple-egl” is used instead of “map” of native app. TOYOTA MOTOR CORPORATION All Rights Reserved.18

Demo② Arbitration of HMI ResourcesØ Use caseWhen executing another application during navigation map displayon running, split the display screen and display multiple applications.NEW・Screen transitions mixed application’s normal and half size display・Non-rectangular or rectangular on-screen displayNormal size displayHalf size display TOYOTA MOTOR CORPORATION All Rights Reserved.On-Screen displayScreen transition ofhalf size appScreen transition tonormal size app19

Schedule TOYOTA MOTOR CORPORATION All Rights Reserved.20

Schedule for HMI-Framework developmentReleasePatch UpdatesHMI-FW has already been released in UCB5.0(EE)Screen layout (1 App: 1 screen)Active source changeFeatures DevelopedStabilizePatch UpdatesNew features(Add)Incomingxxx-xxxx-xxxxScreen layout (2App: 1 screen)On-ScreenFeatures DevelopedCenter displayInstrument clusterStabilizeMulti ECU/DisplayPolicy Manager(Update)Input ManagerCANVehicle info.Steering SWAssumed system TOYOTA MOTOR CORPORATION All Rights Reserved.Touchpad21

Conclusion TOYOTA MOTOR CORPORATION All Rights Reserved.22

ConclusionProposed three requirements for HMI-Framework.Ø ①OEM can choose GUI Enables compelling application development.Ø ②OEM can choose HMI-Manager Enables flexible development for OEM needs.Ø ③HMI-Framework can arbitrate HMI resources Each OEM can easily develop each OEM’s specification.By our proposal, we hope that OEM will promote productdevelopment using AGL and that AGL will become more active. TOYOTA MOTOR CORPORATION All Rights Reserved.23

Thank you once again for taking the timeto join today’s presentation.toshikazu ohiwa@mail.toyota.co.jp TOYOTA MOTOR CORPORATION All Rights Reserved.24

No change in API which Apps refer to Apps can use API of each GUI-lib. Each GUI-lib has layer to adapt to PF Can have many different GUI-libs without change of PF. HMI Apps App FW Layer Service Layer AppLayer GUI-library GUI-lib PAL(*) ForAGL PF *PF Adaptation Layer HMI-Server (e.g.