D5.4 User Evaluation App, First Prototype
ViSTA-TV:Video Stream Analytics for Viewers in the TV IndustryFP7 STREP ICT-296126 296126 co-funded by the European CommissionICT-2011-SME-DCL SME Initiative on Digital Content and LanguagesD5.4User Evaluation App, First PrototypeLibby Miller (BBC),Chris Newell (BBC),Dan Nuttall (BBC),Ant Onumonu (BBC),Andrew Wood (BBC)Project start date:Document identifier:Date due:Submission date:June 1st, 2012ViSTA-TV/2013/D5.431 May 201331 May 2013Project eu24 monthsv1.0FinalPU
ViSTA-TV ConsortiumThis document is part of a collaborative research project funded by the FP7 ICT Programme of the Commissionof the European Communities, grant number 296126. The following partners are involved in the project:University of Zurich (UZH) - CoordinatorDynamic and Distributed Information Systems Group (DDIS)Binzmühlstrasse 148050 Zürich, SwitzerlandContact person: Abraham BernsteinE-mail: bernstein@ifi.uzh.chTechniche Universität Dortmund (TUDo)Computer Science VIII: Artificial Intelligence UnitD-44221 Dortmund, GermanyContact person: Katharina MorikE-mail: katharina.morik@cs.uni-dortmund.deRapid-I GmbH (RAPID-I)Stockumer Strasse 47544227 Dortmund, GermanyContact person: Ingo MierswaE-mail: mierswa@rapid-i.comZattoo Europa AG (Zattoo)Eggbühlstrasse 28CH-8050 Zürich, SwitzerlandContact person: Bea KnechtE-mail: beaknecht@me.comVrije Universiteit Amsterdam (VUA)Web & Media Group, Department of Computer Science, Faculty of Sciences (FEW)De Boelelaan 1081aNL-1081 HV Amsterdam, The NetherlandsContact person: Guus SchreiberE-mail: guus.schreiber@vu.nlThe British Broadcasting Corporation (BBC)56 Wood Lane / Centre HouseLondon W12 7SB, United KingdomContact person: Chris NewellE-mail: Chris.Newell@bbc.co.ukCopyright 2012 The ViSTA-TV Consortium
D5.4 User Evaluation App, First Prototype3Executive OverviewThis document describes the process and results of the ‘User Evaluation App’ first prototype. The goal is tocreate a prototype application that uses the results of the rest of the project in a way that can be directlyevaluated with users. As part of the process of deciding what to build, we made three rapid prototypes showingsome different approaches to using the data.‘Heartbeat EPG’ is an audience-facing TV guide showing hotspots of activity - in real time, how many peopleare watching simulcast and on-demand IPTV and Radio.‘Infinite Trailers’ is an audience-facing video application that plays short video clips from programmes and learnswhat you like.‘Radio Dashboard’ is a tool for programme makers to visualise and analyse real-time listening activity.‘Dashboard’ is being developed further within the project as an additional piece of work. One of ‘Infinite Trailers’or ‘Heartbeat EPG’ will be chosen for further development as a second screen application in the second year ofthe project.
4ViSTA-TVContents1 Introduction51.1BBC R&D IRFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51.2Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51.3BBC iPlayer Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61.4Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62 Prototypes72.1Heartbeat EPG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72.2Infinite Trailers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92.3Radio Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112.4User Testing and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123 Conclusions134 Acknowledgements145 Appendix: User Experience Documentation14
D5.4 User Evaluation App, First Prototype15IntroductionThis deliverable is about the first two parts of Task 5.2:User Evaluation App [BBC; D5.2; M7-M18]a) Design and specify a second-screen prototype application. Enable evaluation of the results of the project with end users, using APIs to TV such as the BBC’sUniversal Controller.Show features such as: What’s popular on the BBC now, shows you might like given your user history,time of day, the area you live in, and semantic explanation of recommendations.The BBC will consult with internal stakeholders for appropriate use cases, precise functionality and featuresfor the application, and create wireframes or other suitable user experience documentation for it.b) Build a prototype App. in accordance with the specification developed. the App will provide feedback to WP4.4 to enhance the recommendation process1.1BBC R&D IRFSThe Internet Research and Future Services (IRFS) group within BBC R&D works by prototyping ideas with realdata in order to demonstrate them to internal and external stakeholders and test and evaluate them with endusers. Our philosophy is to prototype and iterate, so the prototypes people see may not be complete or fullyfunctional, but should help us learn whether the idea is successful and what we need to do to build more stableversions.1.2RequirementsOur task was to create a compelling user application from the anonymised BBC IPTV player (‘iPlayer’) logdata, even though the full ViSTA-TV system was not yet in place.There are many different ways we could have approached this. The requirements were: Novelty: the eventual prototype should not currently exist Relevance to the BBC: it should be interesting to the BBC as an organisation Using the data: it should use the eventual outputs of the project Audience facing: it should result in an end user applicationTo generate a broad range of ideas, we held a half-day workshop with participants from inside and outside theBBC. We introduced the data, split into four groups, and used brainstorming techniques to think about thepotential of the data for a user-facing application.Four ideas came out of this process: ‘Heartbeat EPG’ - a heatmap showing what people are watching now‘Infinite Trailers’ - a channel of continually playing clips of available content on iPlayer, which learns whatyou like, presenting you with better suggestions depending on whether you skip or like an item.‘Radio Dashboard’ - a dashboard of real-time radio-related data for the Audio and Music team (the peopleat the BBC responsible for IP Radio) to quickly see any problemsTaking iPlayer to you - a variety of techniques for integrating iPlayer with your online life more closely.Of these, the first three each had a core idea that we could develop into a prototype in a short period of time.We formed a team comprising of a data processing engineer (Chris Newell), a backend engineer (Dan Nuttall),a frontend engineer (Ant Onumonu), a designer (Andrew Wood) and a producer (Libby Miller), and createdthree prototypes over a month-long sprint, using agile methods to coordinate our work.
6ViSTA-TVFigure 1: Heartbeat EPG, Workshop Sketch1.3BBC iPlayer DataiPlayer is the BBC’s IPTV and Radio player. In March 2013 it had 272 million requests [4]. 20 main channelsare available simulcasting with broadcast (9 TV, 11 radio) and 400-500 programmes on-demand.The BBC’s data for ViSTA-TV consists of anonymised log data from which we can derive the number of peoplewatching any simulcast channel or on-demand programme in any given minute. Chris processed this data tocollate statistics for channels, programmes and user platforms available as text and JSON[5] APIs.Additionally, the BBC publishes its schedule and programme information as open data[1], so that at any pointin time we can easily determine the title, description, image and other metadata for a programme.The BBC also has various internal tools available including some radio track listings, a Twitter API, and aninternal BBC tool called Snippets, which has an API for creating clips of programmes. We also had availablea ‘client-side’ recommendation engine [2] we use in an existing prototype recommender system called Sibyl [8].The engine provides a list of available programmes, which is ordered to reflect the preferences of the user, basedon input about programmes they like and/or dislike.1.4SpecificationsThe outputs of the workshop were informal sketches (see Figures 1, 2), and presentations. In each case wehad a one-paragraph description of the application, and notes from the presentations. We had a relativelyunconstrained problem with no specific client or requirements, and so we used the initial descriptions as to seed
D5.4 User Evaluation App, First Prototype7Figure 2: Dashboard, Workshop Sketchdiscussions as a team about what features were feasible with the data available and the time we had to createthem.Throughout the month-long sprint we maintained an ongoing discussion with various stakeholders within theBBC, showing the prototypes to people (including some who had attended the workshop) at an early stage andgetting their feedback. This technology transfer process continues with the resulting prototypes.The specifications in this case were therefore the descriptions (see below), the designs (included in the Appendix) and the prototypes themselves, screenshots of which are included in this document, and which will bedemonstrated at the review.22.1PrototypesHeartbeat EPGIn some ways this was the simplest of all the ideas, but also very powerful: show in real time what people arewatching and listening to on the BBC’s IPTV application iPlayer - both ‘simulcast’ (programmes watched orlistened to online at broadcast time) and ‘catch up’ (programmes watched or listened to after broadcast). Ascreenshot of the simulcast page is shown in Figure 3.During our initial discussions we concluded that it was important to show updates without reloading the page,and this became the core of the prototype. Our designer, Andrew, decided to take a clean, simple ‘list’ approachto the design, and included a way to show the proportion of the audience watching each programme, highlightingthe most popular item.The design pattern was essentially a table with the top twenty results. Andrew made quick sketches anddiscussed them with the team, which was a very quick iterative process due to the short time available. Theteam agreed on the best option and Ant took the design assets and created the Javascript and HTML necessaryto implement them.
8ViSTA-TVFigure 3: Heartbeat EPG, SimulcastThe basic underlying data APIs that we needed for this prototype were already in place, thanks to the priorwork Chris had done to make them available to other partners within the project. In any final version, thesewould be replaced by data coming from the project via Storm [11].The backend needed to be able to handle multiple incoming data streams and to push that data out to clientsas quickly as possible: Node.js [6] was the perfect match for this. The Node.js application polls the data APIfor new data points. When new data is found, we search for applicable metadata using a combination of theBBC Programmes’ API [1] and our own programmes’ Solr-powered datastore. Once the dataset is complete,connected clients - the webpages - receive the new information over WebSockets [12], using Socket.IO[9].Socket.IO has a very straight-forward API, allowing us to pass a simple JSON object to the client-side codeeach time the data changes.The result is two simple, attractive web pages that update in-place every minute. One shows the the proportionof the simulcast audience watching each of the top twenty simulcast channels, and one shows a similar displayfor on-demand programmes.Although it was initially designed as a user-facing application, we have had interest from our colleagues inAudio and Music at the BBC to make it a dashboard within their office, as well as potentially a way for usersto find interesting radio channels and programmes on the website: people often do not trust recommendations(believing them to be promotions), but do trust what others are doing as a guide to what to watch or listen to.
D5.4 User Evaluation App, First Prototype9Figure 4: Heartbeat EPG Design - possible big-screen versionA design for a possible big-screen version is shown in Figure 4.Heartbeat EPG is also a candidate for part or all of the second screen user-facing application to be developedand evaluated in the second part of the project.2.2Infinite TrailersThe idea for this prototype was a continuously playing channel of BBC TV videos available on iPlayer, whichthe user can choose from to play now, or indicate ‘like’ or ‘dislike’ to get better suggestions. The concept islike flicking between channels but with extra intelligence built in to help the user find something interesting towatch quickly.Figure 5: Infinite Trailers - mobile versionThe prototype uses the API from Snippets to get 30 second clips from the first two minutes of the iPlayer
10ViSTA-TVprogrammes. The list of programmes is determined by a collaborative filtering model developed in anotherproject, called ‘Sibyl‘ [8], as this prototype was developed before recommendations were produced from ViSTATV - other recommendations approaches can be swapped in easily. At any one time, about 200-250 programmesmay appear. The model is currently updated daily.The starting point (if the user does not click) is what’s most popular when the model was generated. If theuser clicks ‘add to playlist’ the system counts this as a ‘like’; unless they have already added it to their playlist,clicking on ‘next’ counts as a ‘dislike’. The growing list of likes and dislikes is input to the recommendationsmodel to present theoretically better suggestions the more information it has.The original concept was for the user to flick between programmes until they found something to watchimmediately, but the recommender algorithm we were using (a collaborative filtering model based on iPlayerwatching data) works best with likes and dislikes, so we added in the ‘save for later’ feature to capture thosefrom the user.As a team, we decided to create a mobile application for this prototype: Ant in particular was interested inlearning more about HTML5 video for mobile, but it was also a good point at which to research the optionsfor this to be a second screen application. Mobile video turned out to be more difficult than we had initiallyassumed, as browser vendors on mobile have different interpretations of some aspects of HTML5 - but we stillmanaged to create a prototype that works on Android phones, iPads and as a bonus, in Chrome, and learneda great deal from the process.Andrew’s design starting point with this sprint was a smart phone interface. The concept is a passive experiencefor the user - flicking through trailers - so the design needed to be simple and uncluttered. A simple title screenlabels the content then fades off and the user has the option to move on, or add the trailer to their playlist.Figure 6: Radio Dashboard main screenOnce the front end was in development mode Andrew had some time to look at other options - the egBoxplatform (a two-screen prototyping system) seemed appropriate to the concept - so he mocked up the twoscreens (trailer and playlist) - and would implement responsive design templates if the prototype was developedfurther.Ensuring that each recommendation was backed by a video clip was the key to a good user experience inthis application. The Sibyl recommendations API is polled every few minutes to keep the pool of availableprogrammes up-to-date. As new programmes are found, their identifiers (’PIDs’) are entered into a multi-statesystem, backed by a SQLite database. Initially metadata is applied using similar techniques to the HeartbeatEPG application. With this data, we then query the Snippets API to see if the programme has been capturedby Snippets and so is available for clipping. If this is successful, we schedule a clip to be taken from the two
D5.4 User Evaluation App, First Prototype11minutes of the programme, of 30 seconds in length. As these clips become available, they are asynchronouslydownloaded and put onto the web server. When all the clips are downloaded, the recommendation data iscopied to the server, refreshing the available programme list.Since the videos from the Snippets Transcoder service are provided in h.264 we could deliver the trailers tousers with a HTML 5 Video player that worked on the desktop and mobile. We also made use of local storageto store the user’s playlist.Video playback currently only works in Chrome, Safari and Internet Explorer. Support for h.264 in Firefox is inthe pipeline and can be accessed via nightly builds. On the iPhone the video plays only in the native player;this is not an issue for the iPad.This application has attracted interest inside the BBC and within the project, and is a very interesting candidatefor prototyping on a second screen and for other testing and evaluation.2.3Radio DashboardUnlike the other prototypes, Radio Dashboard was designed for internal BBC use - to give a technical orproduction team an overview of audio IP streams, but also allowing them to dig into the data in detail toidentify potential causes of increases or decreases in traffic. The design goal was therefore to show as manypieces of information as possible in a way that allowed the user to make sense of them.Figure 7: Radio Dashboard channel detailAlthough our remit was to make an audience-facing application for TV, we felt that this idea had sufficientinterest and relevance to the project - and application and interest to other partners - to pursue it further. Thishas proved to be correct, and development of this prototype continues to TV and other streams within theBBC, and to other datasources.The starting point for design was a ‘bubble interface’ idea: a visual representation of the size of the the audiencefor each radio station represented by the volume of the bubbles rather than an ordered list as in the HeartbeatEPG (Figure 6). The intention was to make something that looked and responded dynamically, so when thepage updated, the bubbles would grow / shrink and move. The bubbles are vector graphic versions (SVG) ofthe radio station logos, so that they could be scaled smoothly as the bubbles got bigger and smaller. A scalehad to be set for the smallest bubble size based on legibility.The user can click on any of the bubbles (stations) for more contextual information - the design pattern usedhere is a simple lightbox which displays a graph showing viewer figures from the last hour on divided into desktopand mobile (Figure 7).
12ViSTA-TVFigure 8: Radio Dashboard channel in depthThe third screen shows even more detail - the number of users joining and leaving the channel in a five minuteperiod and where they go, the number of tweets in that period, and what track was playing, if any (Figure 8).The backend of this system was a combination of the previous two backends. We used the Node.js server fromRadio EPG and added further additional data streams. We complemented these with data from other BBCAPIs; social media information, and music metadata from our internal playout system.It was important for data points to be emitted only when fully-formed with data from all the available sources.We added the multi-state system from Infinite Trailers in order to ensure all sources are integrated beforeupdating the client in real-time. Every five minutes, an aggregation process iterates across the recent datapoints and emits the collated data, producing the bubbles in the activity graphs. Like Infinite Trailers, the datais stored locally in SQLite [10]. This allows us to offer the previous hour of data immediately on connection,while keeping users up-to-date as new information is received.The frontend for this application consisted of three charts built with SVG and HTML. Initially we used a lowlevel JavaScript graphing library called D3.js [3] but switched to Raphaël [7] because it was easier to work withthe vector graphics once they were converted into Raphaël JSON. One challenge we encountered was displayingthe bubble randomly inside the chart without any overlap. This would have been easier if the logos were thesame size and shape. We ended up using a technique called circle packing to ensure the logos did not overlapwhenever an update occurred. We found animation performance greatly increased by animating the SVG item’sparent html elements and not the SVG elements directly.This prototype has been an interesting starting point for displaying large amounts of information about multiplechannels in one place. The team is currently working on a new version of the dashboard which can display allavailable streams (not just radio, but TV and ad-hoc streams for special events), and which allows the user topick the ones they are interested in for display (Figure 9).2.4User Testing and EvaluationWe have begun some initial discussions with usability testing experts within the BBC about how to approachuser testing and evaluation, mostly centred around ‘infinite trailers’ as this particular prototype has attractedthe most attention of the user-facing prototypes.Our approach is not finalised but may consist of
D5.4 User Evaluation App, First Prototype3 Lab tests for usability and feature evaluation A small diary study with followup interviews Tracking behaviour with a followup questionnaire13ConclusionsFigure 9: TV DashboardIn this document we have described the process of creating three working prototypes using the BBC data invarious ways. We used this process as an engaging and efficient way of learning what we should do next. Thegoal was to learn - about the data, architecture, user experience and features, to apply to a more finishedprototype later.We expect the software to be more generally applicable within the project, and have open sourced the frontendcode for Dashboard and started to apply it to Zattoo data using the Storm pipeline. We have also experimentedwith using the code with a completely different dataset - Eurovision counts by country from Twitter in real-timeduring the live event (see Figure 10).Figure 10: TV Dashboard
14ViSTA-TVThere has been enormous value in having working prototypes, even if they are limited in various ways, for us asa team within the BBC and within in the project.Next steps are to focus on applying Infinite Trailers or Heartbeat EPG to a second screen environment andtesting and evaluating the results.4AcknowledgementsWe would like to thank the participants in the workshop, and the many other individuals we have spokento as part of the process, particularly Faith Mowbray, Michael Smethurst, Sacha Sedricks, Chris Kimber, IanKnopke, Nick Humfrey, Vinoba Vinayagamoorthy, Tristan Ferne, Andrew McParland, Olivier Thereaux, VickySpengler, Tony Hirst, Robert Brook, Stephann Makri, Liz Conlan, Vana Ladeira, Elizabeth Lane, Mu Mu, ColinMcLaughin, Paul Rissen, Ulrick Hogrebe; with apologies if we have missed anyone from this list.References[1] Bbc /programmes developer guide. http://www.bbc.co.uk/programmes/developers.[2] Client side tml.[3] d3.js javascript graphing library. http://d3js.org.[4] iplayer performance pack, march .co.uk/mediacentre/latestnews/2013/[5] Json: Javascript object notation. http://en.wikipedia.org/wiki/JSON.[6] Node.js: fast, scalable network applications. http://nodejs.org.[7] Raphael - javascript library. http://raphaeljs.com.[8] Sibyl, a drag and drop recommender for iplayer. http://sibyl.prototyping.bbc.co.uk.[9] Socket.io: Realtime apps in every browser and mobile device. http://Socket.IO.[10] Sqlite. http://www.sqlite.org.[11] Storm: Distributed and fault-tolerant realtime computation. http://storm-project.net.[12] Websockets. http://en.wikipedia.org/wiki/WebSocket.5Appendix: User Experience DocumentationThese are designs made by Andrew in the duration of the sprint.
D5.4 User Evaluation App, First PrototypeFigure 11: Heartbeat EPG Design15
16ViSTA-TVFigure 12: Heartbeat EPG Design
D5.4 User Evaluation App, First PrototypeFigure 13: Heartbeat EPG, Catch-up17
18ViSTA-TVFigure 14: Infinite Trailers - design mobile versionFigure 15: Infinite Trailers - design mobile version
D5.4 User Evaluation App, First PrototypeFigure 16: Infinite Trailers - design mobile version19
20ViSTA-TVFigure 17: Infinite Trailers - design mobile versionFigure 18: Infinite Trailers - design mobile version
D5.4 User Evaluation App, First PrototypeFigure 19: Infinite Trailers - design mobile versionFigure 20: Infinite Trailers - design two-screen version21
22ViSTA-TVFigure 21: Infinite Trailers - design two-screen versionFigure 22: Dashboard initial design
D5.4 User Evaluation App, First PrototypeFigure 23: Dashboard initial design23
24ViSTA-TVFigure 24: Dashboard initial design
The prototype uses the API from Snippets to get 30 second clips from the first two minutes of the iPlayer. . downloaded and put onto the web server. When all the clips are downloaded, the recommendation data is . user testing and evaluation, mostly centred around 'infinite trailers' as this particular prototype has attracted .
Combining SDN and NFV SDN (fabric) and NFV (overlay) are managed separately Increased operational complexity / opex Difficult to optimize across different stacks Lack of visibility for troubleshoot and end-to-end optimization Separate resource pools OpenStack / K8S Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App .
To build your first app with Reading App Builder: 1. Launch Reading App Builder from its icon on the desktop. 2. Click New App on the toolbar. The New App wizard will appear. 3. On the first page of the wizard, specify the App Name, such as Dogon Stories, Supyiré Proverbs, etc. This is the main title of your app and will be seen by the user.
USA YOUTH SABHA SAHAJANAND SWAMI MAHARAJ SEMESTER PART I Swami Shriji Table of Contents Aashirvād App. B Week 1 Week 2 Week 3 Week 4 App. F Week 5 Week 6 App. I 1 2 App. A 3 App. C 10 17 25 32 39 App. H
Once an app is loaded, right-click the app icon on the Teams app bar, and select "pin" Pin a team app Click the " " at the top of a channel and select the app to add from the list PowerPoint 2016Build & run custom apps with Power Apps Expand app screen Click the icon to expand the app to full screen Chat within the app
Creating new Lightning Page using Lighting App Builder Salesforce Lightning pages can be created using Lightning App Builder. To create, navigate to Build Lightning Bolt Lightning App Builder New. Lightning App Builder - App page. In this step, select App page and click on next button as shown below. Lightning App Builder
USING THE SMART KAPP APP. your mobile device near your board's NFC tag (Android mobile devices only). The Apple App Store or Google Play opens. 2. Download and install the app. Starting the app for the first time To start the app for the first time 1. Start the SMART kapp app on your mobile device. The app tutorial appears. 2.
To download the Lutron App onto an Apple device, open the App Store app and using the search feature, search for the Lutron App. Select the Lutron App in the search results and click on GET to begin downloading the application. On the next screen choose Agree & Continue and the Lutron App will begin to download onto the device.
MagicTrick: Your First App Inventor App This step-by-step picture tutorial will guide you through making a magic trick app to make a rabbit appear in a top hat. To get started, go to App Inventor on the web. Go directly to ai2.appinventor.mit.edu, or click the orange "Create" button from the App Inventor website.
