What!You!Can't!Measure,!You!Can't! Improve .
ite400,OrangePark,FL32073888- ‐- ‐ 268- ‐- ‐ 8770· · 904- ‐- ‐ 278- ‐- ‐ 0524- ‐info@techwell.com- ‐http://www.starwest.techwell.com/
loymentpipelineforanomnichanneleCommerceplatform.
10/9/18What you cantMeasure, you cantImprove.Ashwin DesaiVP QualityAgenda Hudl Overview Product Team Measure & ImproveQuality Learnings Metrics 2.0 Wrap up1
10/9/18We help teamsand athletes win.Helping Teams WinCaptureShareAnalyse2
10/9/18Record with yourfavorite device.Use your iPhone, iPad orhard drive camera to recordevery game or trainingsession. Connect to Wi-Fiand the video will upload asit’s captured.Access videoanywhere.Full games and practicescan be shared with the wholeteam to study from anycomputer or mobile device.3
10/9/18Bring lessonsto life.Help your team seeexactly what needs toimprove. Allow players tocritique their ownperformance, or providepersonalized feedback bysharing comments anddrawings.InteractiveVisual ReportsShot charts allow you tostudy every shot type for yourteam and the opponent.View shots and goals from asingle game or the wholeseason in seconds.Click any shot in the chart towatch the video.4
10/9/18Three ways to track statsAt the GameHave an assistant coach,injured player or parent usethe Hudl app to track yourteam stats live.After the GameTrack team and playerstats as you re-watch thegame on any iPad orcomputer.Leave It to UsSend us your video throughHudl Assist and you’llreceive team and playerstats in under 24 hours.We have products for teams atevery level of competition.5
10/9/18We work with the world’s best.20/20English PremierLeague15/23Major LeagueSoccer29/30National BasketballAssociation18/18Australian RulesFootball League18/20Chinese BasketballAssociation4.5MM app downloads4.4MM active usersHudl isthe industrystandard.160K active teams98% of high school football teams41K high school basketball teams30 sports around the world38 hours of video loaded every minat peak6
10/9/18Product Team @ HudlMicroservices tformleroycoresearchmaxprepsmonolithrecruit7
10/9/18Product Team @ Hudl 25 small autonomous squads working on 12 BetsShip early, ship oftenAnyone can work on any code.Anyone can deploy, anytimeDeploys and rollbacks are fast and easy 250 deploys to production per weekProduct Team @ Hudl - 2016Use monitoring in production to understand Quality.Quantitative in-process data was not being collected.Lack of standardization.8
10/9/18GoalImprove Quality delivered byProduct TeamHypothesis – Build Quality In.Reduce rework. Increase flow.9
10/9/18BuildQuality InImprovingQualityImprove Quality.How would we know it hasimproved?You need to measure it, to showimprovement.10
10/9/18ProcessIntroduce concepts of in-processand production qualityStandardize data collectionAgree on MeasurementsCollect MeasurementsEstablish BaselineProcessAnalyze dataIdentify ChangesRepeat Measurements to see ifimprovement in Baseline11
sseBetTeamocilSquadAvengersSquadMeasure Quality for each Bet – use samemeasurements for each Bet.12
10/9/18Add up measurements of individual Bets tounderstand overall quality produced by theProduct Team.Use time intervals, not releases, as a basisto measure quality13
10/9/18Goal – improve Product Team Quality byimproving Quality for each BetQuality metricsConcerns/Questions1)Why have metrics?2)What will we measure?3)What will you do with the metrics?4)Metrics can be misused5)Metrics can be gamed.14
10/9/18Key definitionsBets Investment themes/projectsBet Sub-defects defects found before deploy to productionBet Defects defects for functionality worked on by the bet found after deploy in production.Measurements – for each bet1) Testing effectivenessa)Quality coming into QA Sub-defects per developer per weekb)Quality Leaving QA Defects per developer per weekTesting effectiveness Quality Leaving QA/Quality coming into QA15
10/9/18Measurements for each bet2) Amount of rework - % of total deploys to production that are fixes.3) (External) change fail percentage – Hotfixes per week4) Debt - Open legacy defects5) Customer feedback - Number of support interactionsThe numbers16
10/9/18The trendsMore trends 17
10/9/18Inspect1)2)3)4)5)The teams were collecting the data but were not using the data.Data collection not consistent across teams.Concerns of misuse.Concerns of changed behavior among teams.Hard to change culture!!Adapt Form a team of QAs to focus on metrics.Send a survey to product teamAnalyze the level of adoption, applicability and usefulness of the current Quality MetricsDetermine areas of improvement.18
10/9/18ResultsResults19
10/9/18Qualitative Feedback1) Each bet is different in nature, some more so than others.2) Per engineer metrics are problematic; the current metrics are creating unhealthy pressure.3) Too many metrics.Learnings Keep it simpleAllow flexibility for betsRemove bet to bet comparisonCollaborative v/s competitive20
10/9/18Metrics 2.0Changes Core Metrics Product Team level reporting Three measurements21
10/9/18“Product Team” Core Metrics Internal change failure rate - % of storiesthat have at least one sub-defect loggedagainst them. Testing effectiveness – % of defects foundpre-prod v/s in production External change failure rate - % of totaldeploys to production that are fixes. Flow - # of Deploys per month to productionQuality Improvement Team1)Sub-defects, defects, story understood andapplied consistently.2)Establish Bet lead circles to learn and improve.3)Work with Bets to review their Bet Qualitymetrics and outcomes.4)Establish framework to sustain gains.22
10/9/18“Product Team” Metrics Trends What do they look like?“Product Team” Metrics Trends % of stories with at least one defect –% of defects found in prod –% of prod deploys that are fixes –# of prod deploys per month –23
10/9/18“Product Team” Metrics Trends –Diving in % of stories with at least one defect –% of defects found in prod –High percentage of prod defects was legacydefects.Next steps Use data to get buy in for changes – Focus on defect prevention into QA (eg Test DrivenDevelopment)Focus on defect detection in QA (eg document andreview test cases, “test bash”)24
10/9/18What about squads/bets? Customized Model Up to the teamGuidelines for teams Figure out what works for you.Limit to top three.Make them visible.Use data and demonstrate improvement in quality.25
10/9/18Customized metrics for a betSquad data26
10/9/18What’s Missing?Thanks!Code Quality Improvement Team§ Asma Gulbaz§ Peter Yasi§ Michael Li§ Jaron Ahmann§ Sufyan Farooqi§ Mark Noble§ Mike Korsakas§ Ethan Seyl27
10/9/18Wrap Up1)Measurements key to improving quality.2)Figure out what works for ”your” team.3)Look for counterbalancing measures.4)Look at trends.5)Make measurements visible.6)Make sure teams use data from measurements to improve.7)Expect to make changes to measurements based on team feedback.Quality is an ongoing journey – create aContinuous Improvement culture28
10/9/18Questions?Get in touch.Ashwin DesaiVP QualityAshwin.desai@hudl.com29
the Hudl app to track your team stats live. After the Game Track team and player stats as you re-watch the game on any iPad or computer. Leave It to Us Send us your video through Hudl Assist and you’ll receive team and player stats in under 24 hours. Three ways to track stats
List of core vocabulary Cambridge Pre-U Mandarin Chinese 9778: List of core vocabulary. 5 3 Measure words 把 bǎ measure word, hold 杯 bēi a cup of / cup 本 běn measure word 遍 biàn number of times 层 céng layer; storey 次 cì number of times 段 duàn paragraph, section 队 duì team 封 fēng measure word 个 gè measure word 壶 hú measure word 件 jiàn measure word
MEASURE WIDTH MEASURE HEIGHT 3 2 1 6 5 4 L-FRAME MOLDING How to Measure 1. Use a steel tape measure only. 2. Do not make any deductions, allowances, or additions to your measurements. 3. Measure to the nearest 1 16". 4. Measure window width in three places from the outside edge of the mol
Stravinsky, Rite of Spring o Part 1: Beginning to rehearsal 4 o Part 1: First 4 measures of rehearsal 12 o Part 1: Rehearsal 49 to 4th measure after rehearsal 53 Mozart, Requiem o I. Beginning to end of measure 7 o I. Measure 20-21 o I. measure 32 to downbeat 40 o II. Opening to measure 10 o III. Pickup to measure 53 to end
Inscribed Angle Theorem The measure of an inscribed angle is _ _ the measure of its intercepted arc. In other words, the intercepted arc is _ the measure of the inscribed angle. Guided Examples: Find the measure of the missing variable. 1. 2. 3. Find the measure of arc XZ 4. Find the measure of angle LPN. Theorem
This measure may be reported by clinicians who perform the quality actions described in the measure based on the services provided and the measure-specific denominator coding. Measure Reporting via Claims: CPT or HCPCS codes and patient demographics are used to identify patients who are included in the measure's denominator.
4 Marketo Measure Data Warehouse — All data generated by Marketo Measure is stored in the Marketo Measure data warehouse, which is a Snowflake instance. Additionally, Marketo Measure includes the following integrations: CRM Integrations — Marketo Measure integrates with CRM solutions to house and organize all the data that is captured by bizible.js.
You can also use this top trill key fingering at the end of measure 26, measure 30, and measure 45. 7. In measure 27, you can also use this fingering to play the B naturals, and likewise in measure 49. I prefer to use long B fingerings in these two measures, but if the trill fingering works
at the GRE Verbal Reasoning measure and Analytical Writing measure. Chapter 1 pro-vides an overview of the structure and scoring of the GRE Verbal Reasoning measure. In Chapters 2 through 6, you will find information specific to the content of the Verbal Reasoning measure. In Chapter 7, an overview of the Analytical Writing measure is pre-sented.
