Evidence-Based Training Implementation Guide

Evidence-Based TrainingImplementation GuideJuly 20131st Edition

Evidence-Based TrainingImplementation GuideJuly 2013International Air Transport AssociationMontreal—Geneva1st Edition

NOTICEDISCLAIMER. The information contained in thispublication is subject to constant review in the lightof changing government requirements and regulations. No subscriber or other reader should act onthe basis of any such information without referringto applicable laws and regulations and/or withouttaking appropriate professional advice. Although every effort has been made to ensure accuracy, theInternational Air Transport Association shall not beheld responsible for any loss or damage causedby errors, omissions, misprints or misinterpretationof the contents hereof. Furthermore, the International Air Transport Association expressly disclaimsany and all liability to any person or entity, whethera purchaser of this publication or not, in respect ofanything done or omitted, and the consequences ofanything done or omitted, by any such person or entity in reliance on the contents of this publication. International Air Transport Association. AllRights Reserved. No part of this publication maybe reproduced, recast, reformatted or transmitted in any form by any means, electronic ormechanical, including photocopying, recording or any information storage and retrieval system, without the prior written permission from:Senior Vice PresidentSafety and Flight OperationsInternational Air Transport Association800 Place VictoriaP.O. Box 113Montreal, QuebecCANADA H4Z 1M1Evidence-Based Training Implementation GuideISBN 978-92-9252-191-2 2013 International Air Transport Association. All rights reserved.Montreal—Geneva

EVIDENCE-BASED TRAININGIMPLEMENTATION GUIDETable of ContentsGLOSSARY OF TERMS . ivDEFINITIONS . vEXECUTIVE LETTER . viiDISCLAIMER . viiiBACKGROUND .1123INTRODUCTION AND RATIONALE .51.1Background.51.2Pilot competencies .51.3Evidence .61.4Operator‟s EBT programs.61.5Instructors and examiners .6KEY ELEMENTS .72.1General Principles .72.2Staged Implementation.82.3Baseline EBT Program .82.4Enhanced EBT Programs .112.5Collection and Analysis of Training Data.142.6Integration of Analysis .142.7Enhanced EBT Program Development .152.8Gaining Regulatory Approval.182.9Partnering for Results .20COMPETENCIES AND BEHAVIORAL INDICATORS .213.1Definition .213.2Example derived from the ICAO definition .213.3Uses of competencies in EBT recurrent training .213.4Components of competencies .223.5Example past models .223.6Competencies .233.7Comparison of data .233.8Different systems .233.9ITQI – the new safety tool.233.10EBT competency measurement – example .233.11Guidelines for the consistency of assessments .243.12Guidelines for developing instructor standardization and training modules .24i

EVIDENCE-BASED TRAININGIMPLEMENTATION GUIDE43.13Guidelines for developing crew effectiveness modules . 253.14Implementation guidelines for EBT instructors . 253.15Guidelines for maintaining standards and inter-rater reliability . 253.16Guidelines for improving competency and measurement system . 26INSTRUCTOR TRAINING . 274.1The EBT instructor . 274.2Effective instruction . 284.3Instruction and facilitation. 294.4Facilitation skills . 324.5567Questioning . 324.4.2Active listening . 324.4.3Body language . 334.4.4Observation of behavior . 334.4.5Giving and receiving criticism . 334.4.6Continuous development . 334.4.7Some advice for effective facilitation in a debrief . 33Important characteristics of EBT instructor training . 354.5.1Assessment . 354.5.2Standardizing EBT training and assessment . 354.5.3Expanding practice . 364.5.4Debriefing . 364.5.5Methodology of debriefing . 36PILOT UNDERSTANDING OF EBT AND TRAINING . 375.1Introduction. 375.2Philosophy of EBT history . 375.3Recurrent EBT programs . 38TRAINING METRICS . 406.1Introduction. 406.2Process . 406.3Generation of criteria . 416.4Grouping of Criteria . 416.5Development of grading options . 436.6Techniques to be applied in grading . 45PROGRAM DEVELOPMENT . 467.1ii4.4.1Introduction. 46

EVIDENCE-BASED TRAININGIMPLEMENTATION GUIDE87.2Background from the analysis .477.3EBT recurrent assessment and training matrix .477.4Implementation of a baseline EBT program .497.4.1General .497.4.2Construction of EBT modules .49CONTINUOUS IMPROVEMENT .578.1Introduction .578.2Development of the Baseline Program.578.3Enhanced EBT Program.588.3.1Operational data monitoring .588.3.2Operational characteristics .588.3.3Competency framework .588.3.4Safety Management Systems (SMS) and EBT .598.3.5Reporting Systems .598.3.6Flight data analysis (FDA) .598.3.7Flight deck observation .608.3.8Training Data .608.3.9Participation in data sharing groups .638.3.10Integration of analysis .648.3.11Conclusions and recommendations .64APPENDIX A Competencies and Behavioral Indicators .65APPENDIX B Instructor Training .67APPENDIX C Threat and Error Management (TEM) .70APPENDIX D Example EBT Module .78APPENDIX E Malfunction Clustering .94APPENDIX F Baseline Program Priorities .98APPENDIX G Training Program Development Guidance – Generation 4 (Jet) .101APPENDIX H Training Program Development Guidance – Generation 3 (Jet) .110APPENDIX I Training Program Development Guidance – Generation 3 (Turboprop) .119APPENDIX J Training Program Development Guidance – Generation 2 (Jet) .130APPENDIX K Training Program Development Guidance – Generation 2 (Turboprop) .139APPENDIX L Training Program Development Guidance – Generation 1 (Jet) .148APPENDIX M Example Grading Scales .149APPENDIX N Frequently Asked Questions .153iii

EVIDENCE-BASED TRAININGIMPLEMENTATION GUIDEGLOSSARY OF EBTDAFAAFLFMSFOQAFSTDGA or ARSOPTEMTOivAircraftAirborne Collision Avoidance SystemApproachAdvanced Qualification ProgramAir Transport AssociationAir Traffic ControlApproved Training OrganizationAlternative Training and Qualification ProgramCivil Aviation AuthorityClimbCrew Resource ManagementCruiseDescentEuropean Aviation Safety AgencyEvidence-Based TrainingDecision altitudeFederal Aviation Administration (United States of America)Flight levelFlight Management SystemFlight Operations Quality AssuranceFlight Simulation Training DeviceGo-aroundGroundInitial Operating ExperienceInter-rater reliabilityLandingLine Oriented EvaluationLine Orientated Flight ScenarioLine Oriented Flight TrainingLine Operations Safety AuditMinimum equipment listMulti-crew pilot licenseNew engine optionPilot FlyingPilot-in-CommandPilot MonitoringPilot Not Flying (former term for PM)Quick Access RecorderStandard Operating ProcedureThreat and Error ManagementTake-off

EVIDENCE-BASED TRAININGIMPLEMENTATION GUIDEDEFINITIONSAssessment. The determination as to whether a candidate meets the requirements of the competencystandard.ATA Chapters. The chapter numbering system controlled and published by the Air Transport Association,which provides a common referencing standard for all commercial aircraft documentation.Behavior. The way a person responds, either overtly or covertly, to a specific set of conditions, which iscapable of being measured.Behavioral indicator. An overt action performed or statement made by any flight crew member thatindicates how the crew is handling the event.Competency. A combination of skills, knowledge and attitudes required to perform a task to the prescribedstandard.Competency-based training. Training and assessment that are characterized by a performanceorientation, emphasis on standards of performance and their measurement and the development of trainingto the specified performance standards.Core competencies. A group of related behaviors, based on job requirements, which describe how toeffectively perform a job. They describe what proficient performance looks like. They include the name ofthe competency, a description, and a list of behavioral indicators.Critical flight maneuvers. Maneuvers that place significant demand on a proficient crew.Critical system malfunctions. Aircraft system malfunctions that place significant demand on a proficientcrew. These malfunctions should be determined in isolation from any environmental or operational context.Evidence-based training (EBT). Training and assessment that is characterized by developing andassessing the overall capability of a trainee across a range of competencies rather than by measuring theperformance of individual events or maneuvers.EBT instructor. A person, who has undergone a screening and selection process, successfully completedan approved course in delivering competency-based training, and is subsequently authorized to conductrecurrent assessment and training within an approved EBT program.EBT module. A session or combination of sessions in a qualified FSTD as part of the 3-year cycle ofrecurrent assessment and training.EBT session. A single defined period of training in a qualified FSTD that normally forms part of an EBTmodule.EBT scenario. Part of an EBT session encompassing one or more scenario elements, constructed tofacilitate real time assessment or trainingEBT scenario element. Part of an EBT session designed to address a specific training topicError. An action or inaction by the flight crew that leads to deviations from organizational or flight crewintentions or expectations.v

EVIDENCE-BASED TRAININGIMPLEMENTATION GUIDEError management. The process of detecting and responding to errors with countermeasures that reduceor eliminate the consequences of errors, and mitigate the probability of further errors or undesired aircraftstates.Facilitation technique. An active training method, which uses effective questioning, listening and a nonjudgmental approach and is particularly effective in developing skills and attitudes, assisting trainees todevelop insight and their own solutions and resulting in better understanding, retention and commitment.Factor. A reported condition affecting an accident or incident.Flight crew member. A licensed crew member charged with duties essential to the operation of an

recurrent assessment and training within an approved EBT program. EBT module. A session or combination of sessions in a qualified FSTD as part of the 3-year cycle of recurrent assessment and training. EBT session. A single defined period of training in a qualified FSTD that normally forms part of an EBT module. EBT scenario.