Effectively Transforming Imc Flight Into Vmc Flight: An Svs Case . - Nasa

Three approach maneuvers were created, twoof which were conventional and one representing anadvanced operational concept. The conventionalmaneuvers included a VFR traffic pattern approachin VMC and an ILS in simulated IMC. Theadvanced operational concept was an IMCmaneuver that used the same flight path flown forthe conventional VFR traffic pattern approach;however, this “VMC-like” approach was flown insimulated IMC. Figure 5 shows a gods-eye-view ofthe evaluation maneuvers.The VFR traffic pattern incorporated adownwind leg displaced approximately 1.4 nmfrom the runway at 1,000 ft above the touchdownzone, a base leg, and a 1.5 nm final approachsegment. Data runs were initiated at 100 ktsindicated airspeed (IAS) at 1,000 ft above thetouchdown zone on the downwind leg heading atPoint A (in Figure 5). EPs were instructed to flyparallel to the runway, maintain airspeed andaltitude with flaps retracted. Once abeam therunway numbers, EPs were to select 10 degrees offlaps, reduce speed to 90 kts IAS, and complete theapproach. All pilots were instructed to continue theapproach to 50 ft AGL, or to actual touchdown,depending on their level of experience andprevailing conditions. The VFR traffic patternapproach was included in this evaluation to providea comparison to typical VMC operations.The ILS approach included a 90-degree baseleg, a 30-degree localizer intercept, and a 3.0degree glideslope intercept point approximately 9nm from the runway touchdown markers. Eventhough ILS was not available for Runway 24 atROA, GPS information was used to calculate anddisplay simulated ILS CDI responses. The ILSapproach was initialized at approximately 2700 ftabove the touchdown zone at 100 kts IAS on thebase leg heading at Point B in Figure 5. Onceestablished on the localizer, EPs were to select 10degrees of flaps and reduce speed to 90 kts,intercept the glideslope and complete the approach.Pilots were instructed to continue to the decisionheight (DH) at 200 ft above ground level (AGL)and then callout “DH”. At the DH, the SP wouldsay “continue to 50 ft AGL”, “continue to landing”,or “runway not in sight”, depending on pilot skilland prevailing conditions, and whether or not asimulated missed approach (MA) was required asindicated by the test matrix. The EP’s ability toidentify the missed approach point was part of theexperiment. The test matrix was designed such thatapproximately half of the IMC approach runs (ILSand VMC-like) would include a MA. The run endedeither at touchdown or when positive rate of climbwas established on the MA. The ILS approach wasincluded to provide comparisons with typical IMCoperations.Figure 5. Approach maneuvers overlaid on areacontour map.A waypoint (“NEWT”) was created for theadvanced operational concept on the downwindapproach path, abeam the runway numbers todesignate the point at which the EPs were to select10 degrees of flaps and decelerate to 90 kts IAS. AllEPs flew the VMC-like approach with each of theexperimental display concepts except the BRD. TheVMC-like approach had the same terminationoptions as the ILS approach (i.e., continue to 50 ft,land, or MA). The run ended either at touchdown orwhen positive rate of climb was established on theMA. The VMC-like approach represents a potentialfuture operation that could be enabled by SVSdisplays.For all scenarios, EPs were isolated fromactual ATC communication and asked to monitor asimulated air traffic control channel that would playpre-recorded ATC messages based on aircraftlocation. They then answered questions aboutsimulated air traffic and provided other qualitativedata immediately after each run. Isolating the EPsfrom actual ATC communications provided a moreuniform workload for research evaluations. For thesimulated IMC scenarios, the EPs wore an IFR

training vision restriction device. No visionrestriction was used for the VFR runs.Evaluation pilots received training prior to datacollection to preclude learning effects in the data.Training included a one hour briefing regarding thecharacteristics of the experiment along with adescription of all display concepts, evaluationmaneuvers, and subjective and objective measures.Next, EPs received approximately 2.5 hours oftraining using a desktop simulator where all testconditions were rehearsed. Pilots could elect toretry test conditions until they were comfortablewith all aspects of the experiment. In general, EPsdid not elect to retry training runs and were able todemonstrate acceptable performance. Once thesimulation was completed, EPs receivedapproximately 1.2 hours of flight training in the C206 to become familiar with the aircraft, flight crewprotocol, research apparatus and scenarios.During the experiments, the SPs wouldestablish the aircraft approximately 1.0 nm from thestart waypoint (i.e., Waypoint A for the VFR/VMClike approaches or Waypoint B for the ILSapproach) on the appropriate heading and altitude atwhich time control would be transferred to the EP.The EP would then complete the maneuver setupand perform the evaluation maneuver. Once themaneuver was completed, the SP would resumecontrol of the aircraft and the EP would then answerthe post-run questionnaires using an electronicquestionnaire tool implemented on a pocketpersonal computer.The order of test condition presentation wasrandomized (subject to some operationalefficiencies) to further minimize learning, fatigue,and environmental effects. Generally, datacollection was performed during two approximately1.2 hour flights to avoid fatigue, though a few pilotscompleted the entire test matrix in one 2-hourflight. EP participation in the flight test usuallyspanned two days. A substantial effort was made toconduct the formal research evaluations within 4hours after sunrise in order to reduce atmosphericvariability effects within the resulting data. Allflight operations were conducted in clear-day VMC.Dependent MeasuresPost-run questionnaire results analyzed in thispaper consisted of the 3-D Situation AwarenessRating Technique (SART, [12]), 6-D Task LoadIndex (TLX, [13]), and Display Readability Rating(DRR, [14]) measures. Quantitative data recordedincluded flight path and airspeed errors.To characterize their flight performance, theFAA Practical Test Standards (PTS) were applied.To earn an instrument rating, pilots mustdemonstrate the ability to maintain flight path andairspeed errors within the acceptable limits outlinedin the PTS. As a result, the accepted and desiredIMC performance conditions were based on theFAA PTS criteria as defined in [15]. The resulting“IFR PTS boundary”, used for both the ILS and theVMC-like maneuvers, began at a specified entrygate and ended at the DH. It was bounded by ¾scale maximum deflections (1.5 dots) of localizerand glideslope indicators, and /-10 kts of IASerror. For the VFR-like maneuver, vertical linearflight path error was used along with the distancealong flight path to touchdown to generate ILSglideslope-like deviation data. Lateral linear flightpath error, combined with the distance along theflight path to a point 1,000 beyond the departureend of Runway 24, was used to generate ILSlocalizer-like deviation data. While exceeding theIFR PTS does not directly infer an imminentaccident, boundary violation does provide anindication of accident risk, or exposure.For the VFR maneuver, the VFR PTS criteriawas based on the VFR PTS, described in [16], andalso recommended practices for operations at nontowered airports, [17], to develop assumedacceptable lateral and vertical errors for VFRoperations. The resulting “VFR PTS boundary” wasdefined with dimensions of /-0.5 nm laterally and /-250 ft vertically, and /-10 kts of IAS error at thestart of the maneuver. The VFR PTS boundarylinearly tapered from its full size during the finalapproach segment, with the tapering beginning atapproximately 1.5 nm from the touchdown zone,narrowing from its nominal size down to 150 ftlaterally and 0 ft vertically at the runwaytouchdown location. Analysis of the “VFR PTSboundary” ended at the same DH point as the othertwo approaches. While exceeding the VFR PTSboundary would not directly be a precursor to an

accident, boundary conformation does provide lessexposure to accident risk.Definition of Descriptive MethodAs previously stated, one objective of thiseffort was to more fully define the effects from SVSdisplays and to test the hypothesis that SVSdisplays can effectively transform IMC into VMCflight. Application of MVDA classificationtechniques offered the potential to associateresearch results from the advanced display conceptswith various significant reference condition resultsfrom conventional displays, prevalent withintoday’s GA operations. By including the significantreference conditions within the data set, a limitedrelationship between VMC and IMC flight wasconsidered feasible. These significant referenceconditions were represented by groups of data forspecific combinations of display concept (BRD),pilot skill level (VFR, IFR, and High IFR),maneuver (ILS and VFR), and visibility (IMC orVMC) and were used as input data for the MVDAtechnique.Classification of information is an importantanalysis tool in many industries (Ref. 18). Oneexample is the classification of loan applicants aslow- or high-credit risks based on elements ofcertain accounting metrics to provide a rapid,reasonably accurate expectation of results. MVDAis one such classification method and was employedfor this study, combined with several commonsignificantly different risks of pilot error situations,in an attempt to add increased dimension and clarityto the description of the effects of SVS displays.Two different reference conditions under IMCwere considered as reference conditions forclassification: one condition (Group 1) wasintended to represent an unacceptable risk for IFRoperations while Group 2 was intended to representan acceptable risk for IFR operations. A VMCcondition was defined Group 3 for VFR operationsand was intended to represent a very low-riskcondition. Establishing these three groups of datawithin the MVDA provided an alternative methodto describe the effects from SVS displays. Forexample, results from the advanced displayconcepts could be described as grouping more withone significant reference condition or another. Inaddition, through evaluation of the classificationcapability of various MVDA configurations,assessments of the effectiveness of traditionalmeasures to describe effects similar to those of SVScould be accomplished.Group 1 was associated with the VFR pilotsflying an ILS approach with the BRD displayconcept. VFR pilots lack the skills acquired throughIFR training to integrate the information providedby the BRD display concept to safely perform ILSapproaches. The resulting risk is unquestionablyhigh for this test condition. Reference 19 states that68% of fatal weather related accidents wereattributed to “attempted VFR flight into IMC”mostly involving VFR pilots.Group 2 was defined as the instrument-ratedpilots also performing the ILS approach with theBRD display concept. This test condition reflectsnominally accepted risk since instrument-ratedpilots have received the required training and havedemonstrated the ability to operate the aircraft viaBRDs in IMC. While Group 2 is a nominallyaccepted operational concept, Reference 19indicates that 1.63 fatal accidents per 100,000 flighthours occurred during day IMC flight producingapproximately 20 percent fewer total accidents per100,000 hours but almost three times the rate offatal accidents as VMC.Group 3 was defined as the H-IFR pilotsperforming a basic VFR traffic pattern with theBRD display concept. This condition wasconsidered to have a very low risk due to the levelof pilot skill, inferred by their large number of flighthours and commensurate training, as well as thebenign nature of the VFR traffic pattern maneuver.Reference 19 indicates that 0.7 fatal accidents per100,000 flight hours occurred in VMC. The abovegroup designations defined the subset of runs (thediscriminating conditions) to be utilized in theinitial step of the MVDA.Test MatrixThe test matrix is provided in Table 2. Notethat the BRD display concept was not included inthe test matrix for the VMC-like approach becauseof anticipated very poor performance. In Table 2,VMC indicates that no vision restriction device wasused to restrict the EP’s visibility. Sim IMC

Table 2. Test CVMCVMCVMCSim IMCSim IMCSim IMCSim IMCSim IMCSim IMCSIM IMCThe VFR and IFR pilots evaluated the BRDdisplay concept for the ILS approach twice whilethe H-IFR pilots evaluated the BRD display conceptfor the VFR traffic pattern twice. The repeatevaluations of these test conditions were done toprovide more data to the MVDA developmentprocess. For other data analyses (i.e., besides theMVDA), the repeat runs were not included. Overall,each EP flew 12 total data collection runs tocomplete the run matrix (4 VFR traffic patterns, 4ILS approaches, 3 VMC-like approaches, 1 MVDArepeat).An ANOVA conducted on the SART data forthe Group 1, 2, and 3 conditions revealed the maineffect as being highly significant statistically(F(2,31) 19.9, p 0.001). Mean pilots’ SA for theGroup 1, 2, and 3 conditions are presented in Figure6 which shows that Group 1 and 2 SA scores aresimilar to each other. Subsequent post-hoc analysisyielded no separation of Group 1 from Group 2.This similarity implies that the extensive trainingrequired to become an instrument-rated pilot is notsignificantly reflected in the SA data. Even thoughthe instrument-rated pilots successfully acquiredtheir IFR rating, their ability to decipher andinterpret information from the BRD display conceptand maintain their mental model was very low,especially when compared to the SA of Group 3(note the statistically significant increase in SA ofapproximately 100 units because of VMCvisibility).150D SARTindicates that a standard IFR training visionrestriction device was used by the EPs.10050D D 0-501Data Analyses and ResultsResults presented and discussed in this paperfocus on the aspects of the development andapplication of MVDA and the evaluation ofadvanced display concepts. A complete publicationof all results is planned for a subsequent NASAtechnical paper.Analysis of Group 1, Group 2, and Group 3data Using Traditional MethodsThe results of an initial analysis independent ofthe MVDA using traditional HF tools applied to theGroup 1, Group 2, and Group 3 data are provided inthis section. Analysis of these specific testconditions are included to provide insight into theMVDA results, which will follow.23MVDA Input GroupFigure 6. Mean SART data for the MVDA inputgroups with standard deviation bars.Mean pilot workload, as defined by the TLXdata for Groups 1, 2, and 3 are presented in figure 7.An ANOVA conducted on the TLX data revealedthe main effect as being significant statistically(F(2,31) 7.383, p .002). However, subsequentpost-hoc analysis did not separate Group 1 fromGroup 2, a result similar to the SART data. Thislack of separation implies that the extensive trainingrequired to become an instrument-rated pilot is notreflected in the TLX data either. Although the IFRpilots were able to maintain localizer, glideslope,and airspeed error to within acceptable limits, theTLX data indicates that their workload was still

very high, especially when compared to that of thestatistically separable Group 3 data. And like theSA data, the TLX separation primarily onlyreflected the large differences between the VFR andILS approaches.75D The mean percentages of time EPs were ableto achieve appropriate PTS criteria are presented inFigure 9. Note that for Group 1 and Group 2conditions, the IFR PTS was applied. For Group 3condition, the VFR PTS was applied.D TLX50a 2.5 unit decrease for Group 3. An ANOVAconducted on the DRR data revealed that the maineffect was statistically significant (F(2,31) 4.8,p 0.015). However, subsequent post-hoc analysiscould only differentiate Group 1 from Group 3,with Group 2 data being similar to both Group 1and Group 3.25D 120023MVDA Input GroupFigure 7. Mean TLX data for the MVDA inputsgroups with standard deviation bars.100% Time PTS1D D 8060D 4061D DRR43Figure 9. Mean percent time within appropriatePTS boundary with standard deviation bars.D 2D 012MVDA Input Group23MVDA Input GroupFigure 8. Mean DRR data for MVDA inputgroups with standard deviation bars.Unlike the SART and TLX data, DRR data(Figure 8) actually seem to indicate a difference inthe means between each of the three groups. Notethat DRR scores of 4 require the pilot to answer noto decision tree question “Is it satisfactory withoutimprovement?” and thus, indicate that deficiencieswarrant improvement. For the DRR results, it is notsurprising that Group 1 results indicated thatachieving desired performance was challenging.Group 2 results indicated somewhat better ratings,resulting in a decrease in DRR scores by 1 unit withAs was the case for the DRR data, there seemsto be a substantial difference between all threegroups. Group 1 results indicate a capability to bewithin the IFR PTS boundaries approximately 58%of the time, reflecting the inability of the VFR pilotsto continually decipher information and control theaircraft to within acceptable limits for the ILS inIMC. Group 2 data indicates that instrument-ratedpilots were able to satisfy the IFR PTS standardsapproximately 83% of the time. Group 3 dataindicate that the H-IFR pilots were able to be withinthe VFR PTS boundaries approximately 95% of thetime. This is not surprising since the VFR lateraland vertical PTS boundaries were so large,combined with the pilots’ higher degree of skillachieved over thousands of hours of flight inaddition to operating in clear-day VMC. AnANOVA performed on the PTS data revealedhighly significant results (F(2,31) 13.7, p 0.001).However, subsequent post-hoc analysis placed the

Group 2 and 3 results together and was only able toseparate Group 1 data from the others. The VFRpilots performed significantly worse than theinstrument-rated pilots while flying the ILSapproach with BRD display concept and the H-IFRpilots performing the VFR traffic pattern with BRDdisplay concept.The data analyses provided in this section weregenerated using traditional analysis tools andtechniques that are common to the HF testdiscipline. From these results, it can be seen thatvarious measures were able to distinguishdifferences resulting from the different groups ofdata. However, none of the metrics, takenindividually, could adequately separate all threegroups from each other. If the objective of thispaper were to compare the results from Groups 1, 2,and 3, an inference from SART, TLX, DRR, andPTS results would be required.Some observations that resulted from this typeof analysis when applied to the advanced displayconcepts and maneuvers are merely summarizedherein: 1) The advanced display concepts weresignificantly better compared to the BRD, in termsof SART, TLX, DRR and PTS (lateral and verticalflight path error primarily); 2) the PBG guidanceresulted in significantly better lateral FTE than didthe SCFD due to the turn anticipation provided bythe tadpole guidance and pathway presentation; 3)the incorporation of terrain in the SVS displayconcept significantly improved SART, TLX andDRR results as compared to those of the PBGdisplay concept, generating results similar to thoseof the VFR evaluations of the BRD; and 4) resultsfor the SVS display concept indicated no effect ofmaneuver condition (VFR, ILS, VMC-likeapproaches) for all metrics considered, essentiallyremoving much of the effect of limited visibility.The removal of significant effects of limitedvisibility and maneuver condition was only true forthe SVS display concept.Multi-Variable Discriminant AnalysisThe objective of the MVDA application was toassess and blend the strengths of the variousindividual data measures, both qualitative andquantitative, and endeavor to establish a morecomprehensive metric tool. This tool might beparticularly critical to SVS applications due to theprofound effects that continuously-availablecomputer-generated visibility may have on safetyand operational capabilities.The MVDA development process employedthe three reference groups, discussed previously asinput data. The DA tool in the SPSS statisticalanalysis software package was used to generateMVDA functions of the various research measures.The DA performs linear discriminant analysis fortwo or more groups. The goal of discriminantanalysis is to classify cases into one of severalmutually exclusive groups based on their values fora set of predictor variables. In the analysis phase, aclassification rule is developed using cases forwhich group membership is known. In theclassification or prediction phase phase, the rule isused to cla

in VMC and an ILS in simulated IMC. The advanced operational concept was an IMC maneuver that used the same flight path flown for the conventional VFR traffic pattern approach; however, this "VMC-like" approach was flown in simulated IMC. Figure 5 shows a gods-eye-view of the evaluation maneuvers. The VFR traffic pattern incorporated a