Using Human-in-the-Loop simulation in studies of the trade-off between advanced automation and Situational Awareness

Using Human-in-the-Loop (HITL) simulation, SST supported the Federal Aviation Administration (FAA) in studies of the trade-off between advanced automation and the need of maintaining Situational Awareness (SA) of air traffic controllers in charge of detecting and responding to aircraft traffic conflicts. One example is the degree of reliability of “aircraft conflict detection automation” necessary to support user acceptance and improve performance without negatively impacting operator workload.

SST conducted literature reviews and developed a detailed test plan, schedule, questionnaires, stimulus design, and experimental design. We developed traffic scenarios for both training and experimentation, and prepared experimental materials including training briefing materials, counterbalancing schedule, eye tracker and Heart Rate Variability (HRV) protocol, Workload Assessment Keypad (WAK) anchors, and questionnaires and observer forms. Our human factors experts performed the HITL shakedown and final readiness review for data collection, and trained controllers and sim pilots to participate in the experiment. During data collection, SST monitored experiment conduct and issues and sim pilot performance, calibrated and operated eye trackers, briefed non-human factors experimenters on HITL data collection standards and protocols for participant interaction, and reviewed questionnaire data integrity after runs. SST set up, initiated, and monitored HRV data collection and backed up HRV data, conducted immediate preparation and follow-up for data collection sessions including physiological calibration practice and logging issues with data, and participated in/facilitated debrief sessions. We oversaw data reduction and verified the integrity of the reduced data, and identified and addressed errors including duplicate data. SST conducted analysis using SPSS, Excel, and other tools, on data including simulator outputs and simulation replays, metering performance, eye tracking, questionnaire and WAK data, HRV, operational errors and deviations, and efficiency/throughput. SST was responsible for scheduling and planning all aspects of the overall data analysis and reporting effort. SST prepared a “quick look” briefing to provide the sponsor with initial analysis results on simulator outputs, metering performance, questionnaire, and WAK data.  Finally, SST wrote and delivered the final report to the sponsor.