The purpose of this study was to assess the relative effectiveness of different approaches to checking pilot performance at the end of training: the maneuver validation (MV) and the Line Operational Evaluation (LOE). Because the LOE provides greater contextual cues and integrates CRM skills with technical skills, it should simulate typical line operations more accurately than a traditional maneuver validation. Therefore, we hypothesized that pilots would rate the LOE as more useful than the MV. The results presented below are part of a much larger survey of airline pilots’ experiences in and reactions to their professional training (Baker et al., 2002).
Find specific work or narrow your results by type, topic, program, project, or service by selecting your criteria from the choices at right.
Over the years, the FAA has partnered with industry to develop programs for reporting, classifying, and analyzing safety-related data, but none has been able to integrate data from multiple sources. We are developing a generalized Human Factors taxonomy for classifying de-identified ASAP incident reports, AQP performance ratings, and FOQA output. Eventually this taxonomy will be embedded into a series of searchable computer databases that speak a common language, allowing the search for trends.
Because poor LOS construct validity can have real-world effects on pilot training and performance, assessing and improving the construct validity of Line Operational Simulations is more than just an academic or scientific issue. It is also a practical and political issue in that it involves multiple stakeholders who may have competing concerns. These include safety, justice/fairness, technical feasibility, and cost-effectiveness (Austin, Klimoski, & Hunt, 1996). Therefore, we recommend that all potential stakeholder groups be involved in identifying and improving the construct validity of Line Operational Simulations. These groups may include pilot unions, training staff, flight standards staff, and officials from the regional FAA offices. Moreover, all groups must be prepared to compromise some of their own goals/needs to achieve a balanced solution. In the end, only by working together can industry address the issue of LOS construct validity, and by extension, the quality of pilot crew training and evaluation.
This review—which summarizes the state-of-the art in aviation error reporting, classification, and analysis—serves as the foundation for our future taxonomic research.
In the sections below, we describe the results of a large-scale survey of pilots' perceptions of and experiences in their training. In particular, we focus on their responses to a series of questions concerning Crew Resource Management (CRM) training. This project was a unique opportunity to conduct a scientifically rigorous, large-scale comparison of CRM training programs across multiple airlines. Nevertheless, we recognize that participants' reactions to training are only one measure of a training program's effectiveness.
In this article we use a multifacet measurement technique, the multifacet Rasch model, to analyze the results of an IRR-training program. Our approach is an alternative to the procedures—congruency, consistency, agreement (rwg), sensitivity, and systematic differences—currently used during IRR training within the airline industry. We believe that this multifacet procedure can improve the quality of pilot instructor training by providing pilot instructors with important information that is not available with other techniques. We used the multifacet Rasch method instead of generalizability (G) theory, another multifacet technique. Similar to multifacet Rasch analysis, G-theory provides information about facets—pilot instructors, videotapes of aircrews used in IRR training, and LOS grade sheets—and their interactions with one another. However, G-theory partitions the variance attributable to each of these facets using an analysis of variance (ANOVA) framework and thus focuses on groups as the unit of analysis (i.e., whether or not pilot instructors as a group are reliable or unreliable as opposed to the performance of a particular instructor undergoing IRR training).
Despite the gaps in research, a number of knowledge-elicitation methods available from research on individual CTA seem adaptable to a team environment. Some of these have been used in the team performance arena, whereas others have not. This section suggests potential methods for the different types of team knowledge described in the previous section: methods for eliciting pretask team knowledge and dynamic team knowledge. We list the type of team knowledge and discuss previous attempts (if any) to elicit this knowledge. We also suggest other methods that have potential to tap this knowledge. Although a detailed description of all potential methods is beyond the scope of this chapter, we have attempted to include a brief description of a variety of methods.
This paper sought to provide a comprehensive analysis of a rater training program through the use of multifacet Rasch measurement. The purpose was to display how such an analysis can provide specific information on raters that is useful for feedback, and also important information concerning the performance of the rating form and training materials.
Team task analysis refers not only to an analysis of a team's tasks, but also to a comprehensive assessment of a team's teamwork requirements (i.e., knowledge, skill, ability, and attitude requirements). Like job analysis, team task analysis is important because it forms the foundation for team design, team performance measurement, and team training. Essentially, it is the building block for all "team" resource management functions.
This report summarizes prior efforts to provide guidance in resolving the key issues confronting the development of a national outcome-based reporting system for adult education programs.