Improving the Construct Validity of Line Operational Simulation (LOS) Ratings: Lessons Learned from the Assessment Center
Line Operational Simulations (LOS) are commonly used for training and evaluating pilot crews under realistic conditions. Despite their widespread use, the construct validity of LOS ratings remains largely unexplored. This article provides research-based guidelines for improving the construct validity of LOS ratings.
Because poor LOS construct validity can have real-world effects on pilot training and performance, assessing and improving the construct validity of LOS is more than just an academic or scientific issue. It is also a practical and political issue in that it involves multiple stakeholders who may have competing concerns. These include safety, justice/fairness, technical feasibility, and cost-effectiveness.
Therefore, we recommend that all potential stakeholder groups be involved in identifying and improving the construct validity of Line Operational Simulations. These groups may include pilot unions, training staff, flight standards staff, and officials from the regional FAA offices. Moreover, all groups must be prepared to compromise some of their own goals/needs to achieve a balanced solution. In the end, only by working together can industry address the issue of LOS construct validity, and by extension, the quality of pilot crew training and evaluation.
As with any study, this one has its limitations. First, the available data concerning the construct validity of LOS ratings is limited to a handful of samples, all of which were conducted as LOEs. Second, most of the guidelines for improvement have only been tested with assessment centers, not LOSs. As a result, these preliminary results will need to be verified with other forms of LOS, such as Line-Oriented Flight Training (LOFT). Nevertheless, we hope that at a minimum, this paper will prompt the industry to further explore this important topic. As we noted earlier, the construct validity of LOS ratings is not merely an academic or scientific issue; it has important implications for pilot crew training and evaluation.