Skip to main content
  • About Us
  • Careers
  • Contact

Search form

American Institutes for Research

  • Our Work
    • Education
    • Health
    • International
    • Workforce
    • ALL TOPICS >
  • Our Services
    • Research and Evaluation
    • Technical Assistance
  • Our Experts
  • News & Events

You are here

  • Home
22 Aug 2002
Report

Pilot Instructor Rater Training: The Utility of the Multifacet Item Response Theory Model

From the International Journal of Aviation Psychology

In this article we use a multifacet measurement technique, the multifacet Rasch model, to analyze the results of an IRR-training program. Our approach is an alternative to the procedures—congruency, consistency, agreement (rwg), sensitivity, and systematic differences—currently used during IRR training within the airline industry. We believe that this multifacet procedure can improve the quality of pilot instructor training by providing pilot instructors with important information that is not available with other techniques. We used the multifacet Rasch method instead of generalizability (G) theory, another multifacet technique.

Similar to multifacet Rasch analysis, G-theory provides information about facets—pilot instructors, videotapes of aircrews used in IRR training, and LOS grade sheets—and their interactions with one another. However, G-theory partitions the variance attributable to each of these facets using an analysis of variance (ANOVA) framework and thus focuses on groups as the unit of analysis (i.e., whether or not pilot instructors as a group are reliable or unreliable as opposed to the performance of a particular instructor undergoing IRR training). Unlike the multifacet Rasch model, G-theory is a classical test theory model.

Within classical test theory, an aircrew’s performance ratings on a LOS are a function of their true score (i.e., true performance) and error (i.e., instructor variability and grade sheet variability). The larger the error component, the less reliable the LOS grades. Generalizability studies partition this error term into identifiable components. By identifying potential sources of error and the magnitude of each source’s contribution to the error component, steps can be taken to reduce error and improve reliability. Inasmuch as high reliability is the goal, reducing variability caused by instructors, grade sheets, and crews are important components in achieving that goal. Within the air-carrier industry, IRR training has been the primary strategy used for error reduction. 

The multifacet Rasch technique is an item response theory (IRT) model that focuses on individual elements of the LOS assessment process. This model is useful for pilot instructor rater training because it provides individual-level, as opposed to group-level, information that can be directly fed back to individual pilot instructors. Information about the LOS grade sheet and the videotapes used for practice and feedback can also be gleaned from this analysis. 

PDF icon Pilot Instructor Rater Training: The Utility of the Multifacet Item Response Theory Model (PDF)

Further Reading

  • Using Multifacet Rasch Analysis to Examine the Effectiveness of Rater Training
  • AIR Experts Discuss Workplace Issues at the Annual Meeting of the Society for Industrial and Organizational Psychology
  • A Practitioner’s Insights on Student Misconceptions and Errors
  • Post-Training Feedback: The Relative Effectiveness of Team- Versus Instructor-Led Debriefs
  • Piloting the Good Behavior Game in Brazil
Share

Topic

Workforce
Human Capital

RESEARCH. EVALUATION. APPLICATION. IMPACT.

About Us

About AIR
Board of Directors
Leadership
Experts
Clients
Contracting with AIR
Contact Us

Our Work

Education
Health
International
Workforce

Client Services

Research and Evaluation
Technical Assistance

News & Events

Careers at AIR


Search form


 

Connecting

FacebookTwitterLinkedinYouTubeInstagram

American Institutes for Research

1400 Crystal Drive, 10th Floor
Arlington, VA 22202-3289
Call: (202) 403-5000
Fax: (202) 403-5000

Copyright © 2021 American Institutes for Research®.  All rights reserved.

  • Privacy Policy
  • Sitemap