Team Performance

Evaluation of Medical Team Training

The ultimate goal of this project is to study the efficacy of team training, a new and emerging patient safety practice. Medical errors cause an estimated 98,000 deaths each year according to the Institute of Medicine’s report, To Err is Human. The IOM recommended establishing interdisciplinary team training programs for healthcare providers that incorporate proven methods from aviation. Healthcare is now beginning to apply the principles of crew resource management (CRM)—a training approach developed over 30 years ago in commercial aviation industry. CRM focuses on developing effective teamwork skills and communication to prevent medical errors.

Team training provides a set of tools, methods, and content to build skills in communication and collaboration, such as sharing patient assessments. Interdisciplinary healthcare providers from each hospital department (such as the emergency department, operating room, etc.) receive training as a team.

AIR staff began work on the project by convening an expert panel of national experts in patient safety, aviation, human factors, and industrial and organizational psychology to provide background information and direction for the project. They then conducted acomprehensive literature review and prepared a Technical Report - Medical Teamwork and Patient Safety: The Evidence-based Relation for AHRQ. Another report, Medical Team Training Programs: An Independent Case Study Analysis, is a comprehensive study of three existing medical team training programs identifying the current “best practices” in medical team training. AIR also prescribed a series of recommendations for the future design and implementation of medical team training. AHRQ has endorsed these recommendations, and AIR is in the process of developing a team training program that will be made readily available to public and private hospitals.

Other project goals are to develop a toolkit of resource materials that will support the team training program and to publish an edited book currently titled, The Handbook for Improving Patient Safety through Medical Team Training.

Aviation Safety Action Program

Between 60-80% of all aviation accidents are caused by human error. In this project, we are developing a system to classify human error on the flight deck. When complete, we will use this information to help the airline community develop more effective training programs.

In 2002, NAVAIR Orlando Training Systems Division contracted with the American Institutes for Research to develop a software tool that would serve as a job aid for guiding U.S. Navy training instructors in developing psychometrically sound measures of individual, team, and multi-team performance. The goal was to improve the definition and measurement of performance during training, which in turn would assist in improving the quantity and quality of feedback delivered during debrief. A review of the literature provided guidance regarding the relevant variables in measuring training performance including the difference between training purposes (e.g., instruction versus evaluation), ways to assess behavior (e.g., accuracy, productivity), and available measurement formats (e.g., numeric scale, rating scale). Performance measurement experts were employed to build the business rules that the software tool uses to recommend measurement formats based on the assessment type chosen by the user. Naval aviators participated in cognitive interviews and usability tests with various versions of software. Their performance and comments were used to improve its content and usability.

The resulting software tool, the Performance Measurement Authoring Tool (PMAT, Version 1.1, completed June 30, 2004) has four main functionalities as outlined below (Note that tutorials are available throughout each of these processes that provide users with detailed guidance regarding the underlying logic of performance measurement as well as specific directions regarding how to use the software).

A function that allows the user to build a library of taskwork tasks important in a training event and create measures for them.
A function that allows the user to build a library of teamwork tasks important in a training event and create measures for them.
A function that allows the user to build a library of training participants to help organize the key players in a training event.*
A function that allows the user to build a library of training events to help organize tasks and audiences in a coherent manner.*

Combined, these four functionalities allow the user to build a comprehensive plan for conducting training that includes identifying and defining the behaviors of interest by training participant, deciding what should be assessed with regard to those behaviors, and then creating appropriate measures. Using these measures to collect performance data during training will yield specific information regarding trainee performance, which may ultimately increase communication and learning during debrief.

PMAT is currently being incorporated into the Navy’s Debriefing Distributed Scenario- Based Exercises (DDSBE) project. However, it’s more global and timely appeal lies in the fact that it is a generic stand alone software tool that could easily be adapted to virtually any training environment.

Human Performance Measurement Thresholds

Changing mission requirements and budget cuts require Navy shipboard trainers to continually improve the efficiency and effectiveness of the training that they provide. The purpose of this project is to research the workload limitations of shipboard trainers. The results will be used to develop a series of strategies and tools for reducing trainer workload and tools that support shipboard training data collection.

Building on the job analysis data collected from the PMAT project, we described the characteristics of the shipboard training environment. We then used this information to develop and test a series of hypotheses regarding the best ways to minimize trainer workload. Finally, we developed tools and practices for reducing the workload of Navy shipboard trainers. These tools are being tested to assess their effectiveness.