Project Abstract:

Performance Measurement Authoring Tool

• Industry: Military
• Business Need: Team Training
• Client: NAVAIR Training Systems Division

The goal of this project was to develop a software authoring tool to serve as a job aid for guiding U.S. Navy instructors in developing their own measures of individual, team, and multi-team performance. A review of the literature provided guidance regarding the relevant variables in measuring training performance including the difference between training purposes (e.g., instruction versus evaluation), ways to assess behavior (e.g., accuracy, productivity), and available measurement formats (e.g., numeric scale, rating scale). Performance measurement experts were employed to build the business rules that PMAT uses to recommend measurement formats based on the assessment type chosen by the user. Naval aviators participated in cognitive interviews and usability tests with various versions of PMAT. Their performance and comments were used to improve the software’s content and usability.

The Performance Measurement Authoring Tool (PMAT, Version 1.1, completed June 30, 2004) has a built-in tutor that provides users with detailed guidance regarding the underlying logic of performance measurement as well as specific directions regarding how to use the software. PMAT is organized into four main functionalities:

• - A function that allows the user to build a library of taskwork tasks important in a training event and create measures for them.
• - A function that allows the user to build a library of teamwork tasks important in a training event and create measures for them.
• - A function that allows the user to build a library of training participants to help organize the key players in a training event.*
• - A function that allows the user to build a library of training events to help organize tasks and audiences in a coherent manner.*

These functionalities allow the user to build a comprehensive plan for conducting training that includes identifying and defining the behaviors of interest by training participant, deciding what should be assessed with regard to those behaviors, and then creating appropriate measures. PMAT is currently being incorporated into the Navy’s Debriefing Distributed Scenario- Based Exercises (DDSBE) project. However, its more global and timely appeal lies in the fact that it is a generic stand alone software tool that could easily be adapted to virtually any training environment in the Navy.

*The current R&D Version of PMAT has limited functionality in these areas.