Recent projects
rule

Project:
Development and validation of selection and classification tests for the U.S. Army (Project A)

Client:
U.S. Army Research Institute

Problem
rule

In the early 1980s Congress challenged the military services to validate the Armed Services Vocational Aptitude Battery (ASVAB), using on-the-job performance as a criterion. Previous ASVAB validity studies had used training data as the criterion.

The Army took this challenge one step further and sought:

  1. To develop new selection and classification tests and measures of job performance
  2. To validate the ASVAB and new measures against the job performance measures.

The Army sponsored two large-scale development and validation projects, Project A (concurrent validation study including more than 9,000 personnel) and Building and Retaining the Career Force (a longitudinal validation study of about 40,000 soldiers).

Approach
rule

AIR had prime responsibility for two components of this project. The first component was the creation and maintenance of a longitudinal research database and the design and implementation of validation analyses based on the data assembled.

The database included application, accession, training, and performance information from existing Army and Department of Defense sources on a total of several million Army applicants from FY-81. It also included additional data generated from the testing of 50,000 to 100,000 soldiers during the project.

In conducting analyses of these data, we designed and evaluated solutions to several significant statistical problems:
  • Restriction of range
  • Multiple criterion measures
  • Generalizing validity to all Army occupations
  • Assessing differential classification efficiency
  • Identifying race/gender bias
  • Cross-validation methodology.

We conducted concurrent and longitudinal validity analyses.

We then developed job performance measures for first- and second-tour soldiers in five Army occupational specialties. For each specialty, we identified a representative set of critical job tasks and then developed hands-on, job knowledge, and performance rating measures for each of these tasks. We planned and conducted the collection of these job performance measures in field tests and concurrent validation efforts.

The specialties for which AIR developed job measures included:
  • Radio/Teletype Operator
  • Medical Specialist
  • Administrative Specialist
  • Military Police
  • Light Wheel Vehicle Mechanic

Building the Career Force
rule

The extension of Project A, Building the Career Force, examined performance of soldiers in their second tour of duty who were beginning a career in the Army. The research goal of the Career Force project involved investigating job performance of soldiers over a longer period and examining measures that might improve the prediction of long-term performance.

AIR's responsibilities in this project were threefold:

  1. Develop and update criterion measures
  2. Create and maintain the project database
  3. Design and implement the validity analyses

First, we were responsible for developing and updating criterion measures for several jobs, such as Military Police, Medical Specialist, Administrative Specialist, and Vehicle Mechanic. The measures included hands-on work sample tests and pencil-and-paper measures of job knowledge. To develop the hands-on measures, we conducted task analyses to identify the observable steps necessary for task completion. We designed both hands-on and pencil-and-paper measures to be performance-based, rather than exclusively knowledge-based.

Second, we created and maintained the database. This included collecting data using pencil-and-paper and computerized testing methods and editing the data for obvious problems (i.e., values outside accepted ranges and incomplete identification information). In addition, we devised methods for handling missing data, including imputing predictor and criterion scores. From the main database, we prepared data files for analysts using single sources and combining various sources according to the analysis specifications.

Third, AIR had primary responsibility for designing and implementing the validation analyses. This involved assessing the validity of the predictors for predicting training performance and job performance at various stages in the soldiers' careers. We assessed the factors that define performance over time and evaluated measures of second-tour supervisory performance, such as role-plays and a pencil-and-paper measure.

Results
rule

The results showed that the ASVAB is a very good predictor of job performance, as were several of the new tests. Consequently, the military services have decided to add one of the tests we developed and validated in this effort to the ASVAB. Numerous research papers resulted from these projects, and a book is in press.


Back to top rule