Adaptive Personality Scales: Initial Validation Results

Computer Adaptive
Personality Scales:
A Military and Civilian Application
Walter C. Borman
PDRI, an SHL Company
and University of South Florida
Tracy Kantrowicz
PreVisor, an SHL Company
Outline
• Description of the “adaptive test”
concept
• Development of NCAPS and GPIAdaptive
• Initial validation results
• Conclusions and next steps
1
Description of the
“Adaptive Test” Concept
• Initial ability domain application
• Our work began in the performance
domain
• Forced-choice Computer Adaptive
Rating Scales (CARS)
• Laboratory study showed lower
standard error measurement and higher
validity and accuracy for CARS
CARS Performance Estimation
Process
Very
Ineffective
Somewhat
Ineffective
Effective
Very
Effective
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
First Item Pair
Second Pair
Third Pair
Fourth Pair
Final Estimated
Performance
2
Development of NCAPS
• Idea was to apply CARS concept to
non-cognitive testing
• 19 potential personality constructs were
identified
• Psychologists rated their importance for
79 Navy jobs
• 10 constructs selected based on means
and SDs
Development of NCAPS (Cont.)
• PDRI generated 1725 items at all trait
levels
• Construct and trait level “retranslation”
was conducted
• 1494 items survived, 106-199 per
construct
3
Example Items for
Social Orientation
• It is easy for me to find something in
common with any person I meet (M =
6.36)
• It takes real effort for me to hide my
impatience with people who aren’t very
bright (M = 1.49)
• I am able to make friends when I put
some effort into it (M = 3.63)
Validation Results
• Concurrent validation study designed
with  110 first tour Navy Sailors
• NCAPS and a conventional personality
inventory administered and supervisor
performance ratings gathered on nine
“Navy-wide” dimensions
• Composite of the ratings was reliable
(interrater r = .53)
4
Validation Results (Cont.)
• Unit-weighted composite of 10 NCAPS
conventional against composite overall
performance (r = .13 uncorrected;
r = .18 corrected)
• Unit-weighted composite of 10 NCAPS
adaptive against composite overall
performance (r = .27 uncorrected;
r = .37 corrected)
Validities Associated With Different Numbers of Item-Pairs
(Dependability)
0.25
Validity Coefficient
0.20
0.15
0.10
0.05
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Number of Item-Pairs
5
Civilian Version:
Global Personality Inventory-Adaptive
• Identified 13 dimensions for item
development
• PDRI and PreVisor generated 2679
items for 13 dimensions
• Construct and trait level retranslation
was again conducted
• 2519 items “survived the cut,” 150-200
per construct
Constructs Identified for GPI-Adaptive
compared to NCAPS Constructs
GPI-Adaptive
NCAPS
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Achievement Striving
Social Orientation
Cooperation
Composure
Adjusting to Change
Positive Self-Concept
Responsibility
Thoroughness
Sense of Duty
Self-Sufficiency
Learning Orientation
Directing and Influencing
Innovation
Achievement
Social Orientation
Stress Tolerance
Adaptability/Flexibility
Attention to Detail
Dependability
Dutifulness/Integrity
Self Reliance
Willingness to Learn
Vigilance
6
Validation Results
• Concurrent Validity Study Against Supervisor
Performance Ratings (N=1340)
• Unit-Weighted Composite Against Overall
Performance (r=.28, corrected for criterion
unreliability)
• Validities Asymptote at 5-8 Item Pairs
• Testing Time ½ that of Conventional
Inventory
7