docCorrelates of Assessment Center Performance Across Contexts
download 
Report Transcription
Correlates of Assessment Center Performance Across Contexts
Correlates of AC Performance across Contexts: An Investigation of Aggregated Measures of Executive Performance Jing Jin & Doug Reynolds Development Dimensions International (DDI) 1 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Overview • Common findings regarding personality and executive performance findings • Sample and methods, machine learning focus • Results across several methods 2 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Personality Correlates of Executive Performance Approach Representative Finding Single trait studies Narcissism correlates with exec performance (e.g., Chaterjee & Hambrick, 2011) Extraversion and Openness to Experience relate to leadership performance (e.g., Judge et al., 2002) Big-5 metaanalyses Exec performance studies 3 Ambition (E), Learning orientation (OE), adjustment (N), show stronger correlations than other traits for exec performance (Hogan, 2008) Extraversion, Openness, correlate with exec experience and strategic thinking competency (Dragoni et al., 2011) © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Sample and Methods • Executive assessment results aggregated across industries and regions • Assessments included behavioral measures of executive competencies and personality tests • Machine learning techniques used to explore resulting database 4 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Data source: tech-facilitated assessment center Key characteristics: • Day-in-the-life format • Paired personality testing • Live interactions • Deployed globally 5 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Karen Gates Vice President , Operations Interpersonal Skills P Compelling Communication P Cultivating Networks S Navigating Politics D Influence Business Management D Building Organizational Talent S Driving Execution P Financial Acumen P Operational Decision Making P Entrepreneurship P Establishing Strategic Direction Leadership PERSONALITY PATTERNS COMPETENCIES Started: 02.04.2000 Previous Position: Director, Operations Education: MBA, Wharton Business School Known Aspirations: VP of Eastern Region Sample Output Leadership Effectiveness Inventory Adjustment 72 Ambition 87 Sociability 25 Interpersonal… 100 Prudence 98 Inquisitiveness 88 Learning Orientation 95 Leadership Challenge Inventory Volatile Argumentative 33 10 Risk Averse S Leading Change P Coaching and Developing Others Avoidant S Selling the Vision Arrogant P Empowerment/Delegation 55 Imperceptive Impulsive 24 15 13 11 Attention Seeking Personal Competencies P Executive Disposition S Passion for Results 53 Eccentric Perfectionistic Approval Dependent 69 24 72 Multi-organization analyses Aggregated database Strengths 22,212 Executives Strategic Leaders Establishing Strategic HRIS Direction Financial/Business Acumen Entrepreneurship Company Profiles Strengths Strong Managers Driving for Results Vary by: Prior Assessments ExecutiveLevel? Disposition Communicating with Industry? Impact Experiences? Decision Making Personality profile? Customer Focus Company Performance Machine Learning/Data Mining: When to Use? • Noisy predictor and criterion data • Minimal theoretical grounding • Limited experimental control 8 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Analysis design: • Executive assessment database • Predictors: 18 personality scales • Outcome: – Strategic vs. Non-Strategic profiles – Interpersonal vs. Non-Interpersonal profiles • Predictive Accuracy: Cross-Validation 9 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Variable Importance: Comparison of Three Methods Traditional Regression Relative Importance Random Forests Multicollinearity X √ √ Non-normality X X √ Non-linearity X X √ No A-Priori-Model X X √ 10 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Example: Classification Tree Buy BMW (Y/N)? Income (>100K) Income (<100K) Attend sport event (0) Buy other lux. item (<2) Attend sport event (<4) 11 Attend sport event (>0) Buy lux. item (>2) Attend sport event (>=4) © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Study 1: Predict Strategic Profiles • Predictors: 18 personality traits (pos. & neg.) • Criterion: ratings on strategic competencies Business Acumen Establishing Strategic Direction Entrepreneurship 12 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Example using alternative methods Study 1: Sample 9,566 Executives w/strategic profile data Weak Strategists n = 2219 13 Strong Strategists n = 1391 Vary by: Personality profile? © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Study 1: Variable Importance Random Forests: Logistic Regression: Relative Importance: LearningOrientati… LearningOrientation Ambition Ambition Ambition LearningOrientation Argumentative RiskAverse RiskAverse Volatile AttentionSeeking Perfectionistic Imperceptive Avoidant Avoidant RiskAverse Inquisitiveness Inquisitiveness Perfectionistic Argumentative Argumentative Arrogant Perfectionistic Sociability Inquisitiveness Sociability AttentionSeeking AttentionSeeking ApprovalDependent Adjustment Avoidant Volatile Arrogant Adjustment Imperceptive Prudence Prudence Arrogant Volatile Imperceptive ApprovalDependent InterpersonalSensitivity Eccentric Impulsive Impulsive Eccentric InterpersonalSensitivity Sociability Prudence InterpersonalSen… ApprovalDepend… Eccentric Adjustment Impulsive Accuracy: 65.05% Accuracy: 66.02% Study 1: Feature Selection 70% 69% Cross-Validation Accuracy 68% 67% 66% 65% 64% 63% 62% 61% 60% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Number of Predictors 15 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Study 1: Sample Classification Tree Ambition (Dominance) >= 51 < 51 Non-Strategic(79%) Learning Orientation < 83.5 >= 83.5 Learning Orientation < 41 Strategic (58.8%) >= 41 Risk Averse Non-Strategic(70.2%) > 22 Non-Strategic(60.4%) <= 22 Strategic (56.0%) ACCURACY: CART: 71.76% LR: 66.10% Study 2: Predict Interpersonal Profiles • Predictors: 18 personality traits (pos. & neg.) • Criterion: ratings on interpersonal competencies Cultivating Networks Communicating with Impact Influence 17 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Example alternative methods Study 2: using Sample 9,739 Executives w/interpersonal profile data Weak Interpersonal n = 1100 18 Strong Interpersonal n = 1365 Vary by: Personality profile? © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Study 2: Variable Importance Logistic Regression: Relative Importance: Random Forests: Ambition Ambition AttentionSeeking AttentionSeeking AttentionSeeking Ambition Argumentative Argumentative Argumentative LearningOrientation Inquisitiveness Perfectionistic RiskAverse Perfectionistic Inquisitiveness Perfectionistic RiskAverse Sociability Sociability Avoidant Inquisitiveness Arrogant Adjustment Avoidant ApprovalDependent Impulsive Imperceptive LearningOrientation Arrogant Adjustment Impulsive Prudence Volatile Adjustment Volatile Eccentric Avoidant RiskAverse Arrogant Eccentric Imperceptive InterpersonalSensitivity Prudence Impulsive Imperceptive Sociability ApprovalDependent Volatile Eccentric Prudence InterpersonalSensitivity LearningOrientati… InterpersonalSen… ApprovalDepend… Accuracy: 64.27% Accuracy: 64.18% Study 2: Feature Selection 69% Cross-Validation Accuracy 67% 65% 63% 61% 59% 57% 55% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Number of Predictors 20 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. 16 17 18 Study 2: Sample Classification Tree Ambition (Dominance) ≥ 59.5 < 59.5 Non-Interpersonal Attention Seeking (62.2%) < 66.5 ≥ 66.5 Interpersonal Argumentative ACCURACY: ≥ 65.5 CART: 62.68% LR: 61.66% (68.2%) < 65.6 Non-Interpersonal Interpersonal (56.8%) (62.9%) 21 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved. Conclusions: • Personality variables have consistently shown relationships with executive performance • • Strategic: ambitious, willing to learn, not afraid of risks Interpersonal: ambitious, willing to be on the spot, not argumentative • Complex relationships among variables may be obscured by traditional analyses • Machine learning and “big data” analytics provide useful tools for extending our understanding of these complex relationships 22 © Development Dimensions Int’l, Inc., MMXIV. All rights reserved.