Validating the System

Validating the System
•
What is the academic basis to the Insights Discovery system?
•
Has the Insights Discovery system been validated by official bodies of the psychology profession?
•
Is the Insights Discovery preference evaluator continually tested and updated?
Insights and Jung
External Assessment
The Insights Discovery System is solidly
based on the psychological types theory of Dr
Carl G Jung and one of his leading students,
Dr Jolande Jacobi. Jung proposed that our
personalities are created in the interaction
of four functions and two attitudes. When
combined, these elements generate eight
common personality types. Jung felt that what
makes an individual unique are the different
balances of these functions and attitudes
interacting within them. It is these balances
that make up the Insights 4 Colour System.
The four colours represent observable
behavioural patterns:
In 2005 the University of Westminster’s
Business Psychology Centre performed an
extensive study of the Insights Discovery
Evaluator English Version 3.0. Key statistical
analysis was used to study the IDE’s “item
analysis”, “norm data”, “reliability” and
“validity”.
Figure 2: Pyramid of key psychometric statistics
Validity
Construct Validity
Factor Analysis
Criterion Validity - Concurrent
Criterion Validity - Predictive
Reliability
Temporal Stability
For a more comprehensive review of the IDE’s
properties, please read “An Overview of the
Development, Validity and Reliability of the
English Version 3.0 of the Insights Discovery
Evaluator” produced at the University of
Westminster’s Business Psychology Centre
(bpc).
Test – re–test correlations
Internal Consistency
Split-half coefficients
Item-total correlations
Inter-item correlations
Cronbach Alpha coefficients
Data on Norms
Historic Development - Item Analysis
FIE
RY
H
RT
Showing
no bias
Objective
Detached
Positive
Affirmative
Bold
Assertive
Still
Tranquil
Calming
Soothing
Cheerful
Uplifting
Spirited
Buoyant
RE
EN
An Overview of the Development, Validity and
Reliability of the English Version 3.0 of the Insights
Discovery Evaluator
Develop
ment of
the
Figures
5 through evaluator usin
g item ana
Althoug
8
h the grap graphically sho
lysis has
w
there is
signific
hs
antly imp
a need for presented here the benefit of
evolvin
analysis.
are both
a more
g the item roved its quality
statistic
.
al approac visually impactf
pairs ove
ul
r time.
h to qua
Conseq
ntify the and intuitively
uen
appeali
discrim
The University of Westminster, 309 Regent Street, London, W1B 2UW, UK
ng,
inative
strength tly, t-tests wer
quality
e conduc
of
of the item
ted
distance the relationship
Dr. Corine van Erkom Schurink
between
s plotted on the item ana
lysi
the
in
Below is
an exampl two colour sco the graph. A t-te s data in order
to
con
res
st
e for the
gauge the
tinuous
Director of The Analytical Research Bureau (Pty) Ltd,
‘cool blue within one fram allows you to
statistic
dete
taken betw ly improving vers
e, are stat
al
’ items
from 4
ions of
een 31/1
istically rmine if the
samples
average
the evaluat
signific
16 Central Avenue, Pinelands 7405, South Africa
com
across all 1/2003 and 31/7
ant
ors. Thi
/2004.
25 fram
s English paring data acro or not.
This exa
es, have
ss 4
Version
For exa
mple is
scored 4
3.0 sam
Stewart Desson
mple, the
based on
or more
ple has
sha
resp
evaluat
(out of
been
or, the ‘coo ding of the firs
6) for ‘coo ondents who
,
t row of
on
than the
l
l blue’.
Head of Learning, Insights Learning & Development Ltd,
the
‘earth gree blue’ statement
(exact and table below sho
highligh
ts that alth n’ statement (cal
ws
good), they
m and eve precise) is success that in the 3 rd
Jack Martin Way, Dundee, DD4 9FF, Scotland, UK
ough the
fram
differen
n-tempe
are nev
ce betw
highligh
red). How fully selected mor e of the
ertheles
een them
ted as a
s close
e often
to
problem
is statistic ever, the t-test
empiric
or potentia each other (wh
ally driv
also
This paper presents comprehensive data on the psychometric properties of the Insights Discovery Evaluator (IDE).
ich is not ally significant
en item
l problem
analysis.
(which
ideal).
is a can
The IDE’s capacity to affect both reliable and valid measures of the primary personality preferences coded by the
is
Any stat
didate for
Table 1
ement
- T-tes
improv
model as four colours is assessed. It draws upon an extensive research and development programme undertaken
t on item
ement thro
analysis
ugh furt
of resp
between Insights Learning & Development Ltd. and the University of Westminster, aimed at the development of a
ondents
UK - Eng
her
who
lish
Dr. Stephen Benton
It is an
or (IDE).
ch
Evaluat
es in whi
covery
25 fram
ghts Dis
sisting of
lish Insi
scores the s in a
igh
of the Eng e) evaluator con st’ and then
4 item
The Ins
ion 3.0
rang
st’, a ‘lea 5. Each of the
tains vers normative (a
l blue’.
s a ‘mo
ix A con
e of 1 to
n’ and ‘coo
word pair
ice) and
r
scal
Append
gree
a
cho
fou
th
of
t on
‘ear
(forced
a choice
6
t and mos shine yellow’,
ipsative
between
oses from in between leas
re
‘sun
cho
,
r
sco
a
all
ry red’
the use
options
h giving
n across
called ‘fie
ng two
res, eac
tic mea
t5
erences
remaini
erence sco
ple arithme mple of the firs
sure pref
e
colour pref colours. A sim
frame mea
exampl
ws an exa
have 25
the four
or will
ure 2 sho frames. In this
each of
urs. Fig
ed evaluat
all 25
least), for of the four colo
from
ted.
A complet
ed
(for
0
h
t) and
rt produc
n highligh
ed for eac
bar cha
(for mos
ur has bee
colo
is calculat e of the colour
’s
es
the item
25 fram
exampl
or works,
and an
luat
eva
frames
the
ing how
explain
very
ts Disco
YE
LL
OW
CO
EA
G
UE
BL
D
RE
OL
Figure 1: The Insights Discovery 4-Type Wheel
r (IDE)
Evaluato
Director of The Business Psychology Centre (bpc),
E
IN
SH
N
SU
R25
psychometrically robust evaluator. The full methodological and statistical account of this programme may be
found in technical papers produced at the University of Westminster’s Business Psychology Centre (bpc).
Pair
Psychometric science endeavours to impose objectivity upon the measurement of both human performance andtested
Pair
7
aspects of personality. In order to do so, any questionnaire based on measurement of human behaviour, especially
that based on self-report, must be able to meet certain demonstrable criteria in order to be considered as an
Pair 19
Pair 22
PairIt25
objective measure. This paper sets out to explain these psychometric criteria in easily understandable terms.
is
Pair 26
the authors’ intention to make the statistics and arguments presented understandable to two different professional
B9 & Y9
B10 &
G10
B13 &
G13
B17 &
professional assessment of the evidence presented. Secondly, the wider community of Human ResourcePair 51 B17 G17
& R17
Pair 52
professionals require simple non-technical explanations of these same key statistics, in order for them to assess
the
B18 &
G18
Pair 70
B24 &
appropriateness of the IDE’s to their business. Four categories of information are presented covering ‘item
G24
These colours are measured by the Discovery
evaluator; a 25-frame questionnaire of 100
word pairs, which when completed produces
the Insights Discovery Personal Profile.
Designed by Andi Lothian in the early 1990’s,
Andi, and son Andy, founded Insights Learning
& Development Ltd in 1993.
output
es and exa
colour
luator fram
minant’
al’s ‘do
ple of eva
2 – Sam
an individu
Figure
cribed as
rt is des
bar cha
ur in this
est colo
d colour.
The high
erre
t pref
their mos
analysis’, ‘norms data’, ‘reliability’ and ‘validity’ as these form the bases for demonstrable objectivity.POT
Key
ENTIAL
6.29
10.10
11.85
-8.71
13.20
10.27
-1.09
8.84
0.000
0.000
0.000
0.000
0.000
0.000
0.276
0.000
0.000
3.59
0.000
0.000
scored
lish S2
0.000
7.12
0.000
9.33
22.98
0.000
0.000
12.60
15.67
-1.20
0.000
0.000
4 or more
UK - Eng
t value
on Sig.
of t (2paired
ttailed)
test
CI=95%
2.73
0.006
6.00
0.000
10.53
0.002
(out of
6) for cool
lish S2.2
32.73
26.45
33.10
38.39
t value
on
paire
d ttest
9.53
40.73
26.51
11.56
0.000
11.78
0.000
36.82
0.000
27.99
0.000
28.01
0.000
0.000
0.000
39.75
10.47
28.90
n a high
quality
uen
item ana
quoting
lysis doe
furt
s not nec
n selected her from Paul
essarily
Kline
by
ry, as has
been argu item analysis and in his Psychomet ensure an evaluat
ed, to sho
or
rics Prim
the resu
lts replicat
w the test
er (1997) is valid.
,
ed with
is valid
a new sam “After the
and reli
able”.
ple, it is
ersity
Conseq
tly,
Version 4.0
1 of 70
itemPage
s have bee
Issued by the University of Westminster’s Business Psychology Centre 22nd December 2005 www.wmin.ac.uk/sshl/page-339
necessa
of West
4.0
Version the University
Issued by
minster’s
Business
Psycholog
y Centre
22nd Dece
mber 2005
of 70
Page 6
9
hl/page-33
n.ac.uk/ss
www.wmi
Version
4.0
Issued by
the Univ
Insights Learning & Development Ltd
JackMartin
MartinWay,
Way,
Claverhouse
Business
Park, Dundee
DD4
9FF, Scotland,
UK
Jack
Claverhouse
Business
Park, Dundee
DD4 9FF,
Scotland,
UK
T:
(0)1382 908050
(0)1382 908051
E:[email protected]
www.insights.com
T:+44
+44(0)1382
908050F: +44
F: +44(0)1382
908051
E:[email protected]
www.insights.com
of West
minster’s
Business
Psycholog
y Centre
22nd Dece
mber 2005
www.wmi
blue
UK - Eng
t value
on Sig.
of t (2paired
ttailed)
test
CI=95%
11.85
0.000
23.99
0.000
25.80
0.000
9.23
0.000
18.40
0.232
PROBLE
13.91
M: The
-5.89
statistics
have been computed for each of these four areas and they have been benchmarked against international
0.000
two item
i.e. it is
s (t-tes
33.26
but the
standards. The paper concludes that the measurement of the four colours is both valid and reliable. PRO
distance t) are very clos
BLEM:
e (low
between
The two
positive
the two
items (t-tes
t-value),
items is
t-value),
t) are very
still sign
and the
ificant
close (low
distance
PROBLE
positive/
between
M: The
nega
the two
tive
two item
items is
s (t-tes
NOT sign
and the
t) are in
distance
ificant
inverted
between
order (neg
the two
items is/is ative t-value),
not sign
Howeve
ificant
r, eve
GLOBAL HEADQUARTERS
Insights Learning & Development Ltd
7.44
14.40
B8 & G8
B9 & G9
psychometricians require key statistics and supporting technical information, in order for them to make a Pair 49
profile
mple of
3.11
Pair 28
Pair 37
groups, both of which may have a need to work with the Insights Discovery Evaluator (IDE). Firstly,
UK - Eng
Blue
score>4
VS t value
other
on Sig.
of t (2paired
colors
ttailed)
test
CI=95%
B3 & G3
B7 & G7
Page 13
n.ac.uk/ss
of
hl/page-3370
9
lish S3.0
Sig. of
t (2tailed)
CI=95%
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Insights Discovery©: Validating the System
Item Analysis
There are 100 colour items spread over the
25 frames in the IDE. Item analysis involves
empirically testing the quality of these 100
items and replacing weaker items with better
ones. This is measured by assessing the
responses of participants with clear colour
preferences, i.e. those whose average across
all 25 frames is greater than 5 in one out of the
four colours (“Sunshine Yellow, “Fiery Red”,
“Earth Green” or “Cool Blue”). Figure 3 shows
one of the improvements made in the items
from IDE S1.0 (UK) to IDE S2.0 (UK).
Figure 3: Example of results
of item analysis - graphs show
‘before’ and ‘after’ item changes
Note: Green scoring higher than
yellow on a yellow item before
the alteration.
Insights Discovery Preference Evaluator
Expressive and hopeful
L 1 2 3 4 5 M
Decisive and immediate
L 1 2 3 4 5 M
Powerful and assertive
L 1 2 3 4 5 M
Analytical and discreet
L 1 2 3 4 5 M
Thinking and self-contained
L 1 2 3 4 5 M
Tolerant and laid-back
L 1 2 3 4 5 M
7. Demonstrative and persuasive
L 1 2 3 4 5 M
Questioning and reflective
L 1 2 3 4 5 M
Logical and controlled
L 1 2 3 4 5 M
Immediate and initiating
L 1 2 3 4 5 M
Task-focused and confronting
L 1 2 3 4 5 M
Stable and concerned
L 1 2 3 4 5 M
Discussing and spontaneous
L 1 2 3 4 5 M
L 1 2 3 4 5 M
Discreet and valuing
L 1 2 3 4 5 M
Faithful and helping
L 1 2 3 4 5 M
Considered and
L 1 2 3 4 5 M
Consistent and correct
L 1 2 3 4 5 M
Challenging and determined
L 1 2 3 4 5 M
Figure 4: Dominant Colour
Spread of 75 CFOs
As expected, those in charge of financial
data tend to have a preference for objective
thinking, using facts and performing detailed
analysis. It should be noted whilst this data
indicates that people in certain roles tend to
have a preference for certain colours, it does
not correlate or necessarily relate to how well
they are doing their job or how capable they
are in fulfilling that role.
Reliability
Cronback-Alpha coefficients, α, measure
the error variance on the average inter-item
correlations. When the error variance is
low, which is desirable, the alpha coefficient
approaches 1.00. A value of 0.70 is the
commonly accepted inferior limit.
Analysing 24,224 completed evaluators shows
the four colours to have very high CronbachAlpha coefficients, providing evidence of
reliability as shown in Table 1.
L 1 2 3 4 5 M
L 1 2 3 4 5 M
19. Systematic and
L 1 2 3 4 5 M
L 1 2 3 4 5 M
Encouraging and valuing
L 1 2 3 4 5 M
Question
Responsive and extraverted
L 1 Numbers
2 3 4 5 M 1-25
Results-oriented and fast
L 1 2 3 4 5 M
Counselling and caring
Courageous and independent
Count of ‘Most Lresponses
for people with
1 2 3 4 5 M
Total Sunshine
Yellow score > 5
L 1 2 3 4 5 M
16. Animated and persuasive
L 1 2 3 4 5 M
Decisive and immediate
Powerful and assertive
L 1 2 3 4 5 M
Analytical and discreet
Consistent and correct
14. Strong-willed and purposeful
9. Reasoned
Sensitive and
anddiplomatic
particular
Precise
and
deliberate
Eager and
engaging
Encouraging
andsensitive
valuing
Concerned and
Results-oriented and fast
15. Systematic and principled
L
L
LL
1
1
1
1
2
2
2
2
3
3
3
3
LL 1
1 2
2 3
3
LL 1
1 2
2 3
3
L 1 2 3
L 1 2 3
LL 1
2
1 2 3
3
LL 1
1 2
2 3
3
LL 1
1 2
2 3
3
L 1 2 3
L 1 2 3
LL 1
2
3
1 2 3
LL 1
1 2
2 3
3
LL 1
1 2
2 3
3
L 1 2 3
L 1 2 3
4
4
4
4
5
5
5
5
M
M
M
M
4
4 5
5 M
M
4
4 5
5 M
M
4 5 M
4 5 M
4
5
M
4 5 M
4
4 5
5 M
M
4
4 5
5 M
M
4 5 M
4 5 M
4
5
M
4 5 M
4
4 5
5 M
M
4
4 5
5 M
M
4 5 M
4 5 M
Fun-loving and popular
L 1 2 3 4 5 M
Steadying and moderating
L 1 2 3 4 5 M
Table 1: Cronbach-Alpha Coefficients for
S3.0 (UK)
FastIDE
and reinforcing
L 1 2 3 4 5 M
Before
L 1 2Changes
3 4 5 M
Name:__________________________________
L 1 2 3 4 5 M
Upbeat and hopeful
Social
andand
cheerful
Analytical
evaluating
Faithful
helping
Bold andand
objective
0.92
18. Influential and informal
Precise and deliberate
Immediate
and
initiating
Unassuming
and
responsive
Stable and concerned
13. Constant and attentive
8. Influencing
Resolute and
confident
and expressive
α
L 1 2 3 4 5 M
Social and cheerful
Thinking and distant
12. Analysing and painstaking
7. Friendly
Demonstrative
and persuasive
and entertaining
Questioning
and robust
reflective
Competitive and
Cool
Blue
17. Patient and empathetic
L 1 2 3 4 5 M
Insights Discovery Preference Evaluator
N=
24,224
16. Animated and persuasive
8. Resolute and confident
9. Sensitive and diplomatic
Data on Norms
Count of ‘Most responses for people with
L 1 2 3 4 5 M
Total Sunshine Yellow score > 5
L 1 2 3 4 5 M
6. Accommodating and helping
The norm data for the IDE is of good quality,
being segmented by the language of the
evaluator completed; the country a respondent
is based in;
age (in ten-year
bands) and
occupation. The
occupational
norm data
provides
predictive
validity. The
dominant colour
spread of
Chief Financial
Officers is
predominantly
‘Cool Blue’ and
‘Fiery Red’.
Name:__________________________________
6. Relating and amenable
L 1 2 3 Table
4 5 M
Tolerant and laid-back
22. Objective and daring
17. Relaxed
Patient and
andempathetic
peaceful
Logical and
Factual
and controlled
conventional
Task-focused
and confronting
Lively and congenial
Discussing and spontaneous
23. Animated and enthusiastic
18. Driving
Influential
and
informal
and
realistic
L 1 2 3 4 5 M
L
L
LL
1
1
1
1
2
2
2
2
3
3
3
3
LL 1
1 2
2 3
3
LL 1
1 2
2 3
3
L 1 2 3
L 1 2 3
LL 1
2
1 2 3
3
Discreet
and valuing
Compassionate
and considerate LL 1
1 2
2
Considered
LL 1
Detailed andand
attentive
1 2
2
Challenging and determined
L 1 2
24. Supporting and steady
L 1 2
19. Independent
Systematic and
L
1
and bold
L 1 2
2
4
4
4
4
5
5
5
5
M
M
M
M
4
5 M
4 5
M
4
4 5
5 M
M
4 5 M
4 5 M
4
5
4 5 M
M
3
4 5
3 4
5 M
M
3
3 4
4 5
5 M
M
3 4 5 M
3 4 5 M
3
4
5
3 4 5 M
M
Courageous
and
independent LL 1
Reflective and
thorough
1 2
2 3
3 4
4 5
5 M
M
Responsive
LL 1
2
Good-mixer and
and extraverted
lively Question
1 Numbers
2 3
3 4
4 5
5 M
M 1-25
Counselling and caring
25. Cautious and accurate
L 1 2 3 4 5 M
After
Changes
L 1 2 3 4 5 M
Forthright and straightforward
L 1 2 3 4 5 M
Optimistic and upbeat
L 1 2 3 4 5 M
2: Average
AcceptingFactor
and loyalLoadings.
L Completed
1 2 3 4 5 M in UK 31/11/03
YOU HAVE COMPLETED THE EVALUATOR, PLEASE
ENSURE EVERY
BEEN ALLOCATED
to 2/7/04.
IDEFRAME
S3.0HAS(UK)
Earth WHENSunONE ‘M’, Fiery
ONE ‘L’AND TWO DIFFERENT VALUES
SELECTED FROM 1, 2, 3, 4 OR 5.
Discovery Evaluator 2 Page
(B&W) English UK V3.0 250603Factor 1
Green shine
Red
N=20,948
© Copyright 1992-2003 Andrew Lothian, Insights. All rights reserved.
Insights, Jack Martin Way, Claverhouse Business Park, Dundee DD4 9FF. Tel: 01382-908050 Fax: 01382-908051
Yellow
Earth Green
0.56
0.92
12. Analysing and painstaking
0.93
0.92
Competitive and robust
Friendly and entertaining
Unassuming and responsive
L 1 2 3 4 5 M
L 1 2 3 4 5 M
22. Objective and daring
Relaxed and peaceful
L 1 2 3 4 5 M
L 1 2 3 4 5 M
Sunshine
Factual and conventional-0.05
L 1 2 3 4 5 M
L 1 2 3 4 5 M
Yellow Lively and congenial
L 1 2 3 4 5 M
Factor 2
0.06
0.32
L 1 2 3 4 5 M
13. Constant and
attentive
L 1 2 3 4 5 M
Test-retest reliability is determined
through
theL 1 2 3 4 5 M 23. Animated and enthusiastic
Influencing and expressive
L 1 2 3 4 5 M
Driving and realistic
L 1 2 3 4 5 M
Cool Blue
-0.09
-0.57
Analytical and evaluating
administration of the same evaluator
across L 1 2 3 4 5 M
Compassionate and considerate L 1 2 3 4 5 M
Bold and objective
L 1 2 3 4 5 M
Detailed and attentive
L 1 2 3 4 5 M
Fiery
Red
-0.59
0.19
time. It helps gauge how robust
the items are.L 1 2 3 4 5 M 24. Supporting and steady
14. Strong-willed and purposeful
L 1 2 3 4 5 M
Reasoned
and
particular
L
1
2
3
4
5
M
Independent
and
bold
L
1
2
3
4
5
M
Such tests are generally expected to yield
Eager and engaging
L 1 2 3 4 5 M
Reflective and thorough
L 1 2 3 4 5 M
and sensitive
Good-mixer and lively
L 1 2 3 4 5 M
reliability coefficients ranging Concerned
between
0.70 L 1 2 3 4 5 M
Assessments/Validations
15. Systematic and principled
L 1 2 3 Ongoing
4 5 M
25. Cautious
and accurate
L 1 2 3 4 5 M
and 0.90.
Fun-loving and popular
L 1 2 3 4 5 M
Forthright and straightforward L 1 2 3 4 5 M
Steadying and moderating
L 1 2 3 4 5 M
Optimistic and upbeat
L 1 2 3 4 5 M
Fast and reinforcing
L 1 2 3 The
4 5 M constant
Accepting evolution
and loyal
1 2 3 4 5 M
ofL language
means
The results of the test-retest analysis
WHEN YOU HAVE COMPLETED THE EVALUATOR, PLEASE
FRAME HAS BEEN ALLOCATED
thatENSURE
theEVERY
Discovery
Evaluator will continually
performed on the four colour scores
ONE show
‘M’, ONE ‘L’AND TWO DIFFERENT VALUES
SELECTED FROM 1, 2, 3, 4 OR 5.
evolve.
Insights
Discovery Evaluator
2 Page (B&W) The
English UK
V3.0 250603 Research Team monitors
very high reliability, translating into coefficients
© Copyright 1992-2003 Andrew Lothian, Insights. All rights reserved.
Insights, Jack Martin Way, Claverhouse Business
Park,Insights
Dundee DD4 9FF.
Tel: 01382-908050 Evaluator
Fax: 01382-908051 in its 30+
the
Discovery
ranging from 0.81 to 0.86 for the Pearson
languages on a daily basis. If you require
correlation coefficients and 0.89 to 0.92 for the
any information on this process, please
Cronbach-Alpha reliability coefficients for the
contact [email protected]. Examples of
same data.
the Statistical Reliability and Validity tests
can be obtained from Insights Learning &
Validity
Development Ltd on request.
Confirmatory Factor Analysis was used to
test the hypothesised factor structure of the
Insights Discovery model. Specifically it is
hypothesised that the four sets of 25 colour
based items in the IDE should load onto the
factors such that the polar opposite nature
of the ‘Fiery Red’ vs. ‘Earth Green’ items is
apparent and the polar opposite nature of
the ‘Sunshine Yellow’ vs. ‘Cool Blue’ items is
apparent. The four colours should load onto
their appropriate factor only.
The results in Table 2 successfully confirm this
hypothesis.
Conclusion
Over the past decade, the work of Carl Jung
has attracted increasing interest as people
seek to improve interpersonal dynamics on
both personal and professional levels. The
scientific research of the Discovery System
demonstrates that both Jung’s original
typology theory, and Insights’ ongoing research
to enhance it, has both strong psychological
foundations, and modern scientific application.
Depth psychology and empirical science unites
within the Insights Discovery System.