SE-ROI Report 1006.pptx

Honourcode, Inc.
Systems Engineering
Return on Investment
SE-ROI Research
Near-Final Results Jun 10
Eric Honour
+1 (850) 479-1985
[email protected]
Funding provided by
•  Honourcode, Inc.
•  DASI (Univ of South Australia)
Agenda
 
SE-ROI Project
  Motivation: How much is enough?
  Goals and methodology
 
SE-ROI Results
  Demographics
  Primary Correlation Relationships: Success v. SE
  Eight SE Activities
  Right-Sizing SE
Honourcode, Inc.
Bottom Line
 
Better programs expend
 
 
 
All SE activities correlate well with
 
 
 
Stakeholder acceptance
Cost/schedule control
No SE activities correlate with
 
 
more SE effort overall
more mission definition, more tech leadership
System technical quality
SE today leads to better programs
– but does not lead to better systems.
Results can be used to right-size SE
Honourcode, Inc.
Honourcode, Inc.
SE-ROI Project
Methodology
Industry support
Heuristic Claim of SE
 
Better systems engineering leads to
 
 
 
Better system quality/value
Lower cost
Shorter schedule
Traditional Design
SYSTEM DETAIL PRODUCTION
DESIGN DESIGN INTEGRATION
Risk
Time
TEST
Risk
“System Thinking” Design
Saved
Time/
Cost
Time
Not Known: How Much Is Enough?
Honourcode, Inc.
Project Goals
 
Research objectives
How Much Is Enough?
 
Find out how much of what type of SE correlates with
project success
• 
 
Leading indicators
• 
 
Used during a project to assess the project’s expected future
success and risks based on SE practices used.
Identification of good SE practices
• 
 
What SE practices are appropriate under what conditions.
Appropriate to generate success under different conditions.
Schedule
 
 
 
 
’05-’07 – Technical structuring and definitions
Late ’07 – Started data gathering
Internal reports ’08–’09
Final reports ’10
Honourcode, Inc.
SE-ROI Project
Interviews
• Just-completed programs
• Key PM/SE/Admin
• Translate program data
into project structure
• Program characterization
• Program success data
• SE data (hours, quality,
methods)
Desired Results
1.  Statistical correlation
of SE practices with
project success
2.  Leading indicators
3.  Identification of good
SE practices
Statistical correlation
Honourcode, Inc.
Company Participation
 
Data gathering – minimal impact
 
 
 
 
 
Select 2 to 4 programs
One day of interviews
2-hour sessions with PM+SE of each program
Strong protection of proprietary data
Reports – effective program benchmarking
 
 
Benchmark report within 30 days of session
•  Compares programs against prior data
Quarterly reports from all prior data, all sources
•  Correlations found
•  Leading indicators proven
•  SE practices proven
Honourcode, Inc.
Current Status – Jun 2010
 
 
 
 
 
SE ontology from SE standards –
wide-spread, acceptable terminology
Develop interest base from possible
interview sources (currently ~65)
Create interview data sheets and vet
them through sample interviews
Start program interviews
Gather data from 40+ programs
 
 
 
 
Interviews held with 51 programs
Perform statistical analysis to find
correlative results
Report benchmark results to
participating organizations
Public reports on research results
Honourcode, Inc.
Completed
Oct 05
Completed
Completed
Oct 06
Started 3/07
Completed
Sep 09
In process
In process
In process
Honourcode, Inc.
SE-ROI Results:
Demographics
Basic Demographics
Characteristic
Number of organizations
Number of data points
Funding method
Program total cost
Cost compliance
Development schedule
Schedule compliance
Percent of program used in
systems engineering effort, by
cost
Subjective assessment of
systems engineering quality
(1 poor to 10 world class)
Honourcode, Inc.
ValueSE Data Set
SE-ROI Data Set
Unknown
16
44
48
Unknown
39 contracted,
9 amortized
$1.1M - $5.6B
Median $42.5M
$600K - $1.8B
Median $14.4M
(0.8):1 – (3.0):1
Median (1.2):1
(0.6):1 – (10):1
Median (1.0):1
2.8 mo. – 144 mo.
Median 43 mo.
2 mo. – 120 mo.
Median 35 mo.
(0.8):1 – (4.0):1
Median (1.2):1
(0.3):1 – (2.5):1
Median (1.1):1
0.1% - 27%
Median 5.8%
0.1% - 80%
Median 17.4%
Values of 1 to 10
Median 5
Values of 1 to 10
Median 7
Program “Size”
Honourcode, Inc.
Program/Team Parameters
Honourcode, Inc.
Honourcode, Inc.
SE-ROI Results:
Primary Relationships
Schedule vs. SE Effort
Honourcode, Inc.
Cost vs. SE Effort
Honourcode, Inc.
Overall Success vs. SE Effort
Overall Success
5.0
R! = 0.33568
4.0
R! = 0.02671
R! = 0.26077
3.0
2.0
Value SE data
SE-ROI data
All data
Poly.(Value SE data)
Poly.(SE-ROI data)
Poly.(All data)
1.0
0%
5%
10%
15%
20%
25%
SE Effort = SE Quality * SE Cost/Actual Cost
Honourcode, Inc.
30%
35%
Technical Quality vs. SE Effort
“Technical Quality” is
based on compliance with
KPP thresholds and goals
2.0
Technical Quality
2.0 = Met goals
1.5
R! = 0.03276
1.0 = Met thresholds
1.0
0%
5%
10%
15%
20%
25%
30%
0.0 = Failed to meet
0.5
SEROI Data
0.0 = failed to meet
1.0 = met thresholds
2.0 = met goals
Poly.(SEROI Data)
0.0
35%
SE Effort = SE Quality * SE Cost/Actual Cost
Honourcode, Inc.
Honourcode, Inc.
SE-ROI Results:
Eight SE Activities
Breakout by SE Activities
MD
RE
SA
SI
VV
Mission/Purpose Definition
Requirements Engineering
System Architecting
System Integration
Verification & Validation
Honourcode, Inc.
TA Technical Analysis
SM Scope Management
TM Technical Leadership/Management
Breakout by Phase
8*'0(1&'(9"#'!#(-#./'
!"#$"%&'()'*%+,&(,*%+'!#(-#./'0(1&'
*23"%+"+'45#6%-'*.$7'!7.1"'
'"%#$
'"!#$
-1$
/4$
&"%#$
06$
07$
&"!#$
88$
36$
!"%#$
0-$
3-$
!"!#$
()*+,$ -./$ 0//$ 01/$ 21/$
Honourcode, Inc.
.1/$
3//$
4,5$
Breakout by Success
8*'0(1&'(9"#':!((#:'!#(-#./1'
'"%#$
'"%#$
!"#$"%&'()'*%+,&(,*%+'!#(-#./'0(1&'
*23"%+"+'45#6%-'*.$7'!7.1"'
!"#$"%&'()'*%+,&(,*%+'!#(-#./'0(1&''
*23"%+"+'45#6%-'*.$7'!7.1"'
8*'0(1&'(9"#':85$$"11)5;:'!#(-#./1'
'"!#$
&"%#$
&"!#$
!"%#$
!"!#$
'"!#$
-1$
-1
/4$
/4
&"%#$
06$
06
07$
07$
&"!#$
88$
88
36$
36
!"%#$
0-$
0-
3-$
3-
!"!#$
()*+,$ -./$ 0//$ 01/$ 21/$
.1/$
3//$
4,5$
Successful programs
• More front end
()*+,$ -./$ 0//$ 01/$ 21/$
.1/$
3//$
Poor programs
• More back end
Success/Poor defined by median of cost overrun
•  “Success” programs are mostly on cost
•  “Poor” programs are mostly overrun
Honourcode, Inc.
4,5$
Breakout by Success
!"#$%&'(&)*#+,-./0(*,1#
)"%#$
)"!#$
("%#$
("!#$
'"%#$
.344566738$9:;<:=>6$
'"!#$
?;;:$9:;<:=>6$
&"%#$
&"!#$
!"%#$
!"!#$
*+$
,-$
./$
.0$
11$
Successful
• More mission/purpose defn
• More tech leadership/mgmt
• More Systems Engineering
Honourcode, Inc.
2/$
.*$ 2*$
Poor
• More system integration
• More verif & valid
• Less Systems Engineering
Typical Data:
Schedule vs. Tech Lead’ship/Mgmt
Actual/Planned Schedule
3.0
2.6
SEROI Data
2.2
1.8
1.4
1.0
0%
0.6
2%
4%
6%
8%
10%
TM Effort = TM Quality * TM Cost/Actual Cost
Weaker visual correlation observed for:
SI System Integration
Honourcode, Inc.
12%
Strong visual correlation observed for:
ALL other activities
Typical Data:
Cost vs. Verif/Valid
3.0
Actual/Planned Cost
2.6
SE-ROI data
2.2
1.8
1.4
1.0
0%
0.6
2%
4%
6%
8%
10%
12%
14%
VV Effort = VV Quality * VV Cost/Actual Cost
Weaker visual correlation observed for:
MD Mission Definition
SI System Integration
Honourcode, Inc.
Strong visual correlation observed for:
ALL other activities
Typical Data:
Overall Success vs. Reqs Engr
5.0
Overall Success
4.0
3.0
2.0
SE-ROI data
1.0
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
RE Effort = RE Quality * RE Cost/Actual Cost
Weaker visual correlation observed for:
SI System Integration
Honourcode, Inc.
Strong visual correlation observed for:
ALL other activities
Typical Data:
Tech Quality vs. Reqs Engr
2.0
SEROI Data
0.0 = failed to meet
1.0 = met thresholds
Technical Quality
2.0 = met goals
1.5
1.0
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
0.5
0.0
RE Effort = RE Quality * RE Cost/Actual Cost
No significant correlation observed for ANY activities.
Honourcode, Inc.
6.0%
Effect of SE Activities
 
Which activities correlate to better quality?
Activity
Cost
Schedule
Overall
Technical
Missn Defn*
Perhaps
Yes
Yes
No
Reqs Engr
Yes
Yes
Yes
No
Sys Arch
Yes
Yes
Yes
No
Perhaps
Perhaps
Perhaps
No
Tech Anlysis
Yes
Yes
Yes
No
Tech Mgmt
Yes
Yes
Yes
No
Scope Mgmt
Yes
Yes
Yes
No
Ver & Val
Yes
Yes
Yes
No
Sys Integr
*
For most projects, MD was performed in an earlier phase
Honourcode, Inc.
Honourcode, Inc.
SE-ROI Results:
Right-Sizing SE
Adjustment to SE Effort
 
Raw SE percent of program cost
Cost SE
SE% =
Cost PROGRAM
 
“SE Effort” - adjust for quality of SE
SEE = SEQ ∗ SE%
€
“Equivalent
SE Effort” – adjust for 14
 
characterization parameters
  Multiplicative
factors as in COSYSMO
€
 
 
Select weights to optimize correlation
=0 for no effect; >0 to increase; <0 to decrease
⎛ PPj ⎞Weight j
ESEE = SEE ∗ ∏ ⎜
⎟
.5
⎠
j =1...14 ⎝
Honourcode, Inc.
Effect of
Characterization Parameters
R2=12%
3.0
Value SE data
SE-ROI data
Actual/Planned Cost
2.6
All data
Poly.(Value SE data)
Poly.(SE-ROI data)
2.2
Poly.(All data)
1.8
R2=77%
R! = 0.774
R! = 0.350
1.4
1.0
0%
10%
20%
R! = 0.16587
30%
40%
50%
0.6
Equivalent SE Effort as % Program Cost
Honourcode, Inc.
60%
70%
Quantified Parameter Weights
Parameter increase
F1
Small
System Size
F2
Amortized
Less SE
More SE
Large
Contracted
Development Methods
F3
System
Subsystem
Level of Integration
F4
High-level
Detailed
Definition at Start
F5
Development
Production
Easy
Life-Cycle Stage
F6
Controlled
Proof Difficulty
F7
Independent
Difficult
Development Autonomy
!"#
Honourcode, Inc.
!$%&#
$#
$%&#
Subjective Parameter Weights
Parameter increase
F1
Low
Simple
Less SE
More SE
High
Team Understanding
F2
Complex
Program/System Complexity
F3
Few
Many
Weak
Installation Differences
F4
Strong
Team Process Capability
F5
Light tools
Great tools
Need for & Use of SE Tools
F6
Low risk
High risk
Narrow
Technology Risk
F7
Wide
System Applicability
!"#$%
Honourcode, Inc.
"%
"#$%
Right-Sizing SE
 
 
 
 
Find optimum level of ESEE for the success
measure desired
Estimate characterization parameters
Estimate expected project SEQ
Adjust SE level backwards
(will be automated calculations)
  Apply weights to optimum ESEE level to
determine SEE level
  Apply SEQ factor to determine SE% to use
Honourcode, Inc.
Honourcode, Inc.
Summary
Quantified, Proven Results
 
Better programs expend
 
 
 
All SE activities correlate well with
 
 
 
Stakeholder acceptance
Cost/schedule control
No SE activities correlate with
 
 
more SE effort overall
more mission definition, more tech leadership
System technical quality
SE today leads to better programs
– but does not lead to better systems.
Results can be used to right-size SE
Honourcode, Inc.
Honourcode, Inc.
Systems Engineering
Return on Investment
Questions?
Eric Honour
+1 (850) 479-1985
[email protected]