How to Make Quality Decisions Using a Decision Analysis and Resolution (DAR) Process

How to Make Quality Decisions
Using a
Decision Analysis and Resolution
(DAR) Process
Briefing Objectives
▼ Understand how to make quality decisions using a
Decision Analysis and Resolution Process
 Enable attendees to successfully implement the Decision Analysis
and Resolution (DAR) Process within their organizations and/or
on their projects
− Understand why DAR is important
− Become familiar with the DAR process and the tools and
techniques used in implementing DAR
2
Outline
▼ What is DAR?
▼ Why do we need DAR?
▼ Overview of the DAR Process
▼ DAR Process Details
▼ DAR templates, tools, and references
3
What is DAR?
▼ The purpose of Decision Analysis and Resolution is to
analyze possible decisions using a formal evaluation
process that evaluates identified alternatives against
established criteria
 Or more plainly: a process to make key decisions more objectively
and wisely
4
What is DAR? (2)
▼A formal evaluation process:
 Provides documented, measurable, and objective criteria
− Numeric – weighting to reflect importance
[Non-numeric-more subjective high-medium-low]
 Can vary in formality, type of criteria, methods employed
− More risk?  more formal methods with stringent
requirements and needs
(may require separate plans, months of effort,
meetings to develop and approve criteria,
simulations, prototypes, piloting, and extensive
documentation)
5
What is DAR? (3)
 Has a higher probability of selecting a solution that meets
multiple demands
 Involves:
» Establishing the criteria for evaluating alternatives
» Identifying alternative solutions
» Selecting methods for evaluating alternatives
» Evaluating solutions using the criteria and methods
» Selecting solutions based on evaluation criteria
» Documenting results and rationale for decision
6
Why Do We Need a DAR Process?
▼ Making informed, high quality decisions reduces
organizational/project risk and increases probability of success
 Following a structured and disciplined process for making highly complex
and/or high risk decisions leads to quality decisions
 Helps to gain stakeholder buy-in for project decisions
 Encourages more disciplined objective thinking
 Reduces or eliminates subjectivity and bias
7
Dilbert on Decisions
8
Dilbert on Decisions (2)
9
Perspective
Life is the sum of all your choices. -Albert Camus
▼ We make decisions every day; everything we say and do is the
result of a decision, whether we make it consciously or not.
For every choice, big or small, there's no easy formula for making the right
decision.
 Many managers and Project Managers rely on their gut – their experience and
instincts – to make major decisions. Very risky!
 A downside of this approach is that it is difficult to discuss with others how or
why the manager came to the decision they did. In fact, it is the antithesis of
group decision-making. Also, the decision is not often recorded for future use
and re-evaluation so we can ask:
− “Was this a good decision?” (objectives and criteria)
− “How can we make consistently better decisions across the project /
organization?” (lessons learned)
10
Do All Decisions Need to
Follow a Formal Process?
11
When You Use the DAR Process?
▼ Organizations/Projects need to define their own thresholds
or triggers for executing the DAR process
 Generally, DAR is used for decisions that are high risk or have
high impact on the project
 DAR thresholds or triggers are usually documented in the Project
Plan
▼ Example thresholds and/or criteria:





Schedule impact of >10%
Cost impact of >15%
Quality Impact of >15%
Impact to safety or security
Purchases over $10,000
12
What Decisions Do I Apply DAR To?
▼ DAR Process can be used in many types of
Organizational/project decisions:
Technical decisions
Design decisions
Make or Buy decisions
Software tools
Trade studies
Change Control Boards (CCBs)
− Major changes to processes or products
 Risk Management Boards
− Select a risk mitigation approach
 Contractor source selection






13
Example Guidelines For Invoking DAR
These could be formal “trip-wires”:
▼ When decisions are directly related to topics of medium or high risk
▼ When decisions are related to changing work products under CM (key
baselined documents such as requirements baselines, project plans and
schedules, etc.)
▼ When project management or engineering decisions could cause
unacceptable project / schedule delays
▼ When management and engineering decisions affect the ability to achieve
project goals
▼ When management and engineering decisions relate to safety or security
issues
▼ When there are multiple approaches or multiple solutions to architecture or
design issues
▼ On material procurement where 20 percent of the parts constitute 80 percent
of the total material costs
14
DAR Example
▼ Make-Reuse-Buy trade study
 List four alternatives: make, reuse, buy and do nothing
 List alternatives from all potential vendors.
 Evaluation criteria would include items such as cost, schedule,
performance, risk, number of requirements not met, etc.
15
Overview of the DAR Process
1
2
3
Select Decision
Team
Define the Analysis
Objective and
Establish
Evaluation Criteria
Identify Available
Solutions
4
5
6
Select Evaluation
Method(s) That are
Best for the
Problem Domain
Evaluate Available
Solutions Using
Established Criteria
and Methods
Select Solutions
and Prepare
Evaluation
Report
16
1. Select Decision Team
▼ Process Guidance
 When an issue is identified that requires use of the DAR Process
(according to the Project’s defined guidelines), the project
manager selects stakeholders (including a facilitator) to assist in
decision making.
▼ The Organization/Project’s defined guidelines are entry
criteria to this process. They are the triggers to determine
which issues should be subjected to a formal evaluation
process.
▼ DAR can involve many areas of an Organization/Project.
The Decision Team should have the skills necessary to
address the issue at hand.
17
1. Select Decision Team (2)
▼ Process Guidance
 Team must have necessary knowledge and skills for the issue to be decided
 Team will research and evaluate possible alternatives
▼ Project Mananger needs to select knowledgeable team members to
make informed decisions.
 Some decisions are critical to program success, so there needs to be
confidence that the best decision was made by a knowledgeable team
 3-5 team members is ideal
Possible Consequences – if this task is not
done well
▼ There may be lack of confidence that the
best decision was made if the decision team
lacks appropriate skills and viewpoints.
Implementation Examples – how easy it is to
satisfy this task
▼ Select decision team members who
actually do the work involving the issue, and
those who will be affected by the decision.
▼ Also consider selecting a team member
who is less involved, to ensure a balance of
the “outsider” perspective.
18
2. Define the Analysis Objective and
Establish Evaluation Criteria
▼ Process Guidance
 Facilitator leads team to define analysis objective and establish scope of the
decision effort for each task in DAR Process.
 The team documents and maintains the evaluation criteria, which provide the
basis for evaluating alternative solutions, by doing these tasks:
− Analyze and establish scope of the issue
− Define evaluation criteria
− Define weight values to criteria based on relative importance
− Define scoring algorithms to determine percent satisfaction for criteria
− Define overall scoring thresholds of acceptability
− Document rationale for selection/rejection of evaluation criteria
− Identify acceptable timeline for decision completion
▼ In this task, you establish what you are evaluating and what you
need to know.
19
2. Define the Analysis Objective and
Establish Evaluation Criteria (2)
▼ Process Guidance
 There are many ways to determine and document the criteria for an issue,
including a flow chart, decision tree, triangle tool (with weights at each corner),
or spreadsheet matrix form. Tools that may be used for this activity include:
− Decision Analysis and Resolution Record Form or
− Decision Analysis Matrix
▼ Evaluation Criteria are based on needs, requirements, business
objectives, assumptions, etc, depending on analysis objective.


Can include technology limitations, environmental impact, risks, life cycle
costs, schedule, performance, reliability, etc.
2-5 criteria are usually enough
Possible Consequences – if this task is not
done well
▼ Inappropriate criteria can lead to the
wrong decision that doesn’t solve the issue.
▼Critical requirements can be missed.
Implementation Examples – how easy it is to satisfy
this task
▼ Tie evaluation criteria to business and project
needs.
▼ Solicit knowledgeable team members to establish
criteria.
20
3. Identify Available Solutions
▼ Process Guidance

Facilitator leads team to identify available solutions:
− List available solutions and weighting criteria
− Search related technical/scientific domains, internet, and/or historical
data to validate alternatives
− Identify risks for alternatives
▼ There are many ways to come up with alternative solutions to
your issues, for example:






Brainstorming
Literature search
Market research
Competitor analysis
Internet
Historical data
21
3. Identify Available Solutions (2)
▼ Process Guidance
 Document the proposed alternatives and their risks
▼ Documenting the possible alternatives on a form or spreadsheet
ensures that potential solutions are appropriately captured. Then
viable solutions can be selected for evaluation.
 Documenting the reasons for accepting or rejecting alternatives ensures that
anyone in the future making the same decision will know the rationale behind
the entire decision process so that mistakes are not repeated
Possible Consequences – if this task is not
done well
▼ Not documenting the decision process
could lead to missing a viable solution.
▼ Future project managers would not have
the historical data on why certain decisions
were made, so may repeat unacceptable
solutions.
Implementation Examples – how easy it is to
satisfy this task
▼ List the alternative solutions on a form to
better evaluate and accept / reject each
solution as a viable option for evaluation.
▼ Store the data sheets on a shared
repository; report to Organization if
appropriate to share a best practice.
22
4. Select Evaluation Method(s) That are Best
for the Problem Domain
▼ Process Guidance
 Facilitator leads team in determining Evaluation Methods to be
applied in selecting a solution.
− methods for evaluating viable solutions against criteria vary
for technical/nontechnical issues (simulations, statistical
analysis, decision theory)
− level of detail of a method should be commensurate with
cost, schedule, performance, and risk impacts
▼ There’s no one formula for making the right decision
23
4. Select Evaluation Method(s) That are Best
for the Problem Domain (2)
▼ Process Guidance
 Example Evaluation Methods:
− Cost/Benefit Analysis
− Engineering Trade-Off Analysis
− Delphi Method
− Simulation
− Feasibility Study
- Pair-Wise Comparison
- Prototyping
- Testing Data Analysis
- Reference to “Best Practice”
- User Reviews
▼ Cost and criticality of the decision may be considerations when
selecting which method to use
Possible Consequences – if this task is not
done well
▼ Using a method that is not detailed or
rigorous enough for the criticality of the issue
will not exercise the options well enough,
potentially putting the project at risk.
Implementation Examples – how easy it is to
satisfy this task
▼ Select evaluation method appropriate to
the criticality and complexity of the issue to
be solved.
24
5. Evaluate Available Solutions using
Established Criteria and Methods
▼ Process Guidance
 Facilitator leads team to evaluate the selected alternative
solutions (determined in task 3):
− involves analysis, discussion, and review
 Scoring (determined in task 2) is substantiated
 Conclusions are reached using selected methods (determined in
task 4)
▼ Evaluate the viable solutions from the initial list of possible
alternatives.

As a guideline, there should be 2 to 5 alternative solutions
identified for evaluation
25
5. Evaluate Available Solutions using
Established Criteria and Methods (2)
▼ Process Guidance
 Consider new alternative solutions, criteria, or methods if outputs do not
yield desirable results or are inconclusive
− Repeat tasks 2-5 until one or more viable solutions can be selected
▼ Taking the time to re-evaluate solutions as needed can yield
valuable results.
Possible Consequences – if this task is not
done well
▼Selecting too many solutions to consider
could result in some of them not being given
thorough consideration.
▼ Rushing to a decision when time should
be made to re-evaluate alternative solutions
when necessary may result in wasted
time/effort implementing an inappropriate
solution.
Implementation Examples – how easy it is to
satisfy this task
▼Limit solutions to be evaluated to 2-5
viable options.
▼Balance time available to make a decision
with the criticality of the issue to ensure a
credible solution is selected.
26
6. Select Solutions and Prepare Evaluation
Report
▼ Process Guidance
 Selecting solutions involves weighing the results from the
evaluation of alternatives
− Determine composite score for each alternative
− Assemble and document the data for final selection
− Document results and selection rationale
▼ Once you select your solution, decide if it is too risky – do
you need to re-evaluate?
27
6. Select Solutions and Prepare Evaluation
Report (2)
▼ Process Guidance
 Prepare an Evaluation Report documenting decision process
− include any risks associated with selected solution
− include rationale why other alternatives were not selected
 Submit Evaluation Report to Project Manager for final approval and
distribution
▼ Important to record results and lessons learned for future use
and re-evaluation:
 to know if this was a good decision
 to be able to make consistently better decisions across the project /
organization
Possible Consequences – if this task is not
done well
▼ Not adequately documenting the decision
data may not give the project manager the
needed background for approval.
Implementation Examples – how easy it is to
satisfy this task
▼ Document the rationale for selection on
the initial Decision Analysis Form.
28
Process Improvement Habits
Habits to Break
Habits to Make
Making “gut” decisions without a
Maintain objectivity by assembling a team of
Not documenting the decision
Use a decision form listing alternatives, selection
team and without conducting a formal
analysis, resulting in a subjective
decision.
Trying to evaluate too many
alternative solutions.
process and how solutions were
selected.
knowledgeable team members, getting different
points of view and backgrounds to help analyze the
aspects of an issue before a decision is made.
Limit alternatives to 2-5 viable solutions for
evaluate.
criteria, final selection.
Decision Analysis and Resolution Record
Form
 Decision Analysis Matrix
Document rationale and lessons learned so
mistakes won’t be repeated
Maintain the data on a shared repository
29
DAR Tools and Resources
▼ DAR Record Form
▼ DAR Matrix
▼ Pugh Matrix
▼ Decision-Making Tools:
 http://www.mindtools.com
30
DAR Record Form
31
DAR Decision Analysis Matrix
32
Project Selection Template
( example of a Pugh Matrix)
Project #1
Project #2
Project #3
Balanced scorecard impact
Starting and ending points are easy to identify
Quantifiable potential to reduce cost or increase revenue
Sufficient resources are available to be committed to effort
Process will not be impacted by other near term initiatives
Current process is defined
Metrics and data exist
Process occurs frequently
Favorable impact on customer
Management supports the effort
Project is framed as a target or need -- not as a solution
The process is within our span of control
A process owner is defined
Duration of project is < 4 months
Rating for project (Sum of the column)
Prioritize by using an impact rating scale (High = 9, Medium = 3, Low = 1)
33
Example Decision Matrix:
LSS Project Selection Matrix
34
Select Change Solutions
Scoring Guidelines
Factor
Control
Rating
The extent to which the group/individual controls
the problem and can control the solution
The degree to which the solution actually will
Relevance solve the problem
Low
High
1
3
9
1
3
9
3
$$
1
$
The amount of time, money, people, material, etc.
required to implement the solution, AND the
9
Resources extent that the required resources are actually
$$$
available
Payback
The approximate expected payback/results from
solving the problem (cost vs. benefit)
1
3
9
Buy-in
The degree to which the changes involved in
implementing the solution will be acceptable to
people affected and to any more senior level of
management that must approve the solution
1
3
9
35
PICK Chart
Big
Pay-off
Small
Pay-off
Item
Item
Item
Item
Item
Easy
Item
Item
Item
Item
Item
Item
Item
Hard
Item
Implement
Possible
Challenge
Kill
Item
Item
Item
Item
Item
Item
36
Typical Issues With DAR Implementation
▼ It is unclear who is authorized to make what decisions and when
▼ It is unclear what decisions are made objectively
▼ Project plan does not identify when in the lifecycle and/or under what
circumstances DAR is invoked. And, no documented guidelines to determine
which events or items should be subjected to a formal evaluation process.
▼ A formal process description is not used or tailored
▼ Evaluation criteria are not documented prior to analysis and decision point
▼ Rationale is unavailable when needed to understand / re-use earlier decisions
▼ Lessons learned are not collected and analyzed about the decision
▼ Too few choices are considered for major decisions. Missing a more optimal
solution may cost time, money, credibility, and perhaps even the whole project.
37
Summary
▼ DAR is a process to make key decisions more
objectively and wisely
▼ Making informed, high quality decisions reduces
project risk and increases probability of project
success
▼ Not every decision requires the use of the DAR
process
▼ Resources and templates are available to help you to
implement DAR
38
Summary (2)
▼ DAR should be used to make objective decisions on issues which
▼
▼
▼
▼
▼
involve medium to high risks and have the potential to prevent
achievement of project objectives
A formal evaluation process reduces the subjective nature of the
decision
A formal evaluation process has a higher probability of selecting a
solution that meets the multiple demands of the relevant stakeholders
Guidelines must be established for determining which issues should
be subjected to a formal evaluation process
The generation and consideration of multiple alternatives early in a
DAR process increases the likelihood that an acceptable decision will
be made, and that consequences of the decision will be understood
Risks associated with the implementation of the selected solutions
must be assessed as part of the selection process
39
References
▼ Chrissis, Mary Beth; Konrad, Mike; Shrum, Sandy, CMMI for
▼
▼
▼
▼
Development: Guidelines for Process Integration and Product
Improvement, Third Addition; 2011Pearson Education
Mizukami, Diane; Northrop Grumman Corp., “DAR Appraisal is
Coming,…No Trade Studies Anywhere,…Now What?”; 2008
SEPG Conference
Phifer, Bill; EDS; “DAR Basics: Applying Decision Analysis and
Resolution in the Real World”, March 10, 2004
Stamnas, Les; “Decision Analysis and Resolution Process Area
Workshop”, V1.2, Feb 2010
DAR Process Awareness Training, SSC PAC 2011
40