How to Build Process Performance Models

How to Build Process Performance Models
(PPMs) using only five (5) datapoints
AGENDA:

Concept of Process Performance Models (PPMs)

Challenges in PPM Building

Concept of Bayesian Belief Network (BBN)
(including statistical concepts applied such as binomial distribution,
conditional probability and bayesian probability)

PPM Building Process using BBN approach

Selection of Critical Sub-Processes using BBN-PPM
2
Concept of Process Performance Model (PPM)

The CMMI definition of Process Performance Model is “a description of
the relationships among attributes of a process and its work products that
are developed from historical process-performance data and calibrated
using collected process and product measures from the project and that
are used to predict results to be achieved by following a process.”
Historical
Data
3
Challenges in Process Performance Modeling
Challenges:


LIMITED QUANTITATIVE DATA
low variability of available data
4
Concept of Bayesian Belief Network (BBN)

Bayesian Belief Network (BBN) is a probabilistic graphical model that
represents a set of variables and their probabilistic independencies
(e.g. the probabilistic relationships between diseases and symptoms)
Historical
Data
Experts
Belief
5
Concept of Bayesian Belief Network (BBN)

Figure below is an example of a Bayesian Belief Network. It is a graphical
representation of the underlying probabilistic relationships of a complex
system

The calculation and analysis of a BBN took advantage of conditional and
marginal independences among random variables
Burglary
P (b)
Earthquake
Alarm
P (e)
P (a I b,e)
6
Statistical Concepts Applied_Binomial Distribution
Binomial Distribution (Probability of Success):

is the probability of the number of successes in the N outcomes (given N
number of Yes/No experiments)

e.g. Determine the probability of obtaining exactly 3 heads if a fair coin is
flipped 6 times?
Given: r = 3, N = 6 and π = 0.05
Where:
P(r) - is the probability of exactly r successes
r – is the number of successes
N - is the number of events
π - is the probability of success on any one trial.
Assumptions:
- The number of observations n is fixed
- Each observation is independent
- Each observation represents one of two
outcomes ("success" or "failure")
- The probability of "success" p is the same
for each outcome
7
Statistical Concepts Applied_Conditional Probability
Conditional Probability:


is the probability of an event occurring given that another event has already
occurred
e.g. Figure below illustrates the conditional probability of A given B and B
given A
P(A|B) =
A
A,B
P(A,B )
P(B)
;
B
P(B|A) =
P(B,A )
P(A)
8
Statistical Concepts Applied_Bayesian Probability
Bayes’ Probability:



Bayes' Theorem is a simple mathematical formula used for calculating
conditional probabilities
It is one of the different interpretations of the concept of probability and
belongs to the category of evidential probabilities
The Bayesian interpretation of probability can be seen as an extension of
logic that enables reasoning with uncertain statements
P(A|B) =
P(BIA )P(A)
P(B)
where:
P(A) : is the prior probability of A: the probability that A is correct before the data B are seen.
P(B|A) : is the conditional probability of seeing the data B given that the hypothesis A is true. This
conditional probability is called the likelihood.
P(B) : is the marginal probability of B.
P(A|B) : is the posterior probability: the probability that the hypothesis is true, given the data and the
previous state of belief about the hypothesis.
9
High Level Overview on PPM development through
Bayesian Belief Network (BBN)
Identification of Response (Y)
Identification of PPM Scope
Identification of Influencing factors
Data Collection
(Quantitative Data and Qualitative Information)
Probability Calculations
Selection of Critical Sub-process
10
Model Building – Identification of Response

Identification of PPM response (Y)

Typical responses (Ys):
Ticket Resolution Duration
Ticket Resolution Effort
Backlog Processing Efficiency
11
Model Building – Identification of Stages / Nodes


Identification of stages that impacts the response
PPM scope are identified based from projects’ lifecycle
Response
(e.g. Ticket Resolution Duration)
Projects’ Stages / Nodes:
Stage 1
Stage 3
(e.g. Investigation)
(e.g. System Integration)
Stage 2
(e.g. Fixing)
12
Model Building – Identification of Influencing Factors

Identification of factors influencing the response through
brainstorming with projects’ SMEs as well as its measures
Ticket Allocation
Skill of Resources
Testing Process
Ticket Complexity
Review Process
Stage 2
(e.g. Fixing)
13
Model Building – Collecting Historical Data

Collect project’s historical data
Yes – if the factor / node is
applicable to the ticket or
requirement
No – otherwise
14
Model Building – Probability Computations

Compute for the Binomial and Conditional probabilities
Binomial Probabilities (Probability of Success)
- represents the current probability of success of each factor
Influencing Factors
Probability of Success
(Computed)
Code
Probability of Success
(Experts’ Belief)
Code
Ticket
Complexity
Skill of
Resources
Ticket
Allocation
Testing
Process
Review
Process
100%
40%
80%
60%
60%
100%
40%
80%
60%
60%
Conditional Probabilities (Weights)
- represents the impact of each factor to the response (for that specific stage)
Influencing Factors
Conditional Probability
(Computed)
Code
Conditional Probability
(Expert's Belief)
Code
Ticket
Complexity
Skill of
Resources
Ticket
Allocation
Testing
Process
Review
Process
25%
17%
16%
34%
7%
25%
20%
15%
30%
10%
15
Model Building – Probability Computations

Compute for Stage Probabilities based from binomial and conditional
probabilities of each factor
Ticket Allocation
P(F2)
40%
P(F1)
100%
Skill of Resources
P(F3)
80%
Testing Process
Ticket Complexity
P(F4)
60%
Review Process
P(F5)
60%
Stage 2
(e.g. Fixing)
Stage Probability
65%
16
Model Building – Probability Computations

Compute for End to End Probabilities using Conditional Probability
calculations:
Stage 1
Stage 2
Stage 3
Response
(e.g.
Investigation)
(e.g. Fixing)
(e.g. System
Integration)
(e.g. Ticket
Resolution
Duration)
Stage
Probability

77%
Stage
Probability
70%
Stage
Probability
79%
End to End
Probability
75%
The End to End Probability represents the current probability of
meeting the goal, given current project’s performance. It would help
project to know how well they are performing with respect to their
goal.
17
High Maturity – Selecting Critical Sub-processes

Select the priority stage based on the values of stage probabilities
then select the critical sub-process based from binomial and
conditional probability of each factor
Stage 1
Stage 2
Stage 3
Response
(e.g.
Investigation)
(e.g. Fixing)
(e.g. System
Integration)
(e.g. Ticket
Resolution
Duration)
Stage
Probability
77%
Stage
Probability
Influencing Factors
70%
Stage
Probability
79%
End to End
Probability
75%
Ticket
Complexity
Skill of
Resources
Ticket
Allocation
Testing
Process
Review
Process
100%
40%
80%
60%
60%
25%
20%
15%
30%
10%
Code
Probability of Success
Conditional Probability
Code
Code
18
Benefits of using BBN Models
• In summary: why should we use BBNs?
• BBNs on their own enable us to model uncertain events and arguments about
them. The intuitive visual representation can be very useful in clarifying previously
opaque assumptions or reasoning hidden in the head of an expert.
• With BBNs, it is possible to articulate expert beliefs about the dependencies
between different variables and BBNs allow an injection of scientific rigour when
the probability distributions associated with individual nodes are simply 'expert
opinions'.
• BBN will give an idea of what stage and factors are greatly impacting the
response.
• BBN can be used to quantify how much improvement in the selected critical sub-
process is required for the project to meet their goal.
19
References:

PPQA Website
Link: http://ppqc.blogspot.com/2008/08/help-understanding-process-performance.html

Bayesian Network – Psychology Wiki Site
Link: http://psychology.wikia.com/wiki/Bayesian_network

SEI Website
Link: http://www.sei.cmu.edu/library/assets/practitioner-viewattach.pdf

HyperStat Online Website
Link: http://davidmlane.com/hyperstat/A2301.html
20
Contact Information

Jeanette O. Ruiz
Accenture Philippines
Email Address: [email protected]

Amir Khan
KPMG Consultant
Email Address: [email protected]
21