Automated Software Processes Performance Analysis and

Transcription

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO
Automated Software Processes Performance
Analysis and Improvement Recommendation
César Barbosa Duarte
Master in Informatics and Computing Engineering
Supervisor: João Carlos Pascoal de Faria (PhD)
17 th February, 2012
Automated Software Processes Performance Analysis and
Improvement Recommendation
Master in Informatics and Computing Engineering
Approved in oral examination by the committee::
Chair: Raul Vidal (PhD)
External Examiner: João Miguel Fernandes (PhD)
Supervisor: João Pascoal Faria (PhD)
____________________________________________________
17 th February, 2012
Abstract
Nowadays there is a huge need for quality in software as our lives may depend on it. The
Personal Software Process (PSP) and the Team Software Process (TSP) are process
improvement frameworks tailored to produce virtually defect free software and deliver it on
time. Although these are great frameworks they need tool support to work efficiently and
without proper tool support the amount of data produced by even relatively small projects is
overwhelming. It is a fact that there are today a great number of tools to provide this support to
PSP and TSP, some of them are semi-automated and require little effort by the developer. But
they still lack automation in a very important part of the process, the data analysis. This is where
the improvement actions are concluded from and it requires that the person doing this has a
great deal of experience in PSP and TSP to reach to this improvement actions and conclusions
just by looking at graphs and sheets (if using semi-automated data analysis tools). The result of
this dissertation is a working prototype that does just that, automates the data analysis part and
provides improvement actions even to the most inexperienced PSP user. Through an extensive
research on several performance indicators, a performance model was made based on the Time
Estimation Error performance indicator (PI). It is represented in a tree-like diagram and it shows
the relationships (what affects what and how) between the PIs. By using this model it should be
possible to determine the source of the problem and suggest improvement actions based on that.
The model was applied in a tool created for this purpose called PSP PAIR. It uses as a basis a
database with data from PSP projects and following the performance model, after calculating
the value of each indicator, it shows a set of suggested improvement actions based on that. The
PSP PAIR could provide a really good support to engineers using the PSP in their projects and
help them speed up the analysis of all the data they collect with automation.
i
Resumo
Hoje em dia existe uma grande necessidade de qualidade no software pois as nossas vidas
podem depender disso. O Personal Software Process (PSP) e o Team Software Process (TSP)
são ferramentas de melhoria de processos feitas para produzir software virtualmente livre de
defeitos e entregá-lo dentro do tempo estipulado. Embora sejam ambas excelentes
metodologias, necessitam de apoio de ferramentas automatizadas para trabalhar de forma
eficiente. Pois sem o apoio de uma ferramenta adequada, até para projetos de pequena
dimensão, a quantidade de dados produzidos por eles é esmagadora. É um fato que há hoje um
grande número de ferramentas que fornece algum apoio ao PSP e ao TSP, sendo que algumas
delas são semi-automatizadas e exigem pouco esforço por parte do utilizador. Mas todas elas
têm um defeito, falta-lhes automatizar uma parte muito importante do processo, a análise dos
dados. É nesta fase onde são feitas as conclusões que levam posteriormente a propostas de ações
de melhoria. Isto requer que a pessoa que o está a fazer tenha uma grande experiência em PSP e
TSP para chegar a ações de melhoria e conclusões apenas a partir de gráficos e tabelas de dados
(isto se estiver a usar uma ferramenta que automatiza parte da análise de dados, ou seja, agrega
os dados em gráficos e tabelas). O resultado desta dissertação é um protótipo de uma ferramenta
que faz exatamente isto, automatiza a parte de análise de dados e sugere ações de melhoria que
até mesmo o utilizador mais inexperiente de PSP deverá perceber. Depois de uma extensa
pesquisa sobre diversos indicadores de desempenho, foi elaborado um modelo de performance
com base no indicador de desempenho Time Estimation Error (erro na estimação do tempo).
Este modelo é representado num diagrama em árvore onde é possível ver os relacionamentos
entre os indicadores de desempenho (o que é que afeta o quê e como). Ao utilizar este modelo
deve ser possível determinar a origem do problema e sugerir ações de melhoria com base nisso.
O modelo foi aplicado numa ferramenta criada para o efeito chamada PSP PAIR. Esta
ferramenta usa como base uma base de dados com os dados de projetos que usaram o PSP e
depois de calcular o valor de cada indicador, mostra um conjunto de ações de melhoria
sugeridas com base no modelo de desempenho elaborado. O PSP PAIR poderá proporcionar um
apoio interessante para engenheiros que estão a usar o PSP nos seus projetos e ajudá-los a
acelerar a análise de todos os dados que foram recolhidos com automação.
ii
Acknowledgments
I would like to start by expressing my gratitude towards my supervisor, Professor Doutor
João Pascoal de Faria, for always being available to share his expertise and extensive
knowledge with me and to guide me along this fantastic and life-changing journey that was to
do this dissertation.
A special thanks to Professor Doutor António Manuel Andrade, for bearing with me as I
was working on this dissertation and finishing my studies, and for giving me the time and
guidance I needed.
I must also acknowledge Professora Doutora Ana Paiva, for introducing me to the
Software Engineering community at FEUP, that eventually led me to this dissertation.
Last but not least, I would like to thank my family for all the support, values and ethical
principles they provided me and instill me with through my entire life and in particular, to my
partner and best friend, Sónia Dutra, for making me a better person with all her love, care and
patience, and without whom I would probably not manage to accomplish everything I did.
To those who I, unjustly, forgot to mention, my sincere gratitude.
iii
Contents
1 Introduction.............................................................................................................................
1.1 Context and Motivation......................................................................................................
1.2 Problem Description...........................................................................................................
1.3 Goals and Expected Results...............................................................................................
1.4 Methodology......................................................................................................................
1.5 Structure.............................................................................................................................
1
1
2
3
3
4
2 Related Work and Concepts................................................................................................... 5
2.1 Personal Software Process.................................................................................................. 5
2.1.1 Process Structure........................................................................................................ 6
2.1.2 Metrics........................................................................................................................ 8
2.1.3 Performance improvement results in the PSP training............................................. 10
2.1.4 Tools......................................................................................................................... 11
2.1.5 PSP Student Workbook............................................................................................ 12
2.1.6 Performance Analysis Report................................................................................... 14
2.2 Team Software Process.................................................................................................... 20
2.2.1 Process Structure...................................................................................................... 21
2.2.2 Tools......................................................................................................................... 21
2.3 Fully automated tool for collection, analysis, visualization, interpretation of data.........23
2.3.1 Architecture.............................................................................................................. 24
2.3.2 Sensors and Metrics................................................................................................. 24
2.4 Statistical Process Control................................................................................................ 27
2.5 Conclusions...................................................................................................................... 28
3 Data Analysis and Improvement Actions............................................................................
3.1 Solution............................................................................................................................
3.1.1 Time Estimation Error..............................................................................................
3.1.2 Size Estimation Error...............................................................................................
3.1.3 Productivity Estimation Error..................................................................................
3.1.4 Productivity Stability................................................................................................
3.1.5 Process Stability.......................................................................................................
3.1.6 Defect Density..........................................................................................................
3.1.7 Process Yield............................................................................................................
3.1.8 Review Rate.............................................................................................................
iv
29
30
30
32
36
37
38
38
40
41
3.1.9 Review to Development Ratio.................................................................................
3.2 Performance Model..........................................................................................................
3.3 Recommended Improvement Actions..............................................................................
3.4 Conclusions......................................................................................................................
42
43
44
47
4 Implementation and Evaluation...........................................................................................
4.1 Architecture of the solution..............................................................................................
4.2 PSP PAIR.........................................................................................................................
4.3 Evaluation.........................................................................................................................
48
48
51
56
5 Conclusions and Future Work............................................................................................. 63
5.1 Goal Achievement and Conclusions................................................................................ 63
5.2 Future Work..................................................................................................................... 64
References................................................................................................................................. 66
v
List of Figures
Figure 1.1: Software Process Dashboard Data Analysis [Initiative].............................................. 2
Figure 2.1: PSP Process Evolution [Humphrey, 2005].................................................................. 6
Figure 2.2: PSP Process Flow [Humphrey, 2002b] ....................................................................... 7
Figure 2.3: PSP Process Elements [Humphrey, 2002b] ................................................................ 8
Figure 2.4: Improvements in PSP Training Data [Hayes and Over, 1997].................................. 10
Figure 2.5: PSP Student Workbook – Main menu and Total Defect Density graph.................... 13
Figure 2.6: Time Estimation Charts............................................................................................. 17
Figure 2.7: Time Estimation Table............................................................................................... 18
Figure 2.8: Time Estimation - %Time per Phase and Hours/KLOC per phase........................... 19
Figure 2.9: Process Improvement Models [Humphrey, 2002a] .................................................. 20
Figure 2.10: TSP and CMM defects/KLOC [Davis and Mullaney, 2003].................................. 20
Figure 2.11: TSP Elements [Humphrey, 2002a] ......................................................................... 21
Figure 2.12: Hackystat Browser interface [Johnson, 2008]......................................................... 23
Figure 2.13: Hackystat Architecture [Lofi, 2005]........................................................................ 24
Figure 2.14: Examples of SPC Tools (Adapted from [Statit Software, 2007])............................ 28
Figure 3.1: Time Estimation Error recommended Controls Limits graph...................................31
Figure 3.2: The Probe Estimating Method [Humphrey, 2005]...................................................33
Figure 3.3: Size Estimation Error recommended Controls Limits graph..................................... 34
Figure 3.4: Part Estimation Error recommended Controls Limits graph.....................................35
Figure 3.5: Missing Parts recommended Controls Limits graph.................................................. 35
Figure 3.6: Expurious Parts recommended Controls Limits graph.............................................. 36
Figure 3.7: Productivity Estimation Error recommended Controls Limits graph........................37
Figure 3.8: Productivity Stability recommended Controls Limits graph.....................................38
Figure 3.9: DDUT recommended Controls Limits graph............................................................ 39
Figure 3.10: TotalDD recommended Controls Limits graph....................................................... 40
Figure 3.11: Process Yield recommended Controls Limits graph...............................................41
Figure 3.12: Review Rate recommended Controls Limits graph................................................. 42
Figure 3.13: Review to Development Ratio recommended Controls Limits graph.....................43
Figure 3.14: Performance Model Tree......................................................................................... 44
Figure 4.1: Architecture - Indicators............................................................................................ 49
Figure 4.2: Architecture – Control Limits.................................................................................... 50
Figure 4.3: Architecture – Performance Model............................................................................ 51
Figure 4.4: PSP PAIR Home........................................................................................................ 52
vi
Figure 4.5: PSP PAIR Results Report.......................................................................................... 53
Figure 4.6: PSP PAIR Project Details.......................................................................................... 54
Figure 4.7: PSP PAIR XML Performance Model and Control Limits......................................... 55
vii
List of Tables
Table 2.1: PSP Measures Definitions [Hayes and Over, 1997].....................................................9
Table 2.2: PSP Tool Support........................................................................................................ 11
Table 2.3: TSP Tool Support (adapted from [Fahmi and Ho-Jin, 2008])....................................22
Table 2.4: Summary of sensor data types [Johnson, 2001].......................................................... 25
Table 2.5: Hackystat Sensors and Metrics (adapted from [Lofi, 2005])...................................... 26
Table 3.1: TEE calculations......................................................................................................... 30
Table 3.2: Recommended improvement actions.......................................................................... 45
Table 4.1: Comparative Analysis between PAR and PSP PAIR.................................................. 57
viii
Abbreviations
CAR
CCC
CMMI
CMU
COQ
DDUT
DLD
EP
IDE
KPI
LOC
MP
PartEE
PEE
PI
PIP
PPS
PROBE
PSP
SEE
SEI
SOAP
SPC
TEE
TSP
XML
Causal Analysis and Resolution
Change and Configuration Control
Capability Maturity Model Integration
Carnegie Mellon University
Cost of Quality
Defect Density in Unit Test
Detailed-Level Design
Expurious Parts
Interactive Development Environment
Key Performance Indicator
Lines of Code
Missing Parts
Part Estimation Error
Productivity Estimation Error (not to be confused with Part Estimation Error)
Performance Indicator
Process Improvement Proposal
Project Plan Summary
Proxy-based Estimating
Personal Software Process
Size Estimation Error
Software Engineering Institute
Simple Object Access Protocol
Statistical Process Control
Time Estimation Error
Team Software Process
Extensible Markup Language
ix
1 Introduction
This introductory chapter presents the context and motivation of this dissertation and
describes the problem addressed, the envisioned solution, the methodology and the expected
results of the work.
1.1 Context and Motivation
Nowadays software plays a huge role in our everyday life and it's very important, if not
essential, that software quality is very high to ensure that there are no failures that may result in
economic or human losses. The need to ensure the quality of software products in a costeffective way drives companies and organizations to seek to improve their software
development process, as it is becoming more and more accepted in industrial production in
general and in the software industry in particular that the quality of the process directly affects
the quality of the product [Kenett, et al., 1999] .
“There is a very strong link between software quality and the processes used to
develop it.” [Kenett, et al., 1999]
Software process improvement and measurement go hand in hand: measures are the only
way to prove improvements in a process.
“Measurements are key. If you cannot measure it, you cannot control it. If you
cannot control it, you cannot manage it. If you cannot manage it, you cannot
improve it.” [Harrington, 1991]
Measurement is defined as “the evaluation or estimation of degree, extent, dimension, or
capacity in relation to certain standards (i.e., units of measurement)” [Sciences, 2008].
1
Introduction
High maturity software development processes that make use of metrics and quantitative
methods, such as the Personal Software Process (PSP) 1 and Team Software Process (TSP)2 of
the SEI / CMU3, generate a significant amount of data that can be subjected to automated
analysis, to identify performance problems, determine their causes, and recommend
improvement actions.
1.2 Problem Description
The amount of data generated by using high maturity software development processes, like
PSP and TSP, can be overwhelming even to the more experienced [Burton and Humphrey,
2006]. Automation is vital for data collection and analysis. There are a lot of tools that automate
a big part of the metrics collection, but most of these tools transform the data into charts and
reports as shown in Figure 1.1, and this is not really automating the analysis of the data.
Figure 1.1: Software Process Dashboard Data Analysis [Initiative]
All this data requires a great amount of time to be analyzed thoroughly and, even though
there are some tools to help and guide the programmer in analyzing the data, this analysis is
mostly manual. The great amount of data to collect and analyze is also a problem, as Watts S.
Humphrey says in [Humphrey, 2002a] :
1
2
3
Personal Software Process and PSP are registered service marks of Carnegie Mellon University
Team Software Process and TSP are registered service marks of Carnegie Mellon University
www.sei.cmu.edu
2
Introduction
“Without proper tool support, the volume of data produced by even relatively small
projects is almost unmanageable.”
Furthermore, despite of PSP devise some guidelines, expert knowledge is needed to do a
proper analysis of the data and obtain improvement actions.
1.3 Goals and Expected Results
The main goal of this dissertation work is to develop a tool to fully automate the analysis
of performance data and recommend improvement actions in the context of software
development. In order to complete this goal a Performance Model must be developed.
This work is limited to the PSP methodology but providing for extensibility to TSP and
other methodologies.
The performance model is going to:
•
identify the performance indicators (PI) and their derivation formulas from base
measures;
•
define the PI control limits;
•
define cause-effect relationships among performance indicators;
•
indicate improvement actions for each root cause.
It is expected that, given this performance model, the working prototype of the tool can
receive as input the data from the use of PSP and interpret it using the performance model
developed. The output should be the improvement actions resulting from the following
procedure:
1. Compute the key performance indicators (KPI).
2. Identify performance problems.
3. Identify root causes.
4. Identify potential improvement actions.
5. Evaluate cost-benefit of each action.
6. Select and recommend actions.
1.4 Methodology
A methodology is required to achieve the aforementioned goals. Therefore the following
objectives and phases were established:
•
Elaboration of the State of The Art Report;
3
Introduction
◦
Investigation of cause-effect relationship between these metrics;
•
◦
The result of this investigation should be the aforementioned performance
model
Creation of a prototype of the tool;
•
◦
An assessment of existing technologies is required
Testing and evaluation of the prototype;
•
◦
•
Thorough a rigorous study and extensive investigation all the subjects related
to this dissertation, including the PSP and the TSP.
Repository maintained by SEI/CMU of PSP course data will be used for
evaluation purposes
Writing of the final report and elaboration of a scientific paper.
Some of these objectives may overlap in time because of the nature of this dissertation.
The testing and evaluation phase will overlap with the writing of the necessary documentation.
1.5 Structure
In addiction to the introduction that provides the context of the problem and explains how
it will be solved, this document contains four other chapters. In chapter 2 it is presented the
related work and concepts where the bibliographic review of the areas of investigation and key
concepts is done. All scientific information collected and regarded as relevant is addressed in
this chapter. In chapter 3 of this report a detailed description of the problem addressed in
dissertation and the proposed solution is presented. The details of implementation of the tool
that is the result of the research done in this dissertation are presented in chapter 4. Finally, the
conclusions and future work are presented in the last chapter (5) where the main ideas to retain
from this document are summarized and the goal achievement is presented.
4
2 Related Work and Concepts
This chapter describes the state of the art. Related work is also presented in order to show
what exists in the same area and what were the problems encountered.
In the first section the PSP is explained in a summarized way, its structure, metrics and
tools are addressed. The Performance Analysis Report is also addressed as it plays and
important role in this dissertation.
The focus of the second section is the TSP, its main ideas and goals are presented, as well
as its structure and tools.
In the third section, the Hackystat system is presented. Its architecture, sensors and metrics
are explained in a somewhat detailed way.
In the last section of this chapter, the Process Control Methods and Tools are presented as
they are of the most importance for process control and improvement.
2.1 Personal Software Process
The Personal Software Process (PSP) is a process framework whose purpose is to guide
developers in defining their own processes, planning and tracking their own work and managing
the quality of the products they produce [Humphrey, 2005]. The recommended approach for
learning the PSP is by doing a sequence of small projects guided by a PSP instructor, using
increasingly refined and more complete processes (see Figure 2.1).
5
Related Work and Concepts
Figure 2.1: PSP Process Evolution [Humphrey, 2005]
2.1.1 Process Structure
The PSP process flow is show in Figure 2.2. The first step of the process is planning,
having as input the requirements. There is a script to guide the work and a plan summary to
record the data in this first step. The other steps also have scripts to guide the work and logs to
record the data. In the postmortem phase (PM), the data form the logs is summarized, the
program size is measured and all of this is then recorded into the plan summary form. The
output of this is the finished product and the completed plan summary form. It should be noted
that (contrarily to the more complex TSP) the PSP is designed for guiding the development of
small programs (or components of large programs) by individual engineers, reason why it does
not include requirements gathering nor high-level (architectural) design.
6
Figure 2.2: PSP Process Flow [Humphrey, 2002b]
As illustrated in Figure 2.1 the PSP has six process versions that are labeled from PSP0 to
PSP2.1. They all have a similar collection of scripts, forms, logs and standards. The scripts
instruct the programmer, step by step, through each part of the process. To record data there are
templates (for the logs and forms) and the standards that help guide the work. The elements
described above are shown in Figure 2.3.
7
Figure 2.3: PSP Process Elements [Humphrey, 2002b]
2.1.2 Metrics
There are four base measures that are collected in the PSP to support project management,
quality management and process improvement:
•
Time:
◦
•
Size:
◦
•
product size (LOC);
Quality:
◦
•
time spent in each process phase (i.e., effort);
defects found on the product;
Date:
◦
date of project or task completion.
Every level of PSP offers new measures to aid developers in managing and improving their
performance, and each one of these measures descend from the three basic PSP measures
8
mentioned above. In Table 2.1 are presented and defined some of this derived measures
available in the PSP.
Table 2.1: PSP Measures Definitions [Hayes and Over, 1997]
Measure
Planned Time in Phase
Actual Time in Phase
Total Time
Time in Phase to Date
Total Time to Date
Time in Phase to Date%
Compile Time
Test Time
Defect
Defect type
Fix Time
LOC (Size)
LOC Type
LOC/Hour (Productivity)
Estimating Accuracy
Test Defects/KLOC
Compile Defects/KLOC
Total Defects/KLOC
Yield
Appraisal Time
Failure Time
Appraisal Cost of Quality (COQ)
Failure COQ
Total COQ
COQ Appraisal/Failure Ratio (A/FR)
Review Rate
Definition
The estimated time to be spent in a phase for a project
The sum of time spent (delta times in time log) for a phase of a
project
The sum of planned or actual time for all phases of a project
The sum of Actual Time in Phase for all completed projects
The sum of Time in Phase to Date for all phases of all projects
100 * Time in Phase To Date for a phase divided by Total Time
in Phase To Date
The time from the start of the first compile until the first clean
compile
The time from the start of the initial test until test completion
Any element of a program design or implementation that must
be changed to correct the program
Each defect is classified according to a defect type standard. It
includes 10 defect types in a simple classification scheme
designed to support defect analysis.
The time to find and fix a defect
A logical line of code as defined in the engineers counting and
coding standard
There are 7 LOC types, Base, Deleted, Modified, Added,
Reused, Added & Modified, Total LOC and Total New Reused
Total added and modified LOC developed divided by the total
development hours
The degree to which the estimate matches the result. Calculated
for time and size
%Error = 100*(Actual-Estimate)/Estimate
The defects removed in the test phase per added and modified
KLOC. 1000*(Defects removed in Test)/Actual Added and
Modified LOC
The defects removed in compile per added and modified KLOC.
1000*(Defects removed in Compile)/Actual Added and
Modified LOC
The total defects removed per added and modified KLOC.
1000*(Total Defects removed)/Actual Added and Modified
LOC
The percent of defects injected before the first compile that are
removed before the first compile 100*(defects found before the
first compile)/(defects injected before the first compile)
Time spent in design and code reviews
Time spent in compile and test
100*(design review time + code review time)/(total development
time)
100*(compile time + test time)/(total development time)
Total Cost of Quality = Appraisal COQ + Failure COQ
A/FR = Appraisal Time/Failure Time
Lines of code reviewed per hour
60 * Added and Modified LOC/review minutes
9
2.1.3 Performance improvement results in the PSP training
The results shown in Figure 2.4 show the effect of PSP on Productivity, Predictability and
Quality. This data is from a study of PSP training conducted by the SEI to a total of 298
engineers/students from 23 classes.
The results show that the PSP helps developers establish a mature and disciplined software
engineering practice that:
•
improves cost and schedule predictability (determined mainly by effort);
◦
•
reduces time to deliver the product;
◦
•
Effort Estimation Error graphs
When the PSP is applied in developing components of a larger system, large
productivity gains will arise in the integration and system testing because of
the much higher quality of those components.
produces high-quality, reliable software.
◦
As the training evolves the number of defects per KLOC decreases as
illustrated in the Defects graph
Figure 2.4: Improvements in PSP Training Data [Hayes and Over, 1997]
10
2.1.4 Tools
One of the PSP main strengths is the use of historical data to help developers to do a better
job, but, in order to do this, data has to be collected and analyzed. To many people this can be a
very boring and tedious task if done manually which can lead to frustration. The beneficial
effects of this data collection may not be immediate and obvious. So all this leads ultimately to
errors in the data gathering [Disney and Johnson, 1998].
A lot of tools have been developed over the years to tackle this problem of manual
gathering and provide a semi or fully automated data collection. A comparison between PSP
Studio [Team, 1997], Process Dashboard [Initiative], Hackystat [Laboratory, 2010], PSPA
[Sison, 2005], Jasmine [Shin, et al., 2007] and PSP.NET [Nasir and Yusof, 2005] is shown in
Table 2.2.
Table 2.2: PSP Tool Support
PSP
Process
Studio Dashboard
Hackystat
PSPA
Jasmine PSP.NET
Environment Windows Windows,Lin Eclipse, Visual Open Source IDE Eclipse,
Only
Scripting
Data
Collection
Interface
Planning
Wizard
ux,
Mac,etc
Java
Manual Manual and
and Auto
Auto
Simple
GUI
and easy
to learn
PSP
Project Plan
Template Template
LOC Counter Manual
Auto(but
offline)
No
Defect Sharing No
User Available Developer Developer
Only
Only
No
No
User Privacy
Report
Other
Features
Studio, JBuilder,
etc
Java
Auto
Java,C
Auto
GUI
GUI & Pop-up
windows
Project Plan
Template
Schedule Planning
Template
Auto(sensorbased)
No
Developer Only
No
Graphical PPS,Graph Summaries and
analysis
and Chart
Graph
report
(Estimation
and defect)
Supports
Data collected by
PSP0 to 3
separate software
tools
11
Windows
Microsoft and UNIX
Office,
Platform
JBuilder
Java,XML
PHP
Auto
Manual and
Auto
GUI
GUI & Popup windows
Project
Plan
Template
Auto(sensor- Auto(senso
based)
r-based)
Yes(Auto)
No
Developer and Developer
Project Manager
Only
Yes(Login and
No
Privacy Task)
PPS, Graph Test Report
(Defect
summary
classification and
Ranking)
Consolidation of
individual
schedules/defects
into the team ones
Schedule
Planning
Template
Auto
Yes(Auto)
Developer
Only
Yes
(Login)
Graph, and
Report
summary
-
2.1.5 PSP Student Workbook
This is the tool used by SEI in their PSP course. It helps and guides students to gather and
analyze all the data they need. It is a pretty good support for the learning of this methodology as
it has a lot of support materials (course materials, scripts, forms, templates, etc). It has got a lot
of so-called analysis tools to help the student in the analysis of the data, it basically transforms
the data into graphs of known performance indicators (that require some calculations), as Time
Estimation Error, Size Estimation Error and Productivity. In Figure 2.5 the main menu and an
example of a graph of produced by this tool are shown.
12
Figure 2.5: PSP Student Workbook – Main menu and Total Defect Density graph
This is going to be the basis for the tool built as a result of this dissertation. All the data
gathered using this tool is stored in a database that is going to be used in this tool.
13
2.1.6 Performance Analysis Report
The Performance Analysis Report is the final exercise of the PSP Advanced Course. This
advanced course covers advanced topics of PSP and expands on TSP concepts. As stated in
[SEI-Training, 2011], it essentially teaches software engineers to optimize their process
performance by:
•
applying sound design engineering and verification techniques
•
using a defined estimating process (PROBE) to estimate size and effort
•
tracking and prediction using earned value
•
demonstrating various quality techniques for improving the software process,
product and programmer productivity
•
recognizing quality problems and knowing how to fix them
•
use of statistical methods to understand and improve planning and quality
performance
•
showing how measures can be combined to provide useful data for future project
plans, provide information on process effectiveness, and process improvements
The Performance Analysis Report has a couple of prerequisites: to complete all PSP
programs and to read the chapters 13 and 14 of the book [Humphrey, 2005]. Provided this
prerequisites are completed, the objectives of this assignment are:
•
Define, instrument, and follow a process for personal use
•
Demonstrate performance analysis skills
•
Anticipate the challenges faced in using these processes and methods on the job
and determine how to address them
•
Learn to update a personal process based on personal data
To complete this assignment there are a few requirements that must be followed, but at a
minimum the report must address three fundamental topics, Planning Performance, Quality
Performance and Process Performance. Furthermore, the report must:
•
Characterize current performance
•
Contain the PIPs1 for improvement based on analysis of current performance
•
Set performance goals
•
Identify PIPs to implementation in order to achieve performance goals
The analysis done in this report is based on the three fundamental topics earlier mentioned
(Planning Performance, Quality Performance and Process Performance), and it is supported
mostly on charts and tables from which conclusions can be derived (what went wrong, what
1
The Process improvement Proposal (PIP) is a process defect-reporting procedure. It is a part of the process
improvement process. [Humphrey, 2005]
14
went right, how to avoid doing the mistakes again and how to keep on doing the right things).
These conclusions generally result in PIPs. Lastly a Process Performance Analysis is done. This
includes and gathers all the conclusions from the other topics and PIPs, resulting on detailed
improvement actions.
The automation of this last point of the Performance Analysis Report, the Process
Performance Analysis, is one of the specific objectives of this dissertation.
Using the example [Faria, 2008] of time estimation error illustrated in the charts of Figure
2.6, in the table of Figure 2.7 and in Figure 2.8, one should analyze everything and conclude
that:
•
Time estimation accuracy is poor.
•
Steady improvement of empirical time estimation from Program1 to Program4,
considering that time underestimation in Program3 is caused by size
underestimation (see Hours/KLOC chart in see Figure 2.8). In program 4, time
estimation was based on historical productivity (3rd option in method C).
•
Inferior time estimation in program 5 is justified by process change and more time
spent in DLD that was not gained in subsequent phases.
•
Inferior time estimation in program 6 can be justified by a much smaller program
size (about one third compared to previous programs)
•
Strong time underestimation in program 7 is due mainly to the time spent in DLD,
high for the program size at hand (see chart with hours per KLOC in Figure 2.8).
Still, CODE and CodeReview time correlate well with program size. Looking in
more detail at project artifacts and project data (PIPs), two reasons exist:
◦
Problematic reuse of a large code base with inferior quality standards (lack
of design documentation for update degrades DLD, robustness problems
degrade UT);
◦
Inefficient and unstable DLD process (equations edited in Word, too big
compared to Added&Modified program size) degrade DLD time.
Regarding Productivity stability:
•
•
Overall bad correlation between time and size due to (see explanations above):
◦
process changes;
◦
impact of reuse;
◦
impact of other factors for very small sized programs;
◦
small size range.
Chart of Hours/KLOC in Figure 2.8 shows stability of productivity, except for
DLD.
15
Regarding the Time distribution per phases in Figure 2.8:
•
Growing % of time in DLD, as expected, although too much.
•
Small decrease in % of time spent in UT and CODE, as expected. Doesn't
currently compensate increased time in DLD.
•
% UT time has space for further reduction, by improving the effectiveness of
DLD and DR, because most defects are injected in DLD (this is shown in the
document cited in Quality Performance Analysis).
These are the conclusions for the Planning Performance Analysis - Time Estimation. One
has to do a similar analysis to Planning Performance Analysis - Size Estimation. The Quality
Performance Analysis also has to be made to analyze the Quality Profile, the defect injection
and the defect removal before testing (Yield) and come to a set of conclusions that will help
improving the process.
Lastly, the Process Performance Analysis is made and it should be based on the Planning
and the Quality Performance analysis. It consists on a set of improvement actions derived from
the conclusions of the aforementioned performance analysis. For example, it should state where
there is a need of improvement, why there's this need, and how should this improvement be
implemented.
For example, from the conclusions shown above for Planning Performance Analysis –
Time Estimation a programmer should derive that he must improve time estimation accuracy.
The developer can do this by reducing the time estimation error. Currently the estimation error
is between 25% and 75%; the goal can be reducing it to a number between 0% and 25%. He can
reach this goal by, for example, implementing these couple of measures (improvement actions):
•
Stabilize and make more efficient the Detailed-Level Design (DLD) process to
improve and stabilize productivity: avoid expensive DLD representations (e.g.,
equations in Word), avoid long DLD documents (compared to code size); do
Logic Specifications that can be used as code comments and can be easily written.
•
Widen the size range to have more basis for estimations.
16
Figure 2.6: Time Estimation Charts
17
Figure 2.7: Time Estimation Table
18
Figure 2.8: Time Estimation - %Time per Phase and Hours/KLOC per phase
19
2.2 Team Software Process
Like the SEI Capability Maturity Model, the Team Software Process (TSP) is based on
process improvement principles. As shown in Figure 2.9, the focus of the TSP is the team while
the focus of the CMM is improving the organization's capability .
Figure 2.9: Process Improvement Models [Humphrey, 2002a]
The TSP adds a project management and team management layer to the PSP and guides
teams in improving their performance and producing high quality products on budget and on
time. It is designed for use of teams from 2 to 20 members. TSP is capable of producing
virtually defect free (0.06 defects/KLOC) products as shown in Figure 2.10 [Davis and
Mullaney, 2003].
Figure 2.10: TSP and CMM defects/KLOC [Davis and Mullaney, 2003]
20
2.2.1 Process Structure
The principal elements of the TSP structure are show in Figure 2.11. It is required that all
members of a TSP team are fully trained in PSP to provide them with the knowledge and skills
essential to use the TSP. These elements are the building blocks of TSP. The PSP provides
high-maturity practices for individuals; it provides process discipline, performance measures,
estimating and planning skills and quality management skills. The TSP provides high-maturity
practices for teams, through team building (goal setting, role assignment, tailored team process
and detailed balanced plans) and team management (team communication and coordination,
project tracking, risk analysis).
Figure 2.11: TSP Elements [Humphrey, 2002a]
2.2.2 Tools
The TSP addresses critical business needs like better cost and schedule management and
effective quality management, but in order to properly fulfill these business needs there is one
thing that is essential, a proper support tool. In a study conducted in 2003 with data from 20
TSP projects in 13 organizations [Davis and Mullaney, 2003] one of the most common
comments about the shortfalls of the TSP were tool-related. Here are a couple of examples:
21
“Currently, the most common issue is the TSP spreadsheet. The developers are
finding that small mistakes in data entry can cause a lot of grief (shared by
coaches). Getting a more robust tool would definitely help.”
“Tool issues are preventing us from properly consolidating individual, team, and
organization data. Multiple data entry is frustrating to practitioners.”
So to tackle this major shortfall of the TSP several tools were developed over the years. In
Table 2.3 a comparative study of some of these tools is shown.
Table 2.3: TSP Tool Support (adapted from [Fahmi and Ho-Jin, 2008])
Team Dashboard
Partially Automatic
Data Collection
No
Data Validation
Some
Statistical Analysis Tool
No
Communication/Discussio
n among Team Members
No
Access to Potential
Materials
(forms,templates)
Numeric mainly with
Visualization of
Performance indicators some Graphs and Charts
Database Backup only
Experience Repository
Yes
Defect Management
Yes
Configuration
Management
User manual provided
Process Guide / User
Manual
Minimal
User Friendliness
Partially
Follow SEI provided
forms and templates
22
Point
TSPi Workbook
tVista
Manual
No
No
No
Manual
No
No
No
Automatic
.
No
Partially
Partially
Yes
Numeric
Only
No
Yes
Yes
Numeric Only
-
No
Yes
No
No
Yes
Yes
No
1 page manual provided
-
Nominal
Fully
Nominal
Fully
-
2.3 Fully automated tool for collection, analysis, visualization,
interpretation of data
The Hackystat1 system is an automated tool for software process improvement and “a
technology initiative and research project that explores the strengths and weaknesses of a
developer-centric, in-process, and non-disruptive approach to empirical software project data
collection and analysis” [Johnson, 2001]. Hackystat is one of the first third generation tools as it
provides fully automated support for metrics collection and analysis, which eliminates overhead
and context switching. However, it changes the kind of metrics data that is collected, and
introduces new privacy-related adoption issues of its own. The early tools build to support PSP
were first generation tools and only contained manual support, while second generation tools
had semi-automated support. As stated in Table 2.2, Hackystat makes both data collection and
analysis entirely automatically. A screen-shot of Hackystat browser interface is show below.
Figure 2.12: Hackystat Browser interface [Johnson, 2008]
1
http://hackystat.googlecode.com
23
2.3.1 Architecture
In Hackystat the data collection is sensor-based. These software sensors are attached to the
development tools and automatically send data to a centralized web server.
Its architecture can be described as a client server system as shown in Figure 2.13. The
“clients” are the developer's software tools (purple) and for each of these tools there is a custom
sensor (i.e., plug-in for the given tool, in yellow) that automatically monitors characteristics of
the developer’s process. This data is specific to the tool the sensor was developed for.
Furthermore, in case that a network connection is unavailable, the data is cached and resent
later. The data collected by the sensors is transmitted to the server (Hackystat Web Server, in
green) using SOAP. The web service maintains a data repository for each developer and runs
analysis on the “raw” data sent by the sensors everyday. Further analysis can be invoked in the
Web Browser interface, in order to help developers interpret the data and use it to make
improvements. The result of the analysis can be automatically emailed to the developers for
feedback when new, unexpected, and/or potentially interesting analysis results become available
[Lofi, 2005].
Figure 2.13: Hackystat Architecture [Lofi, 2005]
2.3.2 Sensors and Metrics
The summary of Hackystat's original sensor data types is show in Table 2.4. Every sensor
data type is partial because as Philip Johnson says in [Johnson, 2001] :
“The system cannot collect the total project effort of a developer, or all of the
defects that occur, or all of the times a certain file is edited or a system is compiled.
A key validation task is to ensure that useful developer feedback can be provided
even in the face of partial data about the product and process.”
24
Table 2.4: Summary of sensor data types [Johnson, 2001]
Sensor Data Type
ToolTime
IdleTime
Activity
File Name
Module
Total Size
Differential Size
C/K Complexity
Defect/Syntax
Defect/Unit test
CM/Tag
Defect/Runtime
Description
The time the developer is “busy” using a development environment tool.
ToolTime data appears to be accurate only to within 15 minutes for a given
session.
The time during which a development environment tool is running but the user is
not actively engaged with it such that activities are being recorded.
An event that occurs while using a development environment tool. Activities are
used both to detect idle time and to detect events such as file modification,
compilation errors, and so forth.
The name, including the directory path, of a file manipulated by a development
environment tool.
An aggregation of files. Modules can be identified through explicit
representations such as “projects”, or implicitly through common directory
structures and co-occurrence within recorded activity sequences.
Refers to a collection of sensor data types that calculate different representations
of the total size of a software product. The authors' initial total size measure will
leverage their prior research results on structural size measurement and the
resulting grammar-based tool called LOCC1.
Measures the difference in size between two successive versions of a system.
LOCC implements one approach to differential size measurement that has been
successfully applied to incremental project planning.
Measures the six Chidamber-Kemerer (C/K) object-oriented metrics: Weighted
methods per class, depth of inheritance tree, number of children, coupling
between object classes, response for a class, and lack of module cohesion. Initial
language targets are Java and C++. [Hitz and Montazeri, 1996]
The unsuccessful compilation of a file due to a syntax error. The sensors will not
attempt to infer what specific syntax error occurred or how many were in the file.
Invocation of a specific unit test and its result (pass/fail).
An aggregation of files that have been tagged as belonging to a release.
The unsuccessful compilation of a file due to a syntax error. The sensors will not
attempt to infer what specific syntax error occurred or how many were in the file.
The unsuccessful termination of a program due to a run-time error. Hackystat can
determine only that a run-time error occurred and the “top-level” reason (such as
“Null pointer exception”).
In Table 2.5 all tools with Hackystat sensors are shown, their respective metric and type of
tool. Currently sensors are implemented for IDEs, office productivity applications, size
measurement, testing, configuration management and defect tracking tools. Some of these
sensors are available as part of the Hackystat binary distribution (labeled as Released), others
are labeled as Experimental, meaning that they are currently under development but are not part
of the binary distribution. The sensors labeled as Archive are sensors that have been worked on
in the past, but are not longer under active development.
1
LOCC is an extensible system for producing hierarchical, incremental measurements of work product size that
are useful for estimation, planning, and other software engineering activities. [Dane, 1999]
25
Table 2.5: Hackystat Sensors and Metrics (adapted from [Lofi, 2005])
Measure
Metric
Sensor
Type of Tool
Label
Effort
Active time
Eclipse
IDE
Released
Emacs
IDE
Released
JBuilder
IDE
Released
Vim
IDE
Released
Visual Studio
IDE
Released
MS Office
Office
Released
Review time
Jupiter
Code Review
Released
Size
BCML
Structural Metrics
Optional
CCCC
Structural Metrics
Released
LOCC
BCML
CCCC
Dependency-Finder
Junit
Structural Metrics
Structural Metrics
Structural Metrics
Structural Metrics
Testing
Released
Optional
Released
Released
Released
JBlanket
Testing
Released
Eclipse
IDE
Released
CPPUnit
JBlanket
JIRA
Jupiter
Ant
Testing
Testing
Issue Management
Code Review
Build
No Information
Released
Released
Released
Released
Unix CLI
Build
Released
Eclipse
IDE
Released
CVS
Configuration
Management
Released
Subversion
Configuration
Management
Released
Filemetrics
OO metrics
Complexity
Testing
Defects
Build
Versioning
Profiling
Call dependency
Unit test results
Coverage
Trouble tickets
Review defects
Build invocation
Commits
Load testing
Parallel computing
hackyLoadTest
hackyHPC
26
Released
Released
2.4 Statistical Process Control
In this topic some process control methods are shown in order to understand what the tool,
that this dissertation will result in, is going to adapt and interpret in order to fully automate the
software process performance analysis and corresponding improvement recommendation.
Basically Statistical Process Control is a comparison of what is currently happening with
what happened in the past. A snapshot of how the process will perform is taken and the control
limits for the excepted measurements of the output of the process are calculated. Then actual
data from the process is collected and compared with the control limits previously established.
Measurements that fall off this control limits are investigated and some could later be discarded
[NIST/SEMATECH, 2010]. Key monitoring and investigating tools include:
•
Histograms
•
Check Sheets
•
Pareto Sheets
•
Cause and Effect Diagrams (also called Fishbone diagrams)
•
Defect Concentration Diagrams
•
Scatter Plot
•
Control Charts
Some examples of these tools are illustrated in Figure 2.14.
27
Figure 2.14: Examples of SPC Tools (Adapted from [Statit Software, 2007])
2.5 Conclusions
After the analysis conducted during the state of the art review it is clear that there are a
great number of tools for PSP and TSP. However, this abundance has not resulted in the final
resolution of the problem address so far. Although some of the previously mentioned tools can
provide automated or semi-automated data collection, none of them really automate the data
analysis part. This is a major shortfall of these tools as the peformance analysis is, as stated in
chapter 2.1.6, where software engineers analyze and optimize their process performance. The
“automated” data analysis of these tools is the conversion of all the data into graphs and sheets,
but this requires an experienced analyzer to interpret all this data, and even so the amount of
data can be overwhelming. The main focus of this dissertation is the automation of the
28
performance data analysis of the PSP, and as shown in this document, to this date, there is no
tool that is capable of such.
29
3 Data Analysis and Improvement
Actions
In this chapter it is presented a more detailed description of the problem addressed in this
dissertation and the proposed solution.
As mentioned in the introduction, high maturity software development processes can
generate a huge amount of data that needs to be processed and analyzed. Earlier in this
document it was shown that there are a lot of tools that gather and process all this data.
Generally the data is presented in graphs to the user for a thorough analysis but none that does
what is proposed here.
The problem is that a great deal of knowledge and experience is needed to make the
correct analysis of all this data, identify the problems and come up with improvement and
corrective actions for them. If a programmer with no experience in PSP, or similar high maturity
software development processes, looks at Figure 1.1, a visualization of the data in graphs of a
known tool, he probably couldn't make sense of it, and even if he could, he would not have the
knowledge on how to correct or improve the problems he identified in those graphs. So the need
of a tool that automates this analysis is quite clear here, as even to an experienced PSP
programmer it would make the analysis a lot easier and quicker.
In order to automate this data analysis a study is required to identify the key performance
indicators and all factors that affect or are affected by them.
The PSP Student Workbook (chapter 2.1.5) only has analysis tools that basically
transforms the data into graphs of known performance indicators (that require some
calculations), as Time Estimation Error, Size Estimation Error and Productivity. This is going to
be the basis for the tool built as a result of this dissertation. All the data gathered using this tool
is stored in a database that is going to be used in this dissertations tool prototype.
30
Data Analysis and Improvement Actions
3.1 Solution
To cope with the problem described a thorough investigation of the performance indicators
was made in order to come up with an adequate Performance Model. It is common knowledge
that delivering the product on time is one of the most important things in every project, from
small companies to big enterprises. One of the biggest strengths that SEI claims on their PSP
and TSP is the delivery of the product on time (or with comparatively little deviation) and
virtually free of defects (as low as 0.06 Defects/KLOC as shown in Figure 2.10).
So as the most important and extensive performance indicator seems to be time related we
can use the Time Estimation Error (TEE from the PSP) to evaluate our time estimation accuracy
(the deviation between the estimated time and the actual time). It will be shown that the TEE is
largely influenced by other very important performance indicators and a very broad key
performance indicator.
3.1.1 Time Estimation Error
Time Estimation Error (also known as Time Estimating Accuracy) is calculated using the
following formula:
TEE =
Actual Time− Estimated Time
Estimated Time
(3.1)
The TEE is measured in percentage, so the result should be multiplied by 100.
By doing a little bit of math, presented in table Table 3.1, it is shown that TEE is directly
affected by Size Estimation Error and Productivity Estimation Error. Note that in the PSP, one
arrives at a time (effort) estimate by first producing a size estimate (size of the deliverables) and
then combining the size estimate with a productivity estimate (size units/time units) based on
historical data.
Table 3.1: TEE calculations
1
Estimated Time=
2
Actual Time=
Estimated Size
Estimated Productivity
Actual Size
Actual Productivity
Actual Size
1+2
TEE =
Actual Productivity
Estimated Size
31
−1
Actual Size
so
TEE =
Estimated Size
Actual Productivity
−1
4
SEE=
Actual Size−Estimated Size
Actual Size
=
−1
Estimated Size
Estimated Size
5
PEE =
Actual Productivity−Estimated Productivity
Actual Productivity
=
−1
4+5
TEE =
SEE+1
−1
PEE+1
so
TEE +1=
SEE+1
PEE+1
The proposed control limits for the TEE performance indicator value is shown below, in
Figure 3.1.
If the value is in Green there's no problem, if it is yellow there is probably a problem
somewhere in the PIs that affect TEE and if the value is in Red there's certainly a problem with
this PI. The values of the graphs are based on goal values usually considered for PSP and TSP
projects, present on the literature or orally stated by experts that are "Certified Personal
Software Process (PSP) Developer" and "Authorized PSP Instructor” qualified by SEI of
"Carnegie Mellon University". Note that this applies for all the following control limits graphs
in this document. One should also note that all the following control limits graphs are only for
demonstration proposes only, as some of them are limited because of this. The real Performance
Indicators maximum and minimum values can be different. For example, TEE maximum value
is not limited to 100%.
32
Time Estimation Error Control Limits
-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Figure 3.1: Time Estimation Error recommended Controls Limits graph
Now the question is, what affects Size Estimation Error and Productivity Estimation Error?
3.1.2 Size Estimation Error
SEE, also know as Size Estimating Accuracy, is defined as:
SEE=
Actual Size
−1
Estimated Size
(3.2)
The SEE is measured in percentage, so the result should be multiplied by 100.
But the question is what affects SEE and the formula does not directly answer that. So how
is the size estimated?
PSP uses a method called PROBE to estimate size and effort. Using this method
programmers store all their historical data of the effort and the size of past projects, broken into
individual parts, classified by type (logic, calculation, etc.). Then this historical data is used to
estimate the size and effort of a project's parts that correspond in type and sizes to the historical
individual parts. It uses multiple formulas based on linear regression to calculate the estimates.
The basic steps of the size estimating procedure are shown in Figure 3.2.
33
Figure 3.2: The Probe Estimating Method [Humphrey, 2005]
In this formulas, used by PROBE, were identified three key indicators that affect the Size
Estimation Error. These indicators are the Missing Parts, the Expurious Parts and Part
Estimation Error.
Part Estimation Error is the percentage of error in the Identified Parts, parts that were
estimated and used. PEE is calculated by:
PartEE %=
∑ ( Identified Part Actual Size) −1
∑ ( Identified Part Estimated Size)
The PartEE is measured in percentage, so the result should be multiplied by 100.
34
(3.3)
Missing Parts are the parts that were estimated but not used in the project. The percentage
of Missing Parts is calculated by:
MP %=
∑ ( Missing Part Actual Size)
( ∑ (Missing Part Actual Size)+∑ ( Identified Part Actual Size))
(3.4)
The MP is measured in percentage, so the result should be multiplied by 100.
Expurious Parts are the parts that were not planned (nor estimated) but were used in the
project. The percentage of Expurious Parts is calculated by:
EP %=
∑ ( Expurious Part Estimated Size)
( ∑ ( Expurious Part Estimated Size)+∑ ( Identified Part Estimated Size))
(3.5)
The EP is measured in percentage, so the result should be multiplied by 100.
The proposed control limits for the values of Size Estimation Error (SEE), Part Estimation
Error (PartEE), Missing Parts (MP) and Expurious Parts (EP) are shown below, in Figure 3.3,
Figure 3.4, Figure 3.5, Figure 3.6. The Missing Parts value is always positive or zero because it
is calculated using the parts that were estimated but not used and the Expurious Parts value is
always negative or zero because it is calculated using the parts that were not estimated but were
used.
Size Estimation Error Control Limits
-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
Figure 3.3: Size Estimation Error recommended Controls Limits graph
35
100%
Part Estimation Error Control Limits
-90%
-70%
-50%
-30%
-10%
10%
30%
50%
70%
90%
-100% -80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Figure 3.4: Part Estimation Error recommended Controls Limits graph
Missing Parts Control Limits
0%
10%
20%
30%
40%
50%
60%
70%
80%
Figure 3.5: Missing Parts recommended Controls Limits graph
36
90%
100%
Expurious Parts Control Limits
-100%
-90%
-80%
-70%
-60%
-50%
-40%
-30%
-20%
-10%
0%
Figure 3.6: Expurious Parts recommended Controls Limits graph
So now we know what affects the SEE, and how to calculate everything, but we still need
to know what affects Productivity Estimation Error.
3.1.3 Productivity Estimation Error
Productivity is the ratio of a product’s size to the time expended to develop that product,
generally measured as size measure per hour [Pomeroy-Huff, et al., 2005].
Productivity Estimation Error is calculated using the following formula:
PEE=
Actual Productivity
−1
(3.6)
This formula does not directly suggest any performance indicator. However, since, in the
PROBE method, the estimated productivity is based on historical productivity (past projects) , it
is clear that the PEE is affected by the stability of the productivity. If the productivity is not
stable, it cannot be used for prediction purposes.
The PEE is measured in percentage, so the result should be multiplied by 100. The
proposed control limits for the PEE performance indicator value is shown below, in Figure 3.7.
37
Productivity Estimation Error Control Limits
-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Figure 3.7: Productivity Estimation Error recommended Controls Limits graph
3.1.4 Productivity Stability
Productivity Stability is the variation between productivity in the current project and
historical productivity of past projects (average of latest projects). The Productivity Stability has
the following formula:
Productivity Stability=
Current Productivity−Historical Productivity
Historical Productivity
(3.7)
The Productivity Stability is measured in percentage, so the result should be multiplied by
100.
So why and how does Productivity Stability affect PEE? Because if the productivity is not
stable in the last projects there is a high probability that the PEE is going to have a bad value.
To better understand where the problem is located the Productivity Stability was separated
by phases:
•
Total Productivity Stability;
•
Plan and Postmortem Productivity Stability;
•
Detailed-Level Design and DLD Review Productivity Stability;
•
Code and Code Review Productivity Stability;
•
Unit Test Productivity Stability.
38
The proposed control limits for the Productivity Stability performance indicator value is
shown below, in Figure 3.8.
Productivity Stability Control Limits
-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Figure 3.8: Productivity Stability recommended Controls Limits graph
Going deeper into the problem, what affects Productivity Stability?
3.1.5 Process Stability
The Process Stability should affect Productivity Stability to some degree, because if the
process is not stable, that is, if the process used keeps changing (PSP0, PSP0.1, PSP1, etc), it is
likely that the productivity is not stable either. By changing the process, the programmer
changes the way he does the work, therefore he probably won't be as efficient as he was before.
The opposite can occur too, the programmer can have a higher productivity than in past projects
as he uses a more advanced level of PSP.
The way to calculate this performance indicator is quite simple; if the process used in the
current project is different than the process used in the previous project, then the process is not
stable. The Process Stability is measured as a Boolean value (true or false).
3.1.6 Defect Density
Defect density is the number of defects found per size measure [Pomeroy-Huff, et al.,
2005].
39
The defect density affects the productivity stability as a programmer can never be certain
of how much time he is going to take to correct a specific defect, it is quite unpredictable. So if
there is a high defect density, it is highly probable that the productivity is going to be affected
by it as large variations of productivity from project to project can occur (depending on how
lucky we are in finding the location of the defects).
In order to better understand where the problem is, the defect density was divided into two
performance indicators, the Defect Density in Unit Test (DDUT ) and the Total Defect Density
(TotalDD). Both their formulas are given below.
DDUT =
#Defects found in Unit Test
Actual Size ( KLOC )
TotalDD=
(3.8)
#Defects found
Actual Size ( KLOC )
(3.9)
Both these performance indicators are measured in defects per thousand lines of code
(Defects/KLOC).
The proposed control limits for the Defect Density in Unit Test and Total Defect Density
values are shown below, in Figure 3.9 and Figure 3.10. These values are based on the the
process quality index (PQI). The PQI is produced by taking the product of the five dimensional
values of the quality profile to produce a single quality figure of merit [Humphrey, 1998].
DDUT Control Limits
0
5
10
15
20
25
30
35
40
Figure 3.9: DDUT recommended Controls Limits graph
40
45
50
TotalDD Control Limits
0
50
100
150
200
250
Figure 3.10: TotalDD recommended Controls Limits graph
There is a big difference between the values of the two graphs because, in the PSP is
expected that most defects are discovered and removed in Detailed-Level Design Review
(DLDR) and Code Review (CR) phases, before Unit Test (UT).
3.1.7 Process Yield
Process yield is the percentage of defects removed (in DLDR and CR phases) prior to
entering the compile phase (or before entering unit test if there is no compile phase) [PomeroyHuff, et al., 2005]. One can also call this performance indicator the Review Quality. The
process yield directly affects the defect density in unit test (DDUT) as the greater the percentage
of defects removed prior to the unit test (i.e. the greater the review quality), the lower the
DDUT is and other way around is also true.
This performance indicator has the following formula:
PY =
Defects removed before compile
Defects injected before compile
(3.10)
The Process Yield is measured in percentage, so the result should be multiplied by 100.
The proposed control limits for the Process Yield performance indicator value is shown
below, in Figure 3.11.
41
Process Yield Control Limits
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Figure 3.11: Process Yield recommended Controls Limits graph
3.1.8 Review Rate
Review rate is just the speed at which a programmer does a review (design review or code
review), i.e. the number of lines of code (or text) he reviews per hour. In the PSP, the speed of
design review is also measured a posteriori in a normalized way in regard to to the size of final
code and not the size of the design artifacts. It also provides a way to track and control review
times. In this case it is measure in lines of code per hour (LOC/hour).
This performance indicator is calculated by the following formula:
RR=
Actual Size
Review Time
(3.11)
The results of [Kemerer and Paulk, 2009] show that the PSP review rate is a significant
factor affecting defect removal effectiveness (i.e., the yield indicador described in 3.1.7), even
after accounting for developer ability and other significant process variables. The recommended
review rate of 200 LOC/hour or less was found to be an effective rate for individual reviews,
identifying nearly two-thirds of the defects in design reviews and more than half of the defects
in code reviews [Kemerer and Paulk, 2009].
[Humphrey, 2005] states that the rule of thumb for the control limits of RR is a maximum
of 200 LOC/hour and a recommended of about 100 LOC/hour for effective review rates.
The proposed control limits for the Review Rate performance indicator value is shown
below, in Figure 3.12.
42
Review Rate Control Limits
0
50
100
150
200
250
300
350
Figure 3.12: Review Rate recommended Controls Limits graph
3.1.9 Review to Development Ratio
Another set of useful quality measures is the ratio of the time spent in two process phases
[Humphrey, 2005]. In this case the Review to Development Ratio was used. This PI shows the
ratio between the time spent in reviews (DLD Review and Code Review) and the time spent in
development (Code and Design). R2D affects the Process Yield because if one spends little time
doing reviews and a lot of time coding and doing the design there is a high probability that a lot
of defects will go on undetected (and therefore will not be removed). This affects the PY
because PY is the percentage of defects removed prior to entering the compile phase (or before
entering unit test if there is no compile phase).
This performance indicator is calculated by the following formula:
R2D Ratio=
Actual Time of Reviews (DLD R+Code R)
Actual Time of Development ( DLD+Code)
(3.12)
The proposed control limits for the Review to Development Ratio performance indicator
value is shown below, in Figure 3.13.
43
Review to Development Ratio Control Limits
0
0,25
0,5
0,75
1
1,25
1,5
1,75
2
Figure 3.13: Review to Development Ratio recommended Controls Limits graph
3.2 Performance Model
So what is the result of all the aforementioned? The result is the performance model shown
in Figure 3.14. By using this performance model in the prototype it should be possible to locate
the origin of a problem and recommend a improvement action based on that.
This Performance model is for analyzing time estimating performance based on the main
indicator TEE. It shows all the indicators and relationships between them (what affects or may
affect what). Note that, even though there are a couple of relationships that require further
validation, these dependencies are usually referred in the materials of PSP/TSP and held on
experimental data.
44
Time Estimation Error
affects
cts
ff e
a
Productivity
Estimation Error
Plan and Post Mortem
Productivity Stability
affects
cts
fe
f
a
cts DLD and DLD Review
affe
affects
affe
cts
Process Stability
DDUT
af
fe
ct
s
Based on the formula
Review Rate
Defects Injected
(Total Defect Density)
affects
aff
ect
s
Review Quality
(Process Yield)
Symbol legend
Unit Test
affects
Expurious Parts
s
ct
fe
af
Total
affectsCode and Code Review
affects
Missing Parts
affects
affects
Size Estimation Error
Review to
Development Ratio
Validation required
Figure 3.14: Performance Model Tree
3.3 Recommended Improvement Actions
The recommended improvement actions are associated with the indicators that have the
values off the “good” or green zone. Table 3.2 shows the indicators that have associated
improvement actions. These are the indicators that are in the bottom levels of the tree shown in
Figure 3.14, and affect and influence their respective parents. These improvement actions were
defined based on the judgment and advice of PSP experts (more specifically, a PSP instructor).
45
Table 3.2: Recommended improvement actions
Indicator
Problems identified and
Suggested Improvement Actions
Missing Parts
Problem with the identification of the parts.
Recommended Actions:
•
A more careful Conceptual Design is
recommended, with more detail or time
spent in design.
Expurious Parts
•
A more careful Design is
recommended, with more detail or time
spent in design.
Problem in the relative-size table and/or
•
Review the types of the parts, their
relative size and related topics.
Productivity Stability – Plan and Postmortem Problem with the changes in the way of
working.
•
Try to keep a stable productivity and
check what has been modified lately.
•
If the productivity has increased try to
keep it stable at that new value in future
projects.
Productivity Stability – Detailed-Level Design Problem with the changes in the way of
and DLD Review
working.
•
•
projects.
Productivity Stability – Code and Code Review Problem with the changes in the way of
working.
•
•
projects.
46
Indicator
Problems identified and
Suggested Improvement Actions
Productivity Stability – Unit Test
Problem with changes in the way of working.
•
•
projects.
•
Try to keep the process stable.
•
Changing the process usually leads to a
variation in productivity.
•
Improve the process quality.
•
Do an analysis on the more frequent
and more expensive errors in order to
not repeat them, define preventive
actions.
•
If the value is too high, it is necessary
to reduce it by doing the review more
carefully and slowly.
•
If it is too slow (low value), do the
review in a more focused and efficient
way.
•
Check if the artifacts are
understandable.
•
If the value is too high, it is necessary
to do the review more carefully and
slowly or code faster and more
efficiently.
•
If the value is too low, do the review in
a more focused and efficient way or
take your time coding.
•
understandable.
Process Stability
Total Defect Density
Review Rate
Review to Development Ratio
47
3.4 Conclusions
The proposed solution should broad enough so that it is able to address and cover all the
necessary topics to solve the problem described. This includes the identification of performance
indicators (PIs), calculated from base measures, the definition of the Pis control limits, the
definition of the cause-effect relationships and the indication of improvement actions for each
cause. The performance model scope should be extensive enough to discover the origin of a
certain problem and allow the prototype implementation to recommend improvement actions
based on that. All of the research and development work done and concluded in this chapter was
verified and partially validated, as it is shown in the evaluation chapter (4.3) of this dissertation.
48
4 Implementation and Evaluation
This chapter presents the details of implementation of the tool that is the result of the
research done in this dissertation. It also presents the evaluation of the results.
4.1 Architecture of the solution
The architecture of the solution was the subject of an extensive study to understand the
best way to implement it. It had to be flexible enough so that one can extend the protototype
resulting of this dissertation as future work. But in the other hand, it had to allow to implement
the performance model described above with all those indicators and respective control limits.
The architecture was sub-divided into 3 parts here, for demonstration and explanation
purposes. In Figure 4.1 is shown the indicators part of the architecture. It has an abstract class
named Indicator that has a couple of properties and methods that all the indicators, that are subclasses of it, also have, in addition to their own methods.
49
Implementation and Evaluation
Figure 4.1: Architecture - Indicators
Figure 4.2 shows the control limits part of the architecture. It has an abstract class named
Limit that has two sub-classes named UpperBound and LowerBound. These classes, as their
name indicate, are the upper and lower bound limits of a given indicator. The control limits of
each indicator are all stored in a Performance Model class as shown further on. Note that the
abstract class Limit (and its children) has two properties that represent the green and yellow
limits. The nested type named lights is used for checking which is the color (Red, Yellow,
Green or Null if the value is invalid) of a value. It is the return value of the checkViolation()
function.
50
Figure 4.2: Architecture – Control Limits
In Figure 4.3 the performance model class is shown. This class stores the relationship
between the indicators in a tree collection 1 and maps the control limits of each indicator using a
Dictionary.
1
This is a generic tree collection by Nicholas Butler (CPOL license), available in the Code Project website at
http://www.codeproject.com/KB/recipes/treecollection2.aspx.
51
Figure 4.3: Architecture – Performance Model
4.2 PSP PAIR
The tool made was named PSP PAIR, from Personal Software Process Performance
Analysis and Improvement Recommendation. It should be easy and simple to use as its main
goal is to automate the performance analysis of one or more projects performed by the same
individual (with the data collected in the PSP Student Workbook) and suggest improvement
recommendations, therefore making this task way more simpler than it was.
The interface of the application is a quite usable and simple one. It has a button to load
new data from another database (Microsoft Access *.mdb file, because this is the extension
which the PSP Student Workbook export all its data), a button to load a new Performance model
(xml file, example shown in Figure 4.7) and a button to load the default values, i.e., default
database and default performance model. To start the analysis of the data the user needs to
select the project or projects to be analyzed from the list of projects that were on the database.
One can also choose to analyze all the projects at once by checking the “All” option on the list.
An example execution is shown in Figure 4.4.
PSP PAIR searches for a file named “PSP Assignments_be.mdb” in its content folder and
loads it automatically. If the file doesn't exist or is not found, a Load File dialog appears so that
the user can locate and load the file manually.
52
Figure 4.4: PSP PAIR Home
In Figure 4.5 an example execution is shown, given that the user has chosen to analyze the
data of the Projects 411 and 412.
The results report main objective is to make the visualization of the data really easy, so one
can understand the suggested improvement recommendations and analyze the data if needed.
The performance model used is shown in a treeView, as this helps understanding the
relationship between the indicators, and it shows all the indicators and its respective unit of
measure. The recommended control limits of each indicator are always shown in the first
column, followed by the value of a given indicator in a given project.
Without even understanding the details, the user should be able to tell that in the first row
of data associated with the indicator named “Time Estimation Error”, the value for Project 411
is "good", in the sense that this indicator's value is colored green. Similarly, one should be able
to see that the value associated with the indicator "Process Yield" of Project 411 is "bad", in the
sense that this indicator's is colored red. Finally, in Project 412, also for "Time Estimation
Error", the value is colored yellow, which means that it is neither good nor bad, but somewhere
in between. A bit like a “warning” sign, as it could suggest a potential problem.
The last two columns are always the average values of each indicator for the selected
projects and the percentage of red colored values of each indicator for the selected projects.
These two columns give the user an overview of the state of all projects he selected for analysis.
Although, if the user selected “All” in the main menu, the last two columns are the “All
(Average)” and “All (% of reds)”. They provide an overview of all the projects contained in the
loaded database, useful if the user wants to compare a specific project to the average of all the
projects of the database or if he wants to compare the projects he selected to overall picture.
53
One can also choose to see the suggested improvement actions for the all the projects on
the database (by selecting “All” in the main menu (Figure 4.4) and them clicking the respective
button in the Results Report) or just for the selected and shown projects (by clicking in the “All
Selected (Average)” button).
Figure 4.5: PSP PAIR Results Report
By clicking in the button of a given project, in the titles row of the table, the user can see
the details and the suggested improvement actions of that project. This opens a new window,
shown in Figure 4.6. It shows the project identification, the author of that project, the suggested
recommendation based on the values of the indicators and a treeView similar to the one shown
in Figure 4.5, but with the values, with the respective color, of the project selected.
The problems identified and improvement actions are also with the color of the value of
the indicator that trigger that problem. This helps understanding which problems should be
addressed first.
54
Figure 4.6: PSP PAIR Project Details
To load a new Performance model one must make a XML file similar to the one shown
below in Figure 4.7. The XML allows to create a tree like collection (with concepts like parent
and child) of indicators that is easy to understand and to load into the application. One can also
use the attributes of the nodes to establish new control limits. For example, it is easy to see that
in Figure 4.7 the root (or main) indicator is Time Estimation Error (TEE) and its control limits
are Green from -20% to 20% , Yellow from -30% to -20% and from 20% to 30% and that if the
value of TEE is below -30% and above 30% it is colored Red, as shown in the graph of Figure
3.1.
55
Figure 4.7: PSP PAIR XML Performance Model and Control Limits
56
4.3 Evaluation
To evaluate the prototype resulting of the research done in this dissertation, a Performance
Analysis Report (PAR) made on a PSP Course was used for a comparative analysis between
manual results (from the PAR) and the prototype results. This comparative analysis was made
by providing the prototype of the tool, PSP PAIR, with the same data from a set of projects that
was used to make this particular PAR [Faria, 2008]. Exception goes for program 7 (project 414)
as the data that was used for making the PAR document is not quite the same as the data used in
the PSP PAIR (some amendments were made that are not included in the database), despite this
the project was included in the comparative analysis because the values are quite similar and
valid for comparison purposes.
In order to do this comparative analysis a normalization of the results must be made so that
both set of results are related to the same things, as the results (problems and improvements
actions) of the PAR are, generally, for all the projects and not from a specific one as it's possible
in the PSP PAIR. However, in the PAR document, there are some conclusions and PIPs that are
related to a specific project. In some cases the project ID is not stated, but it is possible to
conclude what project the conclusions and Process Improvement Proposals (PIPs) belong to
based on the data. The comparative analysis of the results is shown below, in Table 4.1.
One must have in mind that the conclusions and PIPs of this PAR are all made by
analyzing a wide range of data, including artifacts and the actual knowledge of the engineer on
those specific projects. Even though much of the numeric data is all stored in the database, there
is a lot of things that the PSP PAIR tool cannot analyze, such as documents and data that is not
in the scope of this dissertation.
These programs/projects were made in the learning process of the PSP. As PSP has several
phases, program number 1 (project 408) uses the process PSP0 so it falls short on a lot of data
needed to PSP PAIR analyze the project properly. Program number 2 (project 409) uses PSP1,
project 410 and 411 use PSP2, and project 412, 413 and 414 use PSP2.1.
Despite all this, the results are quite similar, the conclusions arrived in the PAR document
(such as problems identified) are consistently present in the problems identified by the PSP
PAIR based solely on the data of the indicators and their relationships represented in the
performance model.
57
Table 4.1: Comparative Analysis between PAR and PSP PAIR
Project
ID
Performance Analysis Report
PSP PAIR
Problems Time estimation Error value
Project
408
is too high, but this was
expected as no method was
used for time estimation and
no size estimation was made
(no historical data).
PIPs
The global PIPs do not apply
here because this project is
not using historical data for
estimation purposes.
Problems The process was changed.
PIPs
Project
409
Problems
Value of Time Estimation
Error is on red (72,5%).
Value of Indicator Total Defect
Density suggests a potential
problem.
Recommended Improve the process quality.
Do an analysis on the more
Actions
frequent and more expensive
errors in order to not repeat
them, define preventive actions.
Problems
Values of Productivity Stability
in Plan+Postmortem and in Unit
Test, and Process Stability
suggests a problem.
Problem with the changes in the
way of working. (from
Productivity Stability)
These are the global PIPs
Recommended Try to keep a stable
based on the data from all the
productivity and check what has
Actions
projects for the Time
been modified lately.
Estimation Error.
If the productivity has increased
P1: Stabilize and make more
try to keep it stable at that new
efficient the DLD process to
value in future projects.
improve and stabilize
productivity: avoid expensive
DLD representations (e.g.,
Changing it usually leads to a
equations in Word), avoid
long DLD documents
(compared to code size); do
Logic Specifications that can
be used as code comments
and can be easily written.
P2: Widen the size range to
have more basis for
estimations.
58
Project
ID
PSP PAIR
Problems Time underestimation is
Problems
caused by size
underestimation.
Size underestimation is due to
underestimation of relative
sizes. (corrected in
subsequent
programs/projects).
The process was changed.
PIPs
Project
410
Values of Indicators Part
Estimation Error, Productivity
Stability (Plan+PM, Code +
CR, UT) and Process Stability
suggests a problem.
Problem with the relative-size
table and/ or Problem with
the identification of the parts.
(from PartEE)
way of working (from
Productivity Stability)
Recommended Review the types of the parts,
their relative size and related
Actions
topics.
Estimation Error.
Try to keep a stable
long DLD documents
have more basis for
estimations.
Project Problems
411
Problems
59
in DLD + DLDR and in UT
suggests a problem.
Value of Productivity Stability
in Code + CR suggests a
potential problem.
way of working.
Project
ID
PSP PAIR
Actions
PIPs
Problems Inferior time estimation is
Problems
justified by process change
and more time spent in DLD
that was not gained in
subsequent phases.
Values of Productivity
Stability in Plan + Postmortem
and in DLD + DLDR suggests
a problem.
way of working.
Problem identified in Process
Stability.
Values of Indicators Total
Defect Density, Review Rate
and Review to Development
Ratio suggests a potential
problem.
Actions
PIPs
Review Rate: The value is too
low (slow RR), do the review in
a more focused and efficient
way. Check if the artifacts are
understandable.
R2D ratio: The value is too low,
it is necessary to do the review
more carefully and slowly.
Project
412
60
Project
ID
PSP PAIR
Problems Inferior time estimation can
Problems
be justified by a much smaller
program size (about one third
compared to previous
programs).
PIPs
Project
413
in Plan + Postmortem, in DLD
+ DLDR, in Code + CR and in
UT suggests a problem.
way of working.
Values of Total Defect Density
and Review Rate suggests a
problem.
Actions
Estimation Error.
long DLD documents
understandable.
have more basis for
estimations.
Project Problems Strong time underestimation
is due mainly to the time
414
spent in DLD.
61
Problems
Values of Indicators Missing
Parts, Part Estimation Error,
Review Rate and Review to
Development Ratio, suggests a
potential problem.
Problem with the relative-size
table and/ or Problem with the
identification of the parts. (from
PartEE and Missing Parts)
Values of Productivity
Stability in Plan + Postmortem,
in DLD + DLDR, in Code +
CR and in UT, and Total Defect
Density suggests a problem.
way of working.
Project
ID
PIPs
PSP PAIR
Recommended A more careful design is
recommended, with more detail
Actions
or time spent in design. (for
Estimation Error.
Missing Parts)
Review the types of the parts,
their relative-size and related
topics. (for PartEE)
Try to keep a stable
long DLD documents
have more basis for
estimations.
understandable.
R2D ratio: The value is too low,
it is necessary do the review in
way or take your time coding.
understandable.
62
Project
ID
PSP PAIR
Problems Time estimation accuracy
Problems
(same as time estimation
error).
Productivity, namely at DLD
phase.
Defect density in UT.
Based on the average of all the
projects, problems identified at:
•
•
•
•
Time estimation
error.
Total productivity
stability
Productivity stability
at DLD and DLD
Review
Defect Density in UT
Value of Productivity Stability
in DLD + DLDR suggests a
problem.
way of working.
Value of Total Defect Density
suggests a potential problem.
Global
PIPs
P1: Stabilize and make
more efficient the DLD
productivity and check what
Actions
process to improve and
has been modified lately.
stabilize productivity: avoid
expensive DLD
representations (e.g.,
long DLD documents
have more basis for
estimations.
63
5 Conclusions and Future Work
In this chapter it is presented the main contributions of this dissertation as well as a
summary of the work done. The guidelines for the future work are also presented.
5.1 Goal Achievement and Conclusions
The objectives initially proposed for this dissertation were almost completely attained. The
prototype of the tool does successfully implement the performance model that was made for this
dissertation with good results, but the recommended improvement actions are quite generic and
there was a lot of work that could be done to improve the application PSP PAIR. This is
explained with detail in the Future Work chapter (5.2).
The work done was quite satisfactory, as of all the goals and expected results defined in
chapter 1.3 were achieved, with the aforementioned shortcomings.
The main difficulty was to try to understand all of the concepts behind Personal Software
Process in order to produce a effective performance model. This was mostly due to the fact that
the knowledge I had about PSP and TSP in the beginning of this dissertation was very little.
This difficulty was overcome with a extensive study of PSP and with a great interest in
Software Engineering.
The performance model (chapter 3.2) elaborated does what was defined in the beginning of
the investigation for this dissertation:
•
identifies the performance indicators (PI) and their derivation formulas from base
measures;
•
defines the PI control limits;
•
defines cause-effect relationships;
•
indicate improvement actions for each root cause.
64
Conclusions and Future Work
All these topics are explained in detail in the solution chapter (3.1). Each indicator is
described and explained with the respective control limits and relationships with other
indicators. The performance model with the indicators and respective relationships are
illustrated in tree like diagram in Figure 3.14.
As shown in the evaluation chapter (4.3), in the comparative analysis the results were quite
similar, the conclusions arrived in the PAR (such as problems identified) were consistently
present in the problems identified by the PSP PAIR based solely on the data of the indicators
and their relationships represented in the performance model.
The results of this research support the idea that the performance model is valid and it can
provide a really good support to engineers using the PSP in their projects and help them speed
up the analysis of all the data they collected, by automating it. With some tweaks, it could
maybe be adopted by all engineers using PSP.
This comparative analysis was preceded by a testing phase as there was a lot of
calculations made for each indicator and their values needed to be verified. This was done by
comparing the values of the indicators in the PSP Student Workbook (chapter 2.1.5) with the
values of the calculations made by PSP PAIR. The only fluctuation of values verified was due
to the previously mentioned problem with the data of project 414 (the data is not exactly the
same in both places).
Initially, it was intended to validate the dependencies between indicators that do not result
directly from the formulas. This validation was supposed to be based on data from SEI about the
PSP courses (with hundreds of developers). But, unfortunately, this data was not provided in
time of this dissertation. Therefore, it is a future work to perform the validation of these
dependencies.
5.2 Future Work
The PSP PAIR tool has everything to be a quite good support in the analysis of data from
PSP projects, and due its architecture (chapter 2.3.1) it is quite easy to pick it up and expand it,
as the architecture is simple and understandable.
There are a few things that could help PSP PAIR be a lot better than it is right now,
starting with the improvement recommendations. Presently they are quite generic and are
associated with a specific indicator. There are a couple of ways to improve this, as taking into
account multiple indicators and their values to give some sort of ranking to the improvement
recommendations, possibly using a classification system to do this. Maybe if one indicator is
more important than other it should have more weight in the final recommendations than the
other.
The PSP PAIR only accepts databases in the form of Microsoft Access (*.mdb) files, as the
database used as basis for this work is the one that SEI uses in their PSP courses and it is in this
format. However, it should be possible to import data from other sources, as SQL or, more
simple, Excel.
This tool could also increase its value exponentially if it included some kind of feedback
from the user, and then used this feedback to improve its results (as suggested improvement
65
Conclusions and Future Work
actions). For example, the user could give feedback by identifying which suggested
improvement actions were useful and which were not, then this feedback would be used to
improve future improvement recommendations.
One last obvious thing to do as future work is the extension of this kind of tool for Team
Software Process (TSP) as it would be way more useful for companies and organizations. But it
would also be way more difficult to accomplish this, as it involves a lot more variables than
PSP.
66
References
[Burton and Humphrey, 2006]
Symposium, 2006.
D. Burton and W. S. Humphrey, "Mining PSP Data," in TSP
[Dane, 1999] J. A. Dane, "Modular program size counting," master's thesis, Dept. of
Information and Computer Sciences, Univ. of Hawaii, Honolulu, Hawaii, 1999.
[Davis and Mullaney, 2003] N. Davis and J. L. Mullaney, "The team software process (TSP) in
practice: A summary of recent results," 2003.
[Disney and Johnson, 1998]
A. M. Disney and P. M. Johnson, "Investigating data
quality problems in the PSP," presented at the Proceedings of the 6th ACM
SIGSOFT international symposium on Foundations of software
engineering, Lake Buena Vista, Florida, United States, 1998.
[Fahmi and Ho-Jin, 2008] S. A. Fahmi and C. Ho-Jin, "A Survey on Team Software Process
Supporting Tools," in Convergence and Hybrid Information Technology, 2008.
ICCIT '08. Third International Conference on, 2008, pp. 987-990.
[Faria, 2008] J. P. Faria, "Performance Analysis Report," in PSP Advanced Course, SEI, ed,
2008.
[Harrington, 1991] H. J. Harrington, Business process improvement: The breakthrough
strategy for total quality, productivity, and competitiveness: McGraw-Hill
Professional, 1991.
[Hayes and Over, 1997] W. Hayes and J. W. Over, "The Personal Software ProcessSM
(PSPSM): An Empirical Study of the Impact of PSP on Individual Engineers,"
1997.
[Hitz and Montazeri, 1996] M. Hitz and B. Montazeri, "Chidamber and Kemerer's Metrics
Suite: A Measurement Theory Perspective," IEEE Trans. Softw. Eng., vol. 22, pp.
267-271, 1996.
[Humphrey, 2005] W. Humphrey, Psp (sm): a self-improvement process for software
engineers, 2005.
[Humphrey, 2006]
W. Humphrey, Tsp (sm)-coaching development teams, 2006.
[Humphrey, 1998] W. S. Humphrey, "The software quality profile," Software Quality
Professional, vol. 1, pp. 8-18, 1998.
67
References
[Humphrey, 2002a] W. S. Humphrey, "Personal Software Process (PSP)," in Encyclopedia of
Software Engineering, ed: John Wiley & Sons, Inc., 2002a.
[Humphrey, 2002b] W. S. Humphrey, "Team Software Process (TSP)," in Encyclopedia of
Software Engineering, ed: John Wiley & Sons, Inc., 2002b.
[Initiative]
T. S. P. D. Initiative. Functionality for Individuals [Online]. Available:
http://www.processdash.com/functionality-personal Retrieved July 2, 2011
[Johnson, 2001]
P. M. Johnson, "Project Hackystat: Accelerating adoption of empirically
guided software development through non-disruptive, developer-centric, in-process
data collection and analysis," Department of Information and Computer Sciences,
University of Hawaii, 2001.
[Kemerer and Paulk, 2009] C. F. Kemerer and M. C. Paulk, "The impact of design and code
reviews on software quality: An empirical study based on psp data," Software
Engineering, IEEE Transactions on, vol. 35, pp. 534-550, 2009.
[Kenett, et al., 1999]
R. S. Kenett, et al., Software Process Quality: Management and
Control: Marcel Dekker Inc, 1999.
[Laboratory, 2010] C. S. D. Laboratory. (2010). Hackystat [Online]. Available:
http://code.google.com/p/hackystat/
[Lofi, 2005] C. Lofi, "Continuous {GQM}: An automated framework for the Goal-QuestionMetric paradigm," 2005.
[Nasir and Yusof, 2005]
M. H. N. M. Nasir and A. M. Yusof, "Automating a modified
personal software process," Malaysian Journal of Computer Science, vol. 18, pp.
11–27, 2005.
[NIST/SEMATECH, 2010]
NIST/SEMATECH. (2010). e-Handbook of Statistical
Methods [Online]. Available: http://www.itl.nist.gov/div898/handbook/
Retrieved July 3, 2011
[Pomeroy-Huff, et al., 2005]
M. Pomeroy-Huff, et al., "The Personal Software Process
(PSP) Body of Knowledge, Version 1.0," 2005.
[Sciences, 2008]
I. E. o. t. S. Sciences. (2008). Measurement [Online]. Available:
http://www.encyclopedia.com/doc/1G2-3045301499.html Retrieved June
22, 2011
[SEI-Training, 2011] SEI-Training. (2011). Personal Software Process (PSP)
Advanced [Online]. Available: http://www.sei.cmu.edu/training/p19b.cfm
Retrieved July 20, 2011
[Shin, et al., 2007] H. Shin, et al., "Jasmine: A PSP Supporting Tool," in Software Process
Dynamics and Agility. vol. 4470, Q. Wang, et al., Eds., ed: Springer Berlin /
Heidelberg, 2007, pp. 73-83.
68
References
[Sison, 2005] R. Sison, "Personal software process (PSP) assistant," in Software Engineering
Conference, 2005. APSEC '05. 12th Asia-Pacific, 2005, p. 8 pp.
[Statit Software, 2007]
I. Statit Software. (2007). Statit Custom QC [Online].
Available: http://www.statit.com/statitcustomqc/ Retrieved July 3, 2011
[Team,
1997] D. S. Team. (1997). PSP Studio
[Online].
http://csciwww.etsu.edu/psp/ Retrieved July 20, 2011
69
Available:

Automated Software Processes Performance Analysis and

Transcription

Similar documents

Midnight Club 3: DUB Edition

We`re here when you need us!

How to use the PSP® System

JASC PAINT SHOP PRO

Personal Media Recorder USER MANUAL In one touch... iRecord

Yuengling Lager Chili Dip

AN ALSATIAN REPAST - Barrington Area Library

Carols, scents and Christmas sparkle

HMCS Athabaskan visits NASDAQ during Fleet Week New York City

Delicious Magazine – Australia – December 2015