Cost of Quality - HCL Technologies

Cost of Quality
Needs Your Attention Not Sometimes
But Always
Authors
Manas Chakraborty, Symu Koul, & Suresh
Sundararajan, ETS-IV&VS
HCL Technologies, Chennai
Dr Usha Thakur, ATS Technical Research
HCL Technologies, Chennai
CoQ: Needs Your Attention Not Sometimes But Always
© 2010, HCL Technologies Ltd.
June 2010
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 2
Contents
Introduction ............................................................................................................................................................ 4
Purpose ................................................................................................................................................................... 4
CoQ and Its Relevance ............................................................................................................................................ 5
CoQ Service from HCL ............................................................................................................................................. 7
CoQ as an Integral Part of SDLC ........................................................................................................................ 12
Benefits ................................................................................................................................................................. 14
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 3
Introduction
In and of themselves tools and technologies are not very useful and when they end up in the hands of people
who do not understand their significance, they are as good as not having any. By the same token a software
solution is not very valuable in itself, but it can serve as a powerful tool for enabling and transforming business
operations; in fact, it has the potential of giving impressive business results.
Of late, IT roles, activities, solutions, and services have come under the scanner of business executives and are
being assessed for the extent to which they create (business) value i.e., help company executives in achieving
strategic objectives such as giving their company a competitive edge, increasing profit margins, cutting overall
operational costs, retaining existing customers and obtaining new ones, increasing customer satisfaction,
improving internal communication, and speeding up time-to-market for software products and services. It is,
therefore, not surprising that the impact of software solutions and services is increasingly being measured in
terms of business criteria such as Return on Investment (ROI) and Return on Quality (ROQ). Moreover,
businesses are no longer willing to work within the confines of IT silos; they want IT to adapt easily to the
dynamic needs of business without compromising on quality.
If the goal of software solutions is to improve the way a business is run so that it becomes more profitable and
healthy, then Cost of Quality (CoQ) is a key approach for achieving that kind of improvement. The principles of
CoQ are well established in the manufacturing industry, given its long history.1 Their relevance in determining
the quality of software solutions has been understood only in the last couple of decades, mostly in the context
of the costs incurred as a result of delivering poor quality software. The main idea behind a CoQ approach is to
increase revenue and decrease operational cost by reducing rework and maximizing opportunities.
Purpose
The purpose of this paper is to illustrate the significance of incorporating CoQ principles in software solutions
and to evaluate its impact on business. It also highlights HCL’s capabilities in analyzing IT projects and
implementing the CoQ approach across business domains, which bear ample testimony to the fact that CoQ is
a business opportunity and not a cost centre. This paper will be of interest to IT professionals who are
exploring ways to help their companies speed up time-to-market and cut operational cost.
1
For an interesting read on this see, Matthew Littlefield, "The Cost of Quality: Benchmarking Enterprise Quality
Management," (2007) http://www.hertzler.com/php/portfolio/white.papers.php [April 2010]->represents when the site
was accessed.
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 4
CoQ and Its Relevance
Any discussion on the subject of CoQ leads one to acknowledge the contributions of Juran, Gryna and Crosby2
who emphasized that it is more economical to pay attention to quality early in the lifecycle of a product (by
incurring prevention and appraisal costs) than to wait for products to fail (for non conformance) and then fix
them (thereby incurring failure costs). Realizing its significance, the Software Engineering Institute (SEI) has
made quality an integral part of its Capability Maturity Model Integration (CMMI) best practices.3 Furthermore,
many organizations have made an effort to improve the quality of their software by integrating the principles
of Six Sigma.4
According to Crosby,




Quality means conformance to requirements, not goodness
Quality is achieved by prevention, not appraisal
Quality has a performance standard of Zero Defects, not acceptable quality levels
Quality is measured by the Price of Non-conformance™, not indexes5
In other words, CoQ refers to the cost that companies undertake to rectify the mistake they made and have to
correct it. By the same token CoQ refers to the cost companies incur to ensure that the software solutions they
deliver to their clients are free of defects. To achieve that goal, companies need to incur costs (read ‘invest’) in
the form of resources, tools and activities throughout the lifecycle of software to identify and prevent defects
as well as potential failures. The seriousness of problems arising from software errors was very well highlighted
2
For their earliest contributions the CoQ debate see J. Juran and F. Gryna, Quality Control Handbook 4th ed., McGrawHill, New York, 1988. See also P. Crosby, Quality Is Free: The Art of Making Quality Certain McGraw-Hill, New York, 1979.
3
For an implementation of quality principles, see Dennis R. Goldenson, et al., "Why Make the Switch: Evidence about the
Benefits of CMMI," (2004) http://www.sei.cmu.edu/library/assets/evidence.pdf [April 2010].
4
While adherence to Six Sigma principles in software has its share of critics, they have been used for improving the
quality of software and reducing the overall CoQ. See, for instance, Dr. Bill Eventoff, "Applying Six Sigma to Software
Development: A Practical Guide," (2002) http://www.swqual.com/SQGNE/presentations/200405/Eventoff%20May%202005.ppt [April 2010]. See also Maneesh Aggarwal, "Six Sigma Meets Software Development,"
http://www.isixsigma.com/index.php?option=com_k2&view=item&id=1307:&Itemid=49 [April 2010].
5
Cited in Kevin Weiss, "Biography - Philip B. Crosby: June 18, 1926 – August 18, 2001,"
http://www.philipcrosby.com/25years/crosby.html [April 2010]. For an interesting article on calculating the ROI of CoQ,
see Sandra A Slaughter, et al., "Evaluating the Cost of Software Quality," COMMUNICATIONS OF THE ACM August
1998/Vol. 41, No. 8. This paper is also available at http://www.cse.buffalo.edu/~hungngo/SCE-Papers/p67-slaughter.pdf
[April 2010].
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 5
by a study commissioned by the Department of Commerce's National Institute of Standards and Technology
(NIST) in the US in 2002.6 The study acknowledged that
[s]oftware is error-ridden in part because of its growing complexity. The size of software
products is no longer measured in thousands of lines of code, but in millions. Software
developers already spend approximately 80 percent of development costs on identifying and
correcting defects, and yet few products of any type other than software are shipped with such
high levels of errors. Other factors contributing to quality problems include marketing
strategies, limited liability by software vendors, and decreasing returns on testing and
debugging, according to the study. At the core of these issues is difficulty in defining and
measuring software quality. The increasing complexity of software, along with a decreasing
average product life expectancy, has increased the economic costs of errors. The catastrophic
impacts of some failures are well-known. For example, a software failure interrupted the New
York Mercantile Exchange and telephone service to several East Coast cities in February 1998.
But high-profile incidents are only the tip of a pervasive pattern that software developers and
users agree is causing substantial economic losses.7
In 2008 IDC conducted a survey among quality managers of 139 North American companies. The survey
respondents estimated that if 100% of software defects were addressed and remediated prior to production,
they would experience a 32% cost savings.8 In 2008 Original Software conducted a qualitative opinion survey of
senior IT executives and CIOs from organizations with average annual revenue of $5.8billion and an average IT
budget of $764 million. The survey revealed that over 60% of them felt a strong need to improve the quality of
software applications.9
6
See Michael Newman, "Software Errors Cost U.S. Economy $59.5 Billion Annually: NIST Assesses Technical Needs of
Industry to Improve Software-Testing," (2002) http://www.nist.gov/public_affairs/releases/n02-10.htm [April 2010]. See
also Bill Hoffman, "Battling Software Defects One Developer at a Time," (February 23, 2010)
http://www.itbusinessedge.com/cm/community/features/guestopinions/blog/battling-software-defects-one-developerat-a-time/?cs=39611 [April 2010].
7
Ibid. Similar concerns were highlighted by Herb Krasner. His research led him to conclude that "[t]ypical manufacturing
CoQ, rang[es] from 5 to 25 percent of company sales, [which] contrasts significantly with CoSQ. With the present state of
software engineering practice we can expect CoSQ to range from 10 to 70 percent of development costs. Even accounting
for the margin between production costs and sales, CoSQ appears to be roughly twice manufacturing CoQ." See Herb
Krasner, "Using the Cost of Quality Approach for Software," (February 2006), p. 7,
http://lifelong.engr.utexas.edu/pdf/sqi/cosq-article-new06.pdf [April 2010].
8
See Melinda-Carol Ballou, "Improving Software Quality to Drive Business Agility," (June 2008)
http://www.coverity.com/library/pdf/IDC_Improving_Software_Quality_June_2008.pdf [April 2010]. This survey was
sponsored by Coverity Inc.
9
See "Software Quality and Testing: A CIO Perspective," http://www.origsoft.com/whitepapers/ [April 2010].
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 6
CoQ Service from HCL
HCL’s CoQ service is about total quality management of software in the pre or post development phase. It is
most helpful in determining the structure of cost across the lifecycle of software and in developing and
implementing best practices strategies. This service includes:
 Thorough assessment of Quality Assurance (QA) activities and cost structure in four areas: planning,
engineering, review and testing, and rework
 Detailed gap analysis between corporate vision and existing QA practices in relation to where the
organization concerned wants to be
 Root-cause analysis of both costs: good and poor quality
 Recommendations for improving CoQ
 Identification of CoQ business cases
 Implementation of high quality standards
As illustrated in Figure 1, a good part of the CoQ exercise is contingent upon parameters used for determining
the quality of ‘good’ and ‘poor’ quality.
Figure 1: Cost of Quality Parameters
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 7
Cost of Good Quality: Appraisal activities, among others, measure, evaluate and audit products or services to
assure conformance to quality standards and performance requirements; their aim is to ensure that software
is free from defects and conforms to customer requirements. Prevention activities ensure that necessary
training, processes, tools, and hardware as well as resources are available for performing appraisal activities.
Since both the activities are pre-emptive measures, they are regarded as the cost of good quality.
Cost of Poor Quality: When detects (arising, for instance, from non-conformance to customer requirements)
are not caught during the development phase of software and are noticed just prior to delivery, they require a
great deal of re-work and re-testing effort. If software defects come to the surface after delivery, the cost of
review and root-cause analysis effort are significantly higher. Irrespective of when the failures come to light
(before or after delivery), they both reflect non-conformance to customer requirements, and are hence
referred to as cost of delivering poor quality software.
The value of being conscious about the cost of good and poor quality is fully realised only when, as shown in
Figure 2, the assessment data obtained from various stakeholders (at a client organization) is entered in HCL’s
CoQ analysis tool.
Figure 2: CoQ Modeling Results
Furthermore, as illustrated in Figure 3 and Figure 4, the HCL analysis tool allows for the quantification and
consolidation of quality parameters. Consequently, senior management at client organizations can then focus
on areas that require further improvement and develop a strategy for keeping the CoQ at acceptable levels.
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 8
Figure 3: Quality Parameters Reflecting As-Is Situation
Figure 4: Quality Parameters Reflecting a Reduction in CoQ
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 9
The HCL team uses an indexing system to make the results of the CoQ exercise meaningful to client
organizations. For instance, in the sample shown in Figure 5, the index number of 4, (which represents low
quality and high cost) was arrived at following a cost and quality assessment of testing practices at a client
location. The results allowed senior management to identify areas that required attention in order to reduce
cost and improve software quality.
Figure 5: CoQ Indexing
Based on its root cause findings, the HCL CoQ team identifies initiatives for closing the gaps between As-Is and
the desired objectives. As shown in the sample in Figure 6, the proposed initiatives are clubbed into solution
themes (group of initiatives), prioritized for ease of implementation, and then mapped to an IT strategy. The
HCL team also calculates the business benefits of implementing its proposed solution and suggests numerous
options for its implementation such as building the proposed solution in-house organically, partnering with a
third party to obtain the required skills, developing skills with third party via a co-sourcing/out-sourcing
strategy, or out-sourcing the entire solution to another company.
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 10
Figure 6: Theme Based Solution Proposition
As illustrated in the sample in Figure 7, the HCL team prioritizes the themes in its proposed solution and maps
them in a 2/2 matrix to show specific business benefits and ease of implementation. This exercise helps client
organizations to focus on recommendations that are not only easy to implement right away but also offer
business benefits (in terms of quality, cost, and productivity). Moreover, as and when required, client
organizations can extend the same recommendations to other projects in phases.
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 11
Figure 7: Business Benefits of Proposed Solution
CoQ as an Integral Part of SDLC
Although it is widely acknowledged that CoQ principles and practices should be used throughout the lifecycle
of software, most software teams find themselves walking a tightrope between the needs of business teams
(for faster time-to-market) and development teams (who require more time for comprehensive testing).
As shown in Table 1, in cases where development teams have the mandate to integrate CoQ principles and
practices throughout the lifecycle of software, HCL’s CoQ team can assist client organization in several
concrete ways to overcome challenges that are faced by software, testing teams, and allied service teams.
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 12
Testing
Phases
Planning
Phase-Wise Activities
Requirements analysis and
implementation feasibility
Identification of tools for test
management, defect management,
test automation and performance
testing
High Level
Test Design
Design analysis and feasibility study
Identification of:

impact to upstream and
downstream application and
aligning testing activities to
cover all areas

test environment and test
data set-up
Business Benefits
to Client
HCL Deliverables
Reduction in effort
and cost to fix defects
(discovered later)
Reports

Requirements analysis

Tool evaluation
Documents

Testing estimation

Test strategy

Microsoft project plan

Non functional
requirements

Test scenario
identification

Technologies and skills
set identification
Ensuring the
involvement of the
CoQ team right from
requirements phase
Reports

Detailed design
feasibility analysis

High level test
scenario and impact
analysis
Documents

Requirements
Traceability Matrix

Hardware and
software
requirements and Test
Data control sheet for
gathering test data
Understanding
software functionality
Avoid spend on
rework
Upload test cases in test
management tools
Having resources with
skills set in testing
techniques
Launch of new
features /application
within budget and on
time
Reports

Test summary

Daily status

Defect status
Having resources who
are adept at
addressing challenges
related to test
environment and test
data set-up
Savings realized on
tool cost and
rationalization of
tools usage across
multiple projects
Savings in effort and
cost related to defect
fix at early stage
System capacity planning and load
modeling
Detailed
Test Design
Detailed test case preparation and
review of test cases (by business
users/SMEs) to ensure complete
coverage of test cases against
functionality
Challenges
Ensuring the
involvement:

Domain experts

Performance
engineering
experts
Performance scenario identification
Test
Execution
Smoke testing to ensure basic
functionalities are working such
that the entire QA team can be
involved for testing all the features
System integration testing,
regression testing, user acceptance
testing, performance testing,
Identification of performance
bottlenecks and recommendations
fixes
Script execution in coordination
with other resources involved for
monitoring
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 13
Testing
Phases
Support
Phase-Wise Activities
Defect verification and impact
analysis
Update test cases related to defect
fix
Support for enhancement and
impact analysis
Calculation of Return on
Investment ( ROI) based on
application readiness, number of
release and nature of
enhancements
Business Benefits
to Client
HCL Deliverables
Increase in customer
satisfaction resulting
from better
performance of
software
Documents

ROI based automation

Production defect
trend analysis
Savings in effort and
cost due and increase
in revenue due to
readily available test
automation
framework and ease
of maintenance
Challenges
Collecting data on

customer
satisfaction data

revenue
generation

penalties paid
and support
charges
Having resources with
advanced knowledge
of test automation
Customization of existing HCL test
automation framework requiring
minimal maintenance
Table 1: CoQ and SDLC
Benefits
If software products suffer from a high number of defects, low quality, and require a great deal of manual
effort and rework, then it is next to impossible to achieve business objectives such as speed up time-to-market
and lower operational cost. What the CoQ approach from HCL allows client organizations to do is not only
identify problem areas (which result in poor quality of software) but also implement a holistic plan of action
for achieving business objectives and monitoring its progress in a measurable manner on a continuous basis.
Cost of Quality: Needs Your Attention Not Sometimes But Always
Page 14