which test cases? - AG Software Engineering

Transcription

Software Engineering and
Scientific Computing
Barbara Paech, Hanna Valtokari
Institute of Computer Science
Im Neuenheimer Feld 326
69120 Heidelberg, Germany
http://se.ifi.uni-heidelberg.de
[email protected]
RUPRECHT-KARLS-UNIVERSITÄT HEIDELBERG
Schedule Second Day
9:00 Testing
Project Management
10:00 Break
10:30 eXtreme Hour
12:00 Lunch
13:00 Tools, Exercises
Incl. a Unit test
short Code Documentation
break
16.00 End
Vorlesung – SE and SC – SS 2010/11
© 2010 Institut für Informatik, Ruprecht-Karls-Universität Heidelberg
2
Quality Assurance, Testing
3
Background of scientific software
=
Reality
Conceptual model
Mathematical model
?
sun()
Simulation
Computer Program
∂F
∂x
Numerical model
4
Quality assurance for scientific software
 1. Code verification
Scientific
• Revies
• (Unit) Testing
Algorithm
 2. Algorithm verification
• Is the implementation of the
mathematical model correct?
• i.e. Grid convergence
testing
 3. Scientific Validation
• Verify if the result is accurate
• If possible: compare results
with
experiment results
Code
Verification
Verifikation
Validation
5
Analytic quality assurance
 Goal: check whether the product satisfies the requirements
 Methods
•
•
•
•
Proof (complex theorem provers needed, only specific domains)
Test (probe product with specific inputs)
Review (systematic reading)
Metrics (automated determination of characteristics)
6
[http://www.sei.cmu.edu/tsp/index.cfm]
Personal Software Process (PSP)
 The PSP is a personal process for developing software or
for doing any other defined activity. The PSP includes
• defined steps
• forms
• standards
 It provides a measurement and analysis framework for
characterizing and managing your personal work.
 It is also a defined procedure that helps you to improve
your personal performance.
7
What is Quality?
 Basic definition of quality: meeting the users’ needs
• needs, not wants
• true functional needs are often unknowable
 There is a hierarchy of needs.
•
•
•
•
•
Do the required tasks.
Meet performance requirements.
Be usable and convenient.
Be economical and timely.
Be dependable and reliable.
8
The PSP Quality Focus -1
 To be useful, software must
•
•
•
•
•
install quickly and easily
run consistently
properly handle normal and abnormal cases
not do destructive or unexpected things
be essentially bug-free
 Defects are not important to users, as long as they do not
•
•
•
•
affect operations
cause inconvenience
cost time or money
cause loss of confidence in the program’s results
9
The Economics of Quality
 With common quality practices, software groups typically
•
•
•
•
spend 50+% of the schedule in test
devote more than half their resources to fixing defects
cannot predict when they will finish
deliver poor-quality and over-cost products
 To manage cost and schedule, you must manage quality.
10
Testing Alone is Ineffective
Overload
Hardware
failure
Configuration
Safe and secure
region = tested
(shaded)
Unsafe and insecure
region = untested
(unshaded)
Operator
error
Resource
contention
Data error
11
Defect-removal Methods -1
 The principal ways to find and fix defects are by
•
•
•
•
•
compiling
unit testing
integration and system testing
team inspections
personal reviews
 Since you will likely have to remove lots of defects, you
should use the most efficient methods.
12
Defect-removal Times
10000
Time in Minutes
1405
1000
100
25
22
10
32
5
2
1
Design Design Code
Code
Review Inspect. Review Inspect.
Source: Xerox
Unit
Test
System
Test
Defect-removal Phase
13
Defect-removal Methods -2
Best Of
 In a personal review
for Scientific
Software
• you privately review your product
• your objective is to find and fix defects before test
 Reviews are most effective when they are structured and
measured.
 Use reviews for requirements, designs, code, and
everything else that you develop.
 Also continue to use inspections, compiling, and testing.
14
Defect-removal Rates -1
 Even at the personal level, it is more efficient to find
defects in reviews than in testing.
•
•
•
•
Unit test finds only about 2 to 4 defects per hour.
Unit test finds about 50% of the defects.
Code reviews find about 6 to 10 defects per hour.
Practiced reviewers can find 70% or more of the defects before
compiling or testing.
15
Why Reviews are Efficient
 In testing, you must
•
•
•
•
detect unusual behavior
figure out what the test program was doing
find where the problem is in the program
figure out which defect could cause such behavior
 This can take a lot of time.
 With reviews and inspections, you
•
•
•
•
•
follow your own logic
know where you are when you find a defect
know what the program should do, but did not
know why this is a defect
are in a better position to devise a correct fix
16
Review Principles
 PSP reviews follow a defined process with guidelines,
checklists, and standards.
 The PSP review goal is to find every defect before the first
compile or test.
 To address this goal, you should
•
•
•
•
•
review before compiling or testing
use coding standards
use design completeness criteria
measure and improve your review process
use a customized personal checklist
17
Test process
Test Manager
Test planning and control
Test specification
Test Designer
Test Automation
Bug Tracking
Test scripting
Testware Management
Test execution
Tester
Test protocol
Test evaluation
18
Test specification
 Define
• Test object (e.g. system, component)
• Test cases (Black box or white box)
• Test end criteria (e.g. how many % of the test cases must be
successfully performed, at which defect density do you stop the
testing)
19
Test case definition
 Test case input
• logical (value ranges)
• specific (specific values)
 How to define the expected behaviour: find a test oracle
•
•
•
•
Requirements
User handbook
Executable prototype
Old versions
Big problem for
scientific software engineering!!
20
Test methods
 Black-Box:
•
•
•
•
•
Only use knowledge about the interface
Test the visible behaviour
No control of the test execution
Can not identify unnecessary code
Example: equivalence classes, state based
 White-Box:
•
•
•
•
Use knowledge about the internal structure of the code
Control the test execution
Can not identify missing requirements
Example: statement coverage, branch coverage, condition
coverage
21
Test case input
 Positiv Test:
• Typical correct input
 Negativ Test:
• Incorrect input (tests the exception handling)
 Intuitive / experienced based Test:
• Uses knowledge about typical defects
• Should always be performed in addition to other test methods
22
Example: which test cases?
 Function:
int search (int[] a, int k)
pre: a.length > 0
post:
(result >= 0 and a[result] ==k) or
(result == -1 and
(not exists i. i>=0 and i< a.length and a[i] == k)
)
23
Equivalence class
 Equivalence class: subset of all inputs which invoke similar
program bevahiour
 Test one representative per equivalence class
 Typical equivalence classes
• Correct / incorrect inputs
• Boundary values
 Gets more complicated with several input parameters
24
Test driven development
 Main idea:
Best Of
for Scientific
Software
• write test cases first
• continuous testing
25
Testing principles (1)
 Principle 1
Complete testing not possible
 Principle 2
»Program testing can be used to show the presence of
bugs, but never to show their absence!«? Edsger Dijkstra
 Principle 3
Start early with testing (see defect removal times)
 Principle 4
Defects are not evenly distributed in the code. If you have
found many defects at one place, look for more.
26
Testing principles (2)
 Principle 5
Test cases must be managed (evaluated, updated)
 Principle 6
Test effort has to be adapted to the context (more for
critical systems)
 Principle 7
A defect-free system does not guarantee customer
satisfaction
27

which test cases? - AG Software Engineering

Transcription

Similar documents

Version management - AG Software Engineering