How to comply with ISO26262 efficiently By Steve Barriault –

How to comply with ISO26262 efficiently
– the case for highly automated testing
By Steve Barriault
Technical Sales & Marketing Manager –
Asia
© Vector Software Inc, all rights reserved.
Company introduction
• Founded in 1990 in Rhode Island by embedded engineers
• First release of VectorCAST in 1994
• VectorCAST provides unit and integration testing, as well as system test
coverage tools that are uniquely automated
• These tools can be run on a host, a simulator or directly on a target
• Have worldwide offices and representation
Netherlands
Sweden
Rhode Island
Japan
Georgia
Korea
Arizona
China
London
France
Italy
India
Israel
Software testing under ISO 26262
• ISO 26262 is IEC 61508 adapted to the needs of
the Automotive Industry
• Adopts a similar approach to software testing and
code coverage requirements to other, longer-lived
standards (such as DO-178B)
• The challenge: meet its requirements in an
industry where deadlines are coming up much
faster than in Aerospace
– In order to achieve success, efficiency will be key!
Our experience with standards
• Our roots are in the Aerospace industry, where our
15+ year of experience enabled us to work with all
the leading organizations
• Our Automotive business is expanding quickly, with
a growing list of companies using VectorCAST
• Requirementsbased testing
• Interface
testing
• Fault injection
test
• Resource
usage test(*)
(*) Source: Table 12 and 15:
Methods for software unit/integration testing
Metric Activities
• Unit test
• Integration
test
• System test
By Goal
By Scope
Tests in ISO 26262
• Structural
code
coverage
• Testrequirement
association
Requirement-base testing
• Ensures that the software fulfill its mission
– Sometimes called functional test
• Strongly recommended for all levels of ASIL, for both unit
and integration testing
• Stubbing can be performed to enhance your ability to test
low-level requirements in isolation
– They “replace” your existing code so you can better control your
inputs and outputs in the code
– But stubs can take a long time to be generated with scripting-based
tools
– With VectorCAST, the stubs are automatically generated in seconds,
with no user input whatsoever
Requirement-base testing
• The link between the requirement and the test case should
ideally be documented
– In VectorCAST, it can be. Our unit and integration tests can be linked
to specific requirements.
– The test case data (PASS|FAIL) that demonstrate requirements can
be uploaded to a requirement management system such as DOORS
DOORS®
requirements
VectorCAST
Test cases
Execution on
host,
simulator
or target
Req. 1
Test 1
PASS/FAIL
Req. 2
Test 2
PASS/FAIL!
Req. 3
Test 3
PASS/FAIL
External interface test
• External Interface Testing is a subset of functional testing.
• Highly recommended for all ASIL level, both unit and
integration
• It verifies that:
– Functions sent data out in the appropriate format and delivery
mechanism
– Functions that receive data in the appropriate format perform
correctly
– That the behavior when receiving data that is not formatted directly is
known
• Can also be tested by VectorCAST
Other types of test
• Fault injection test:
– Voluntarily inject arbitrary faults to test safety mechanisms
(ex: by corrupting values of variables)
– Recommended for unit/integration testing,
strongly recommended ASIL D (and C in integration)
– In VectorCAST, can provide test cases that have faulty values and
verify that the defensive code gets invoked
• Resource usage test:
–
–
–
–
Often only doable on target or at least simulator
Recommended for unit/integration testing
Strongly recommended ASIL D
Our superior degree of target integration can also help you do some
of this, but perhaps not all
Generating test case values
• Based on requirements
– Strongly recommended for all ASIL, unit/integration test
• Equivalence classes:
– This method may be used to partition possible input values of
external interfaces
– Strongly recommended ASIL B, C and D
– VectorCAST has a facility to generate automatically such partitioned
test cases
• Error guessing:
– Here, the tester tries to test errors that are suspected to be error
prone
– Only recommended all ASIL
– Also easily possible in VectorCAST
Generating test case values
• Analysis of boundary values
– Try values approaching, at, or crossing the boundaries, including out
of range value
– Can mean the type range or the functional range
– Strongly recommended for ASIL B, C and D
• VectorCAST has extensive tools to do this
– Auto-generation of MIN-MID-MAX test cases for all the extreme
variable type values
– Import from CSV functional range values – and execution of these in
test cases
Code coverage
• At decision
point, both
TRUE and
FALSE to be
executed
MCDC
• One test case
minimum to
execute one
line of code
Branch
Statement
• Lets you know when you have been “testing enough”
• Different criteria that require more or less test cases to
achieve
• VectorCAST supports all three criteria recommended by
ISO 26262 (and the “other criteria” – function/call coverage)
• All operands
must
independently
affect the
outcome of the
condition
How coverage criteria stack up
• Statement
if((a || b) && c)
– Line of code executed at least once
1 test case required
• Branch
if((a || b) && c) T
2 tests cases required
– Both the TRUE and FALSE
branches are executed
• MC/DC
F
if((a || b) && c) RESULT
– All operands can independently
affect the outcome
a
b
At least n+1 test case
c
required
© Vector Software, all rights reserved
13
F
F
T
F
T
F
T
T
F
T
T
T
F
T
F
F
Knowing what needs
to be done!
Green: Fully covered
(good)
Red: Not covered
(bad)
Orange: Partially
covered (?)
+/- critical software
Statement
+
Branch
+
MCDC
very critical software
What you get - Automaticity
• Unit test environments are generated automatically:
– All drivers and stubs generated with NO user input
• Constructing a test case is done through point and click OR CSV
– NO scripting of ANY kind
• Both Black Box and White Box are allowed
• Execution on target is done 100% automatically
– You click a button, and the rest is done for you
• Execution reports are generated 100% automatically
• Both GUI and command line are available
• Re-running any test environment in regression mode is automatic, even
if the test cases are modified, or if the underlying code is modified
– Regression testing is completely automated
What you get - Flexibility
• Full, guaranteed support for C/C++ of ANY complexity
• Users can control the value of ALL parameters, return values, global
data and data coming from stubs – even for pointers, exceptions, etc.
• Can test individual values, special values (NAN, positive infinity, etc),
range of values, list of values, even call code to generate Monte Carlostyle of test cases
• Can create complex test cases that set state machines and test their
transition from one state to another (compound test cases)
• Creation of test cases from CSV
• Automatic test case generation based on basis path analysis, MIN-MIDMAX, and more so as to give a leg up during structural coverage
• Code coverage is displayed in an easy-to-understand way
Other capabilities you get…
• The ability to test libraries – even if you don’t have access to the code!
• The capability to test as you develop (agile development), or even to first
write test cases before writing code (test-driven development)
• The debugger can be used to control test case execution (so the tool
becomes a test vector generator for debugging too)
• The ability to perform timing calculations, in some conditions can also
be done while other processes from the OS are executing
• The ability to stub library functions, if desired…
• … and much more
Quite simply, you are getting a complete test bench that
enables you to comply with ISO 26262 efficiently, so you
can still meet demanding deadlines!
What about tool qualification?
• ISO 26262 Part 8, Section 11 mentions tool qualification
– The objective of the qualification of software tools is to provide evidence of
software tool suitability for use when developing a safety-related item or
element, such that confidence can be achieved in the correct execution of
activities and tasks required by ISO 26262
• Depends on how critical the tool’s reliability is to the quality
of the code
• Process more flexible than DO-178B, but if you need tool
qualification, Vector Software has a long experience of
providing this service
– We can provide you documents demonstrating the tool performance
as adequate in your environment: same compiler version, board,
debugger, and tool version
Conclusion
• ISO 26262 contains a number of recommendations that
have been proven effective in other industries
– It does not reinvent the wheel but builds on a rich heritage to customize an
unique standard to the needs of the Automotive industry
• Unit/integration testing and code coverage can be very timeintensive, which no one in this industry can afford!
• The only way to meet ISO 26262, increase code quality and
still meet your deadlines is to invest in test automation
– Fortunately, VectorCAST tools have both the automation and
flexibility required for you to achieve ISO 26262 compliance in a
timely manner
Questions?