SE420 Software Quality Assurance

SE420
Software Quality Assurance
http://dilbert.com/strips/comic/2009-06-22/
Lecture 12 – Industry Systems and
Analysis by Type
October 28, 2014
 Sam Siewert
Reminders
Assignment #5 Due 11/9, Late Date is 11/12
Remaining Assignments [Top Down / Bottom-Up]
– #6 – Complete Code, Refine and Run all V&V Tests and Deliver
Track Bugs with Bugzilla - http://prclab.pr.erau.edu/
Update your Project Code on GitHub https://github.com/
Assignment #6 Posted this Wed, Due 12/1, Late 12/4
 Sam Siewert
2
Regression Testing and Test
Automation
Regression Testing
(Test Automation)
Re-test Units After Defects are Fixed
Re-run I&T and System Tests
Automation from Checkout, Build, Test Sets, Verification & Validation
of Expected Results
 Sam Siewert
3
Discussion and Q&A
Industry Specific Guidelines, Standards, and Audit
Mission Critical Systems (Life, Financial and Property Risk)
– Enterprise Data Systems – Financial, Medical Records, Government, HR, etc.
Storage, Networking, Web Access, DBMS
Security and Data Integrity is Critical, Business Continuity
– Aerospace – Civil Aviation, Defense Systems, NASA, Commercial Space
– Medical Systems – Diagnostics, Therapeutics, Monitoring, Laboratory
Automation
– Energy – Refineries, Nuclear Power Generation, Distribution, Exploration and
Mining
– Security – National, Corporate, Personal
– Transportation – Public, Private
Cyberphysical Systems - http://cyberphysicalsystems.org/
NSF Conference – 2006, Austin Texas, http://varma.ece.cmu.edu/cps/,
focus on position papers (similar to 1968 NATO Software Crisis
Conference)
Original Paper(s) Introducing Concept http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-8.pdf
 Sam Siewert
4
Cyberphysical Systems
 Sam Siewert
5
Emergent Cyberphysical Application –
ITS [Intelligent Transportation System]
Intelligent Transportation - http://sites.ieee.org/itss/
Intel in Intelligent Transportation http://www.intel.com/content/www/us/en/automotive/exp
eriencing-future-intelligent-transportation-video.html
Is this a Good Idea? - It’s a Complex System of
Systems with Major Security Challenges
Opens new Markets and Can Optimize Traffic, both
Personal and Commercial [E.g. Truck Fleeting with SelfDriving Trucks, I-5 California]
 Sam Siewert
6
Emergent Cyberphysical Application –
Commercial Space Transportation
NASA Has Outsourced Space Station Servicing and
Access in General
– http://www.spacex.com/
– http://www.sncorp.com/
– OSC - http://www.orbital.com/ [Antares 2014 Accident]
Space Tourism and Commercialization
– http://www.virgingalactic.com/statement-from-virgin-galactic/
[SpaceShip Two Accident in 2014]
– XCOR Aerospace - http://www.xcor.com/
– Boeing CST-100 - http://en.wikipedia.org/wiki/CST-100
A Complex System of Systems with Major Safety
Challenges
 Sam Siewert
7
A Decade of Space Tourism
Most Recent
Early Space
Tourists
 Sam Siewert
http://en.wikipedia.org/wiki/Space_tourism
8
Traditional Mission Critical Systems
Standards and Guidelines for Systems and Software Testing
Enterprise Data Systems – Commercial Standards and IV&V Labs
(E.g. Medusa Labs Test Tools), Standards for Protocol (SCSI T10
and SAS, Fiber Channel T11, FCoE, iSCSI) and Performance Test
Standards (TPC, SPC), SNIA RAID
Aerospace – NASA Standards
(https://standards.nasa.gov/documents/nasa ), Military Standards
(MIL-STD-498), FAA Launch and Reentry, FAA
Medical Systems – IEC 62304, E.g. Commercial Support
Energy – NIST Testing, DoE STD-1172
Security – TCG, NIST ITL, NIST FISMA, NIST FIPS, NIST TIRs –
Overview
Transportation – FAA DO-178B, Software Assurance, V&V, NHTSA
Policies on Automated Vehicles
 Sam Siewert
9
Infamous System Software Defects
Rear Admiral Grace Hopper
Infamous & Famous Field Defects
1. Toyota ABS Software Recall, 2010
2. Windows Genuine Advantage Outage, 2006
3. Mars Climate Orbiter Units Interoperability,
Loss, 1999
4. Mars Pathfinder Priority Inversion,
Recovered, 1997
http://en.wikipedia.org/wiki/Software_bug
5. Ariane-5 501 Cluster Launch Software
Failure, 1996
6. Pentium FPU Bug – Software Acceleration
RCA, Patch, Test, Fix!
Co-Processor Recall, 1994
7. ATT 4ESS Upgrade, Outage, 1990
8. Therac-25 Radiation Therapy Patient
Overdosing, 1985-86
9. NORAD False Alarms, User Error for Test
Mode, Device Failures, 1979/1980
10. Apollo 11 Guidance Computer Overload,
1969 - Recovered
http://history.nasa.gov/SP-350/ch-11-4.html
 Sam Siewert
10
Hands-On Coding Exercise(s)
Examples-RAID-Unit-Test, stripetest.c
[siewerts@localhost Examples-RAID-Unit-Test]$ ./stripetest Baby-Musk-Ox.ppm Baby-Musk-Ox.ppm.replicated
read full stripe
…
hit end of file
FINISHED
[siewerts@localhost Examples-RAID-Unit-Test]$
[siewerts@localhost Examples-RAID-Unit-Test]$ diff Baby-Musk-Ox.ppm Baby-Musk-Ox.ppm.replicated
A,B,C,D Strips
A
 Sam Siewert
XOR[A,B,C,D]
B
C
D
XOR
11
Discussion
Are Standards and Guidelines Sufficient?
What Can be Done to Improve?
Could there be One Standard?
NTSB Aviation Accidents (RC - SW?)
http://mercury.pr.erau.edu/~siewerts/se420/documents/Li
nux/Linux-Programming-Top-Errors.pdf
 Sam Siewert
12