Test How to Succeed in Test Automation

Test
How to Succeed in Test Automation
Björn Hagström & Davor Crnomat, Testway AB
Introduction
•
•
•
•
•
•
The need for Automation
Cost of Automation
Plan to Automate
Tools
Test Approach
Summary
The Need for Automation
•
•
•
•
Expanding regression suites
Rapid release-cycles
Test complexity
Repetitive tasks degrade and divert
attention
• ...
Cost of Automation
Cost
Manual Testing
Automated Testing
Test Cases run
Plan to Automate
• Treat an automation project as you
would treat any other development
• Start test development early
• Allocate resources
• Do not forget training
Test Automation Project
Requirements
There need to be clear and verifiable requirements of what the
automated tests should do
Analysis
How the tests are interacting with the AUT, how to obtain test data
etc.
Design
Error handling and logging, robust and scalable tests
Test
Verify that the tests indeed find and logs defects
Maintenance
Scalability and knowledge transfer
Identify areas with potential gain
Documentation
Create Test Data
Test Setup
Test Execution
Go for the juiciest targets first!
Tool Selection
• Do we really need automation? What problems
are we trying to solve?
• Is test automation’s solution cost effective?
• Write long list of potential tool candidates
http://www.sqa-test.com/toolpage.html
http://www.stickyminds.com/tools.asp
• Prepare list of required features and use it to
reduce the list
• Contact the tool vendors
• Evaluate tools (max 3-4 tools)
Tool Requirements
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Easy of use, easy of maintaining
Platform support
Data base support, File I/O
Tool customization
Test result logs and reports
Scripting language (arrays, list, structures, conditional logic, loops, create
and call functions…)
UI element identifiers (identify UI elements as objects)
GUI Map (name mapping of UI identifiers)
Error handling, Unexpected error recovery
The debugger
Command line script execution
Vendor (Maturity, financial stability, market share, support)
Licensing, pricing
The user community
Etc.
Contact Tool Vendors
•
Questions to ask tool vendors:
1.
2.
3.
4.
•
What does this tool do? What are benefits of using this tool?
How can this tool solve my problems?
Why is this tool better then other similar tools?
Etc.
Questions to ask tool users:
1.
2.
3.
4.
How does this tool perform? Are there any bottlenecks?
What is your impression of the vendor
Which other tools did you consider when purchasing this tool?
Etc.
An extensive list of the questions you can find here:
http://www.grove.co.uk/pdf_Files/STT_Questions.pdf
•
Ask vendors for demonstration
Tool Evaluation
• Read publications which have evaluated some of
testing tools from your list
• Summarize information that you got from
vendors and users
• Conduct an in-house competitive trial
It should be run by the evaluation team.
If you do it alone, don’t ”fall in love” and don’t let the license price
be your main issue.
• Make recommendation/ decision
• Write a short written report
Tool Implementation
• Set up team
• Start pilot project
Pilot project is short-term, non-critical and significant.
Implementation, maintenance and failure analysis is explored*
• Evaluate pilot project
• If pilot project is successful plan for phased
installation
* Software testing practitioner - Grove Consultants
Automation Test approach
Test automation is software development:
•
•
•
•
•
•
•
•
•
Code review
Code standards
Naming conventions
Error handling
Reusability
Documentation
Source control
Maintainability
Etc.
Automation Test
Automation test should be:
•
•
•
•
•
•
Independent
Maintainable
Reliable
Robust
Readable
Documented
Capture/Replay
Test automation tools vendors’ marketing:
“Tests can be created quickly and easily by non-programmers.”
”Test Automation is very easy. Just record the scripts at any
given point of time and replay them whenever you want to.”
Do you really believe it?
Good Automation Practices I
•
•
•
•
•
•
•
•
Plan automation
Educate developers on the automation requirements
Start small and focus on modularity and reusability
Build and document libraries
Use wrapper functions
Build initialization and clean-up routines
Apply error handling
Build error recovery
Good Automation Practices II
•
•
•
•
•
•
•
•
•
Use GUI maps
Make your code machine independent
Avoid Bitmap comparisons,
Use API
Use data-driven testing technique
Make test units short
Log your tests
Add comments to your script’s code
Document your tests
Keyword-driven testing
Advantages:
• Testers don’t need programming skills to build new test
scenarios – involve more users
• Modularity and reusability
• Test scenarios can be used as test documentation
Disadvantages:
• Time to set up initial framework
• Programmers dedicated to implementing and maintaining
the framework are still needed.
• It could be hard to control test cases (spread sheets)
• Time consuming to learn all keywords
Summary
• Do not try to automate everything
• Treat test automation as a software
development
• Tool selection is a small project
• Consider alternative solutions
?
Remember!
Enter the evaluation form and be a part of making Øredev even better.
You will automatically be part of the evening lottery