Mobile Testing

Transcription

Mobile Testing

T17
Class 10/6/2011 1:30 PM "Mobile Testing:
Old Wine in a New Bottle?"
Presented by:
Manish Mathuria
InfoStretch Corporation
Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Manish Mathuria
InfoStretch Corporation
Manish Mathuria has more than fifteen years of professional and management
consulting services experience. As the CTO and leader of the mobile testing practice at
InfoStretch Corporation, Manish leads the team in developing architecture, scalability,
performance, and IT strategies for customers, especially in the mobile domain. He
specializes in leading enterprises into the mobile domain. InfoStretch provides
certification and testing services to some of the leading national and international app
stores. Manish and his team have tested and certified more than 8,000 mobile
applications in InfoStretch Mobile Lab. He speaks regularly at leading quality and
software engineering conferences worldwide.
Mobile Testing
Old Wine in a New Bottle?
Agenda:
STARWEST, 2011
phone: (408) 727-1100
|
website: www.infostretch.com
All trademarks are the property of their respective owners.
©2004-2010
Corporation.
All rights
reserved.
©2004-2011InfoStretch
InfoStretch
Corporation.
All rights
reserved.
|
email: [email protected]
Global Mobile
Landscape
Review
phone: (408) 727-1100
|
©2004-2010 InfoStretch Corporation. All rights reserved.
|
•Global Mobile Landscape
•Need for Mobile Testing
•How is mobile testing different
•Challenges of Mobile Testing
•Mobile Testing Models
•Transforming into a mobile tester
1000+ Mobile Network Operators
operating world wide
‐
100+ Mobile Application Stores 100+
Mobile Application Stores
already launched
More than 15 Mobile Platforms
100+ Mobile Handset Manufacturers
phone: (408) 727-1100
|
|
More than 1 Million Apps
20 Billion Applications Downloaded
phone: (408) 727-1100
|
|
4
Smartphones in Enterprises
1. 90% of IT Managers surveyed are planning to implement new
mobile applications
2. Nearly one in two IT Managers believe that successfully managing
Mobile Applications will top their priority list.
3. 21% of CXOs in US are looking
g to introduce 20+ Mobile
Applications into their organizations
Sources: Ask.com, Sybase, Gartner
phone:
(p)
(408)(408)
727-1100
727-1100|
|(w) ww.infostretch.com
|
(e) [email protected]
| email: [email protected]
|
Presenter: Rutesh Shah
©2004-2011
What Do All These Numbers Mean??
phone: (408) 727-1100
|
|
All these apps need to be Tested.
phone: (408) 727-1100
|
|
It’s a Herculean Task
phone: (408) 727-1100
|
|
Mobile Testing
Is it an old wine in a new bottle or is it
diff
different?
?
phone: (408) 727-1100
|
|
Mobile Testing is Different
phone: (408) 727-1100
|
|
What’s Changed (from traditional software testing)
Mobile is fragmented
Mobile is dynamic
Mobile requires logistics
phone: (408) 727-1100
|
•Many platforms (Blackberry, Android, iOS, S60, WP7, etc…)
•Many form factors (touch, keyboard, slide, landscape, etc…)
•Many networks (3G, 4G, Wi‐Fi, Wi‐Max, etc…)
•This is much more than the “firefox vs. explorer” issue
•More platform changes (8 major version changes in 2010)
•Handset “life” is short (6‐18 months)
•How to identify what is available on each device?
•How to test interoperability with other device applications?
•How to test location based services?
•What are the standards of usability?
•How to test in different locale on different carrier network?
•Many handsets
•Actual networks
•Accounts, SIM, Access ‐ expensive, time consuming and resource intensive •usability, network connectivity, online/offline content, call interruptions, varying form factors, networks, and device providers
|
11
New Needs and Challenges
Testing g
Challenges
Software Development Cycle
Enterprise Testing Challenges
phone: (408) 727-1100
|
12
•How to identify which devices to test on to support your user base?
•How to identify device limitations and characteristics for feature testing?
•How to report observations during testing on devices to developers?
•How many versions of Mobile OS tests need to be repeated?
•How to find out unique features of each device?
•How to test on Pre‐Market
devices?
•How to test on Pre
Market devices?
•Connectivity to native devices – peripherals
•Mobile SDLC is in its nascent stages
•No set quality and development process
•No Standards
•Agile development, nightly builds
•Location based, orientation, social media integration
•When to do full test vs partial test?
•Different methods to deploy ‐ making testing more complex
•Ad‐hoc testing – beyond what is documented
•Difficult to test in the field
•What to expose to the outside world •No VPN on old devices
•How does the app deal with the backend system changes?
•How to push updates and test them?
•No security standards
•Minimum vulnerability database for apps as well as mobile browsers
•No app development standards
•No standard privacy policies and laws set
|
What has NOT changed
The basic ALM is the same
• Agile development, nightly builds
• Test management and defect tracking platforms
• Source control
• Value and true integration to your existing ALM infrastructure is important to increase testing efficiency
Manual testing is still here (and kicking)
• Exploratory testing is still manual
• Manual testing infrastructure is important due to logistics (you’ll need screenshots…)
Automation requires efforts
phone: (408) 727-1100
|
• Automation value is built over time
• You’ll need an automation expert to create value
|
13
Mobile Testing
Models
A Comparison
phone: (408) 727-1100
|
|
Mobile Testing Models
• Testing
g on Simulators/Emulators
• Testing on Real Handsets
• Testing in the Cloud using Remote Handset
Access
• Testing using Crowd Sourcing
• Testing using Managed Testing Services
phone: (408) 727-1100
7/19/2011
|
|
15
Testing on Mobile Simulators/Emulators
Simulators: Software that duplicates the mobile handset as closely as possible
Emulators: Hardware that is designed to aid testing effort but not the real mobile handset
PROS

Useful for testing during development (speeds development effort)

Useful to verify usability and debugging

Lower cost of testing CONS

Can’t do OTA Testing

Can’t do peripheral connectivity testing

Not suitable to validate carrier/Appstore based based
restrictions (Locked devices) Examples: Activated on real carrier network
(p) (408)(408)
phone:
727-1100
727-1100|
7/19/2011
|
|
©2004-2010
16
Testing on Real Handsets
Testing of applications on the real handsets as used by the end user
PROS
CONS
A Complete platform for
Very costly due to investment in mobile handsets and carrier plans

Time consuming as most of the testing effort is manual

Testing by global teams is not feasible
validating user experience
Well suited for peripheral connectivity and LBS tests
Examples: Activated on real carrier network
phone:
(p)
(408)(408)
727-1100
727-1100|
|
|
©2004-2011
Testing in the Cloud using Remote/Virtual Testing of application using remote handset access technology
PROS

Most cost‐effective testing solution for real user experience

Ability to capture/audit‐trail testing effort (images, video)

Automation of selected tests can be leveraged for monitoring
CONS

Not reliable for highly interactive app/game testing
Not suitable for peripheral device connectivity testing
Examples: Device Anywhere, Perfecto Mobile
(p) (408)(408)
phone:
727-1100
727-1100|
7/19/2011
|
|
©2004-2010
18
Testing using Crowd Sourcing
Leveraging a global community of professional testers with handset accessibility to perform selected tests
PROS

Cost‐effective testing solution

Suitable for global footprint testing across geography, carriers and handsets

Faster time to market
CONS

Not suitable for mission critical applications (security, information privacy…)

Reliability is limited as audit trail may not be adequate

Crowd sourcing platforms are evolving ‐ difficult to manage the testing efforts
Examples: uTest, mob4Hire
(p) (408)(408)
phone:
727-1100
727-1100|
7/19/2011
|
|
©2004-2010
19
Testing using Managed
Work with testing experts with handset lab access to optimize testing outcome
PROS

CONS
Higher Quality ‐ A Complete testing solution for most effectiveness

Leverage knowledgebase and expertise across testing models/tools/technologies

Readiness to deal with new technology/handset challenges

May not be most cost effective

More suitable for Mission/Business Critical Enterprise applications
Examples: InfoStretch, AppLabs, Lionbridge
(p) (408)(408)
phone:
727-1100
727-1100|
7/19/2011
|
|
©2004-2010
20
Training your software QA team for testing mobile
apps
Transferable skills
• Fundamental testing knowledge
New skills
• Usability testing
• Knowledge of mobile- Basic mobile
functionality
• Basic device capability etc. Carrier related set up
• Mobile architecture
• Enterprise mobile components like bs , cms etc
• Other traits of mobile testers
• Destructive testing – Think like end users
phone: (408) 727-1100
|
|
Landscape of Mobile Automation Tools and
Technologies
• Automation
technology on devices
hosted in the cloud
• Available in private
cloud mode or public
cloud mode
Cloud Based
phone: (408) 727-1100
|
• Primarily works with
image recognition
• Works across device
platforms
• Works on traditional
object recognition
paradigm
• Scripts are richer, more
capable though have
to be scripted per
device platform.
GUI – Platform
Independent
GUI – Platform
Dependent
|
Test Case Automation Architecture
Browser
Open
Browser
Go To
Make
Payment
Sign On
Test Script
Text Verify
Confirm
Payment
Sign Off
Make Payment Test case
Address
ATM
Search
Browser
Open
Select
Acct
View
Schedule
Payment
Sync Text
Quick
Links
Nick
Name
Browser
Go To
Sign On
Sign Off
ZIP ATM
Search
Make
Payment
Sync
Image
Confirm
Payment
Framework
Function Library
Android
HTC Aria
iOS
Devices
Android
HTC Droid
Testcases
|
|
Old Wine or New?
Mobile Testing is different!
New mindset
New tools
Many new things to worry about
Mobile Testing is not different !
Same fundamentals of Quality Assurance apply Learning new skills
Mobilize yourself
Use new models
Use new tools
phone: (408) 727-1100
BB 8520
LG CU620
LG
VX9000
Device Implementations for Sign On (10+)
Device Implementations
phone: (408) 727-1100
BB Torch
|
|
Functions