Mobile Game Testing eBook | SmartBear

Contents
Introduction.............................................................................................4
User Expectations..................................................................................5
Methods of Testing for Video Games.....................................................7
Black Box Testing...................................................................................8
White Box Testing...................................................................................8
Automated Testing’s Role in Video Game Development........................ 9
Video Game Testing on a Mobile Platform........................................... 11
Mobile.............................................................................................................. 11
Data.......................................................................................................... 11
Network Availability................................................................................... 12
Gestures................................................................................................... 12
GUI............................................................................................................ 12
Portrait and Landscape Views.................................................................. 12
Processing Power..................................................................................... 12
Memory..................................................................................................... 13
Battery....................................................................................................... 13
Fragmentation of devices.......................................................................... 13
Gaming............................................................................................................ 14
Audio......................................................................................................... 14
Graphics.................................................................................................... 14
Accessibility.............................................................................................. 14
Databases................................................................................................. 14
Conclusion............................................................................................15
TestComplete
by SMARTBEAR
2
Introduction
We’ve all heard the stories of video game testers—recent
high school graduates sitting in a room, playing video
games for twelve hours straight, drinking Mountain Dew
and documenting any bugs they can find. Many kids think
of this as a dream job. But what they don’t understand is
that it’s probably one of the most tedious, mind-numbing
jobs one might fall into. As a software tester, it’s not particularly the scene you want to think of when you imagine
your day-to-day work life, yet this is the culture of testing
video games. There isn’t any test automation tool that can
help minimize the tedium. It’s all manual user acceptance
testing and, the truth is, video game testing probably ruins
video games for most of the people who attempt this vocation.
This eBook is about the exploding industry of mobile gaming and the reality of testing video games for mobile devices. If the nuances of mobile app testing and video game
testing weren’t bad enough, combine the two and you have
a perfect storm for any software tester. Even the most
seasoned software tester might be hesitant to approach
mobile gaming.
If you go into the Apple store or the Google Play store, every day there is a new list of games that have come to the
market. A recent study by eMarketer showed that by 2015,
approximately 162 million people will be playing games
on mobile devices. To put that into perspective, it amounts
to about half of the entire US population playing mobile
games in some fashion. So, despite the fact that mobile
gaming can be a testing nightmare, the benefits of producing a hit game could outweigh the risks. The eMarketer
TestComplete
by SMARTBEAR
3
study also claims that user growth could push revenues for
the mobile gaming market to $1.78 billion in 2013. Clearly,
there is money to be made!
User Expectations
So you’ve decided that developing a mobile game is a
great way to make money. Before jumping right into game
development, you must understand who your end users are and what genre and platform caters to the users’
gaming needs. As with all software
development, the user will be your
best friend and worst enemy. By
listening to your user’s feedback on
design and mechanics you increase
the chances of being successful.
However, by ignoring the needs of
your users, you risk alienating them
and destroying your credibility as a
video game developer.
Here is an example:
The main point of a Diablo game—what users came to
love the Diablo games for—was the constant slaughtering
of monsters and demons with the sole purpose of leveling up, customizing your character, and finding cool loot to
which you could sell off or trade with other players.
Yes, Diablo III had cool loot, the graphics were better,
performance was fine, but there was something very wrong
with this iteration of Diablo. With Diablo III, Blizzard introduced the auction house as a way to make real life and
in-game money off of the rare loot a player could find, with
TestComplete
by SMARTBEAR
4
a small fraction of all real-money transactions going into
Blizzard’s pockets.
The issue here was that the gear a character needed to be
strong enough to progress further in the game became a
bottleneck when you had no choice but to use the auction
house to get the gear your character needed to progress.
Essentially, Blizzard’s attempt to add a feature that was
supposed to be optional became the sole basis of the
game. Users found themselves spending more time looking for gear on the auction house instead of going out and
fighting monsters.
We are writing tell you about an important change to
Diablo III: we’re going to be removing the gold and realmoney auction house system from the game.
When we initially designed and implemented the auction
house system, the driving goal was to provide a convenient
and secure system for trades. But after much review and
player feedback, it became increasingly clear that despite
the benefits of the AH system and the fact that many players around the world use it, it ultimately undermines Diablo’s core game play: kill monsters to get cool loot.
We’re working out the details of how the auction house
system will be shut down, but we wanted to share the news
as soon as we made the decision in order to give everyone
as much advance notice as possible. Please note that the
final shutdown will occur on March 18, 2014. We will keep
everyone informed as we work through this process, but
feel free to check out our blog post on the subject, and stay
tuned to Diablo 3.com for further details.
The Diablo III team
TestComplete
by SMARTBEAR
5
The point is that you must be in touch with whom you are
creating the game for and the game must be tested not
only for bugs but for fundamental issues, such as unwanted features and/or requirements that are irrelevant to
stakeholders. Sometimes, simplicity is the best approach.
In the case of mobile games, you can always add features
later via patching once you have a better idea what your
users really want. However, this should not be an excuse
for releasing buggy software and patching later since this
will eliminate your chances at being reputable. Those bad
reviews don’t go away after you release version 2.
Methods of Testing for Video Games
You’ve built game requirements based on feedback from
all stakeholders. The development team is about to start
developing the game and the testing team is about to start
testing the developers’ work. Welcome to the world of
video game testing! Your users are going to pick apart your
game and tell you everything that is wrong with it, so test
for anything that would ruin the user experience.
Every gaming company tests their game differently, because there are so many different genres of games and
since software testing methods are always dependent on
the context of the project. Despite the differences in testing
methodologies from team to team there are two main categories of testing that always seem to be used for testing
games.
¿¿ Black box testing – Testing functionality without
checking the code
¿¿ White box testing - Testing and examining code as
opposed to functionality
TestComplete
by SMARTBEAR
6
It is important to understand that these are generic test
“strategies” and not “types” of testing. For instance, functional testing would be a specific type of testing that falls
under the black box testing strategy. Likewise, unit testing
would fall under white box testing.
Black Box Testing
Black box testing is the method of choice for testing video
games. This testing strategy is done without access to the
code. It is done by the tester to interpret how a user would
play the video game in normal use. When you hear about
the scenarios and picture a group of recent high school
graduates in a room drinking Mountain Dew and playing
and testing video games, they are black box testing.
Not only is black box testing an easy way to emulate a
user, it is also a cheap way. Black box testers need minimal technical knowledge, but good hand-eye coordination
from heavy gaming is highly sought after. You need testers
who can play a game all the way through quickly and then
play again going a different path through the game.
Black box testing is advantageous in that it can separate
the player’s perspective from that of the developers. However, since the tester does not see the code, if a defect is
found it is harder for the developer to pinpoint the segment
of code that is erroneous.
White Box Testing
White box testing is testing with access to the code and is
the technical side of video game testing. For example, unit
testing falls under this testing strategy. White box testing
TestComplete
by SMARTBEAR
7
is advantageous since it’s easier to see what inputs can
be effective in testing. Cleaning up code that could cause
bugs is done in this strategy.
Even though only a small amount of white-box testing
directly relates to game play or user experience, it is still
important in optimizing the code for better game performance and having better coverage overall for the gaming
application.
Automated Testing’s Role in Video Game
Development
So how about automated testing? Wouldn’t implementing
a test automation tool such as TestComplete help increase
velocity amongst video game testers? Well, yes and no.
Maybe your game has a database, like Diablo, in which
there is a list of items and the possible properties and rarity
those items can have. Using TestComplete for data-driven
testing in a video game database would work wonders and
save time for your team. It’s not that test automation can’t
be used, but rather the context determines how it is used.
Simply put, because there are so many variables in a video
game such as user interaction, 3D environments and AI,
a test automation tool could not easily emulate gameplay.
Some companies even have used automated bots (AI
to emulate users) to speed up the testing process. The
problem with this approach is that, for example, a bot goes
to click on an item in the game but then a creature runs in
front of the bot as it goes to click on that item, the test will
fail. A user would just move and click the item if the creature got in the way.
TestComplete
by SMARTBEAR
8
In another situation, you could request what amounts to
a “temporary easter egg” from the dev team that your tool
can use to “warp” right to the shop. This would allow you to
set variables such as “player level” so your tool can check
to see if the correct items are available for you based on
those variables.
So it is important to realize that automated
testing may not be applicable for all projects
but that does not mean there aren’t benefits to using test automation for
game development. Some of
the main benefits are to
understand the risks
of game components
and finding defects
faster and earlier.
Since video games are
becoming massive in scale, test
automation will also be a primary
approach for regression testing
as game code gets larger with
each update.
Then there is always the case of the
platform in which your game is developed for. TestComplete has been proven as a terrific tool for automated functional testing for HTML5 and Flash applications, and video
games for those web technologies are no exception.
In actuality, developing games for HTML5 is becoming
more popular due to the advancements of the technology as a multi-platform approach to gaming. For example,
TestComplete
by SMARTBEAR
9
instead of porting a web game from the web to a mobile
device, you could simply develop a game for HTML5.
Since HTML5 is supported across almost every popular
platform – from web to mobile - you would have a multiplatform game without the hassle of porting the code over for
each platform.
Video Game Testing on a Mobile Platform
Video games and mobile devices have a lot in common
when it comes to testing; they both have many variables
the limit how testers can approach them. You can imagine
that developing a game on a mobile platform is twice as
difficult. Not only are you dealing with the nuances of video
game testing, you basically throw a wrench at software
testing with the variables involved in mobile development,
for example, having no idea what the attributes are of the
device your game will be deployed to—since the “most
popular devices” on the day your game is released may
not have even been announced when you start your testing. This is the main reason that mobile gaming is a work
in progress and many games are based off of a simplistic
GUI.
Below is a list of variables that need to be considered
when developing a game for a mobile platform.
Mobile
Data
¿¿ How much data? What data?
¿¿ What are your privacy policies?
¿¿ What are you accessing on their device?
¿¿ What is your end users’ tolerance for privacy
TestComplete
by SMARTBEAR
10
encroachment?
Network Availability
¿¿ How does your app perform on various carrier
networks as opposed to Wifi?
¿¿ Will your app be a data hog that causes users to
watch their bandwidth usage as a result?
¿¿ Is the performance on Wifi so markedly improved
as to discourage the end user from using the app in
any other environment?
Gestures
¿¿ Does your app give the user gesture options that are
confusing or that create conflict with other apps?
¿¿ If they are not willing to adapt to the gestures you
have incorporated in your app is the app navigable
without them?
GUI
¿¿ Does your app have a command line feel to it or is
it appealing to even the most non-technical of users
(who will be a vast majority of most app audiences
anyway)?
Portrait and Landscape Views
¿¿ What happens to your app when your end user turns
his or her device on its side or on its head?
¿¿ Does it respond quickly?
¿¿ Is it responsive at all?
¿¿ Are there any fields that suddenly become
impossible to navigate because of the limited space
due to the keyboard expanding for use?
Processing Power
TestComplete
by SMARTBEAR
11
The last thing you want your app to be associated with
is making your user’s device so slow because it is a tremendous resource hog. If within those first few hours of
downloading your app your customer notices that nothing is working as well on their device as it did before they
downloaded your app then your app will be immediately
removed.
Memory
Let’s be honest. Most mobile device users are pushing
their memory limits to the nth degree on any given day.
¿¿ How will your app impact device memory?
Battery
What impact will your app have on the battery life of a
mobile device? If an end user downloads your app then
suddenly sees their battery life fall through the floor you
may be viewed as a troublemaker. Not good.
Fragmentation of devices
There are an ever growing list of devices, screen sizes,
form factors, etc. that make up the mobile marketplace.
It’s not cheap to make design alterations for every make
and model. Also, once you make accommodations for one
you may end up breaking something with another.
Will you need to pick and choose which devices are going
to represent the largest portion of your target market then
go from there? This is a critical decision that must be taken
seriously.
TestComplete
by SMARTBEAR
12
Gaming
Audio
¿¿ Do sound effects play at the correct time?
¿¿ Does music play at the correct time?
¿¿ Do sound effects and music combined cause
cancellation at certain frequencies?
¿¿ Is the sound overall clear on the device in which it is
playing?
Graphics
¿¿ Are devices that supposedly meet hardware
requirements able to render graphics at a stable
frame rate?
¿¿ Are there unwanted graphical errors appearing on
the screen, such as, artifacts?
¿¿ Is the layering correct?
Accessibility
¿¿ Is your game accessible to your target market?
Databases
¿¿ Is the game accessing the correct data from the
correct server?
TestComplete
by SMARTBEAR
13
Conclusion
I hope you now have a better idea what to look for when
developing a mobile game. As stated before, there isn’t a
right or wrong approach to testing a mobile game but you
need to at least try to consider all the possible variables
both physically and virtually. Sometimes it’s best to try
to list every possible situation a player would be in when
playing a game. Even if you do manage to identify every
possible variable, scenario and their interactions, testing
them will be a combination of targeting the moving parts of
mobile apps and games. With practice and determination,
you should be able to accomplish great testing on great
new mobile games without breaking the budget or letting
your competitors beat you to market, ensuring your games
are easy to understand for the players… and easy on the
eyes.
TestComplete
by SMARTBEAR
14
Try TestComplete
Get started with
automated testing
today
About SmartBear Software
More than one million developers, testers and operations professionals use SmartBear tools to ensure the quality and performance of their
APIs, desktop, mobile, Web and cloud-based applications. SmartBear
products are easy to use and deploy, are affordable and available for
trial at the website. Learn more about the company’s award-winning
tools or join the active user community at http://www.smartbear.com, on
Facebook or follow us on Twitter @smartbear and Google+.
SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900
www.smartbear.com ©2013 by SmartBear Software, Inc. Specifications subject to change
EBOK-TC-20131004-V1-GAMETESTING