Submitting UITests at the Command Line Overview

Submitting UITests at the Command Line
Overview
This guide discusses how to submit your UITests tests to Xamarin Test Cloud using the
command line. This scenario is appropriate for continuous integration servers or builds that
performed at the command line using a script. It will also discuss strategies for signing your
Android APK’s with custom keystores.
Requirements
It is assumed that UITests have been written and are running in your local environment, and
that you have the test-cloud.exe application installed via NuGet.
It is assumed that a Test Cloud project has been created, your account has been added to
the team for that project, and you have the Team API Key. Please see the Organizations &
Teams document for more Details.
Familiarity with the command line is required.
Internet access to the Test Cloud web servers is required. Firewalls should be configured to
allow traffic to the Test Cloud servers at the following IP addresses:
195.249.159.238
195.249.159.239
The Test Cloud servers require port 80 and 443.
Android applications must request the INTERNET permission in AndroidManifest.XML.
Submitting UITests
Tests are submitted to the using test-cloud.exe command line application, which is
distributed as a part of the Xamarin.UITest NuGet package. This is a multi-purpose utility
that will submit both Android and iOS tests, and it works on both OS X and Windows.
The following table describes some of the more important command line parameters for
test-cloud.exe, which may be found in the packages directory as shown in the screenshot
below. It is not a complete list of all parameters.
Parameter
--assembly-dir
<PATH>
--devices
<DEVICE ID>
Description
The path to the folder holding the test assemblies.
The device ID that was provided in the Test Cloud Upload dialog.
--user <EMAIL> The e-mail address of the team member submitting the tests.
--nunit-xml
The filename to which test results are exported, formatted as NUnit
<FILENAME>
results XML. Optional.
--sign-info
Android only. Supply a signing information file that will be used to sign the
<FILENAME>
Test Server APK. See the section below for more details. Optional.
--dsym
iOS only. Will upload the dSYM files along with the application and tests.
<FILENAME>
This allows for more detail in the log files. Optional.
--category
<NUNIT-
NUnit category to run. (Can be used multiple times)
CATEGORY>
--fixture <NUNITFIXTURE>
NUnit fixture / namespace to run. (Can be used multiple times)
--test-chunk
Run tests in parallel by method
--fixture-chunk
Run tests in parallel by fixture
It is important that all commands be run from the solution directory. As an example, the
following screenshot of the CreditCardValidation folder on Github identifies the solution
project for the CreditCardValidation-UITest project:
It is important that all commands be run from the root of the solution directory. As an
example, the following screenshot of the CreditCardValidation folder on Github identifies the
solution project for the CreditCardValidation-UITest project:
Note: Xamarin Test Cloud will terminate test runs that take longer than 180 minutes.
Submitting iOS Tests
The following snippet is the simplest command line that could be used to submit an iOS
application and tests to Test Cloud:
test-cloud.exe submit <PATH-TO-IPA> <TEAM API KEY> --devices=
<DEVICES> --assembly-dir=<PATH-TO-TEST-ASSEMBLY-DIR> --user=<EMAIL>
Submitting Android Tests
The following snippet is the simplest command line that could be used to submit an Android
application and tests to Test Cloud:
test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> --devices=
<DEVICES> --assembly-dir=<PATH-TO-TEST-ASSEMBLY-DIR> --user=<EMAIL>
Only release builds of an Android application should be submitted to Test Cloud.
test-cloud.exe will sign the application APK and the test server APK with the same keystore
before uploading to Test Cloud. It is possible to control this signing step as described in the
next section.
Signing Android APK's
test-cloud.exe will use one of three algorithms to sign the APKs prior to submission:
1. Use Build-in Keystore. This is the default behaviour. test-cloud.exe will sign the
application and the test server APK using it’s own keystore.
2. Provide your own Keystore. The second option is to provide your own keystore
(along with credentials) that test-cloud.exe will use to sign the application and test
server APK. The following snippet shows the additional command line parameters:
test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> keystore
<PATH-TO-KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIASPASSWORD>
This option allows for complete control over what key will be used to sign the application, but
it presents a potential security risk because the application keystore must be included in
source code and the password for the keystore will be in clear text in build scripts.
1. Use a signing information file. This option involves using test-cloud.exe to extract a
few files from a signed test server APK, which are then stored in a small zip file, known
as a signing information file. The main reason for using a signing information file is so
that it can be added to source code control. This ensures that the application and test
server are consistently signed with the same key information.
The following snippet shows how to create the signing information file:
test-cloud.exe gen-sign-info <PATH-TO-SIGNED_APK> <PATH_TO_KEYSTORE>
<KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS-PASSWORD> --dir <DIRECTORYTO-SAVE-SIGN-INFO>
When run, test-cloud.exe will create a file called testserver.si in the directory specified by
the --dir parameter.
Note: The signing information file must be recreated each time the Xamarin.UITest package
is updated.
Summary
This guide discussed how to use test-cloud.exe to upload Xamarin.UITests and
applications to Xamarin Test Cloud for testing. It also discusses several options for how to
sign Android APKs prior to submitting test runs to Test Cloud.