Test Cloud

Xamarin.Android APK Tests on Test Cloud

Leveraging Test Cloud seems like a good idea, this document aims at figuring out the requirements for our team.

Current APK Tests

Xamarin.Android has various APK-based tests that currently run on emulators/devices.

In general, they operate by running adb commands:

adb install sometestapp.apk
adb shell am instrument -w $(InstrumentationType)
adb pull /path/to/TestResults.xml

The tests APKs are using NUnitLite, its instrumentation runner drops the test results in a directory on-device. Jenkins ingests the NUnit reports after they are pulled from the device.

Requirements for Test Cloud

We need a way to:

• Run these NUnitLite on-device tests in Test Cloud
• Get the on-device test results locally
• Should work within a CI build from Jenkins or VSTS

There are a few problems here:

• Test Cloud operates by running an NUnit project on desktop
• How do the on-device tests get back to the desktop machine in Test Cloud?
• If running test-cloud.exe (CI machine, etc.), how do these test results get back to the calling machine?

Parallelization (Nice to Have)

• Is there a way Test Cloud can parallelize these tests? In particular, the BCL tests are slow due to the number of tests.
• Test Cloud parallelizes tests bases on test fixture or test methods. But since the tests are on-device, how will this work?

Exploration

I took a stab at running a set of APK tests on Test Cloud. Github repo here.

My approach was:

• Modify XA's NUnitLite (code here), that adds a "backdoor" to run the tests and return the NUnit xml results as a string
• Write a desktop NUnit project with a single test that calls the "backdoor" method
• Then what do we do with the XML? For now I printed them to the console.

See a successful test run here. View the Test Log to see the on-device test results.

Final thoughts

One approach suggested by James Clancey:

• Call Assert.Pass or Assert.Fail, passing the test results XML from on-device
• Write a simple program to "unwrap" the inner test results from the Assert.Pass/Fail call so they can be consumed by a CI system
• Currently it seems the test results file from the BCL tests are too large, and we are hitting a limit somewhere

We will see if this approach is viable, Clancey is still exploring this. It may be a better option to wait for Test Cloud to support for on-device, non-UI tests directly.