flank/flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab

Flank

Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.

Flank is YAML compatible with the gcloud CLI. Flank provides extra features to accelerate velocity and increase quality.

Documentation is at flank.github.io/flank

Github action documentation is at repository

flank's Issues

Support Parametrized

While running tests in FTL with and without Flank i see some tests not running in Flank case:
seems all of them has @RunWith(Parameterized::class) in common.

Is it possilble to support them?

support android test orchestrator flag

Google Cloud SDK 167: "Promoted Firebase Test Lab to beta for running instrumentation test with Android Test Orchestrator. Use --use-orchestrator to run with Orchestrator."

This "allows you to run each of your app's tests within its own invocation of Instrumentation" and avoid shared state and crashes taking out other tests.

Would it be possible to have a configuration option for enabling this?

Error when test apk has kotlin interfaces with @Test methods implementations

When flank generated the shards, firebase gave a 400 saying that the test is invalid.
This gist has the complete log.

The apks (with I already sent to @renas, if anyone want to test) run successfully on my machine.

It was also a bit hard to understand what logs are from what shard. Maybe putting a common prefix to logs about what shard they're refering to could help debugging.
E.g:
https://gist.github.com/thalescm/7e64bc80fc2a32dcb8eef50fc0a96aac#file-flank-shard-error-L51-L56
all lines could start with something like SHARD X: <real log>

[Feature] Run all tests

java -jar flank.jar app.apk test.apk
Usage: Flank

I'd expect ^ to run all tests. Instead I have to specify a package. Thoughts on allowing all tests to run when the package is omitted? This is more in line with the gcloud CLI. Package is optional and default is to run everything.

How many numShards to use?

I am trying out flank for the first time.
What should be my guideline?

Add --no-record-video config option

There was added support of passing the argument --no-record-video. It would be great to pass it with flank to get faster test results

Add git tags for versions

Request: Add git tags for versions

For example, ideally there would be a git tag for 1.4.1

Flank-1.3.0 Crashes - Can't run program gcloud

Was trying to test Flank but can't get it to execute correctly.

• Using Powershell though also tried Google Cloud SDK Shell
• Windows 10 Version 1703 Build 15063.413
• Google Cloud SDK 159.0
• Have gcloud beta 2017.03.24 installed as well
• Using Flank-1.3.0
• java version 1.8.0_131

Running command as follows:
java -jar Flank-1.3.0.jar app\build\outputs\apk\app-debug.apk app\build\outputs\apk\app-debug-androidTest.apk com.amazonaws.devicefarm.android.referenceapp

Am greeted with this error message. Note gcloud command is working fine for me when used outside of Flank.

java.io.IOException: Cannot run program "gcloud": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at com.walmart.otto.tools.ProcessBuilder.start(ProcessBuilder.java:24)
at com.walmart.otto.tools.ProcessExecutor.executeCommand(ProcessExecutor.java:31)
at com.walmart.otto.tools.Tool.executeCommand(Tool.java:69)
at com.walmart.otto.tools.GcloudTool.getProjectName(GcloudTool.java:118)
at com.walmart.otto.Flank.getProjectName(Flank.java:153)
at com.walmart.otto.Flank.start(Flank.java:40)
at com.walmart.otto.Flank.main(Flank.java:73)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 8 more

FTL | JUnit test time is always zero

The timing info in the results.xml files is wrong. Firebase is aware of the issue but they probably won't fix it anytime soon. I think overriding the xml files with the actual time as recorded by flank might be helpful. This would allow, among other things, accurate cost estimation.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<testsuite errors="0" failures="0" hostname="localhost" name="" skipped="0" tests="2" time="1.823" timestamp="2017-05-26T15:40:36">
  <properties/>
  <testcase classname="com.instructure.teacher.ui.DeviceTest" name="checkAnimationsOff [NexusLowRes-25-en-portrait]" time="0.0"/>
  <testcase classname="com.instructure.teacher.ui.DeviceTest" name="displaysPageObjects [NexusLowRes-25-en-portrait]" time="0.0"/>
</testsuite>

The issue is time="0.0" which isn't true.

[bootstraponline] number of changes to the object exceeds the rate limit.

renas commented on f074037 a minute ago
@bootstraponline I'm getting this error now when I run test suites containing large amount of tests:

Uploading [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk] to Firebase Test Lab...
ERROR: (gcloud.firebase.test.android.run) Could not copy [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk] to [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/] ResponseError 429: The total number of changes to the object walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk exceeds the rate limit. Please reduce the rate of create, update, and delete requests..

Before when we only had a root project we didn't experience this error. Seems to have come with the sub-folder object that we now create.

f074037#commitcomment-22496961

moving this to an issue so it can be tracked properly.

No tests?

It looks like the only test flank has is empty?

It'd be nice to setup some E2E tests on Travis CI using encrypted variables.

Add more tests filtering options

At current state, Flank supports running tests based on:

• package (single)
• class (single)
• test name (single)

It's possible to filter tests in such a way passing the package parameter when invoking Flank.
In addition to that is possible to run all but some tests defining the skip-tests property in the config file.

Taking inspiration from what AndroidJunitRunner supports, we could add these execution options:

• running multiple tests
• running all tests in multiple classes
• running all tests except those in a particular class
• running all but a single test
• running all tests except a particular package
• running all tests with a given annotation(s)
• running all tests without a given annotation(s)

Related to #89 (comment), since it would require a new CLI arguments mechanism.

IntelliJ Java lint warnings

The IntelliJ linter is picking up some weird Java issues. We should probably fix them. And then add linting to the Travis CI build.

Bucket upload folder still not unique

public String uploadAPKsToBucket() {
SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd-hh-mm-ss");

This will break the moment two PRs are submitted at the same time. This isn't unusual with a large team. If you checkout the Google cloud Python SDK code, they add some random numbers in addition to the timestamp.

Observing that logcat/ video and other artifacts are not being created for all the shards

Hello,
Thanks for the great project and open-sourcing it. I have been successfully able to run our entire suite of ~60 tests with minimal effort locally and also on our Jenkins.
I am observing a rather strange behavior: All the tests though executed in a separate matrix are essentially showing the same artifacts including logcat/video.mp4 file. In the gsutil pull down command I can clearly see only one video file being generated.

[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_7874888483897436532.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_9131515297737460228.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_8667431412748016253.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_8330274942631895392.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_9101555559323915915.xml...
[android] Operation completed over 49 objects/16.2 KiB.                                    
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/artifacts/com.godaddy.gdm.telephony.uitests.SwitchAccountNumberTest-switchAccountTest-switchAccountTest-1.jpg...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/artifacts/video.mp4...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/artifacts/coverage.ec...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/bugreport.txt...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/logcat...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/instrumentation.results...

When I open the test_results file I can clearly see that tests are being executed and some have proper failure logs with them. But I can't find the test being executed in logical. Is there a setting I am missing?

My config.properties look pretty normal

deviceIds=Nexus6P
os-version-ids=25
orientations=portrait  
locales=en
environment-variables=coverage=true,coverageFile=/sdcard/coverage.ec
directories-to-pull=/sdcard
shard-timeout=5 
shard-duration=-1

numShards=  
shardIndex= 
debug-prints=true
fetch-xml-files=true
fetch-bucket=false
gcloud-path=
gsutil-path=

do let me know what details you need to diagnose this problem.

findbugs warnings

./gradlew findbugsMain

Generate client bindings from API discovery JSON

Currently the API jars are sourced from maven or the Android Studio repo. The gcloud SDK generates APIs using google/apitools. Using that same approach for Flank would allow us to generate an always up to date Java binding.

• https://github.com/google/apitools
• def GenerateApi in /googlecloudsdk/api_lib/regen/generate.py#L52
• gcloud-cli/tools/regen_apis/

The new generation tool is googleapis/toolkit however I haven't found the protobufs required to make that work for the testing API.

Empty results

The local results folder is empty. There's also no link to the corresponding Firebase test run.

$ java -jar flank.jar app.apk test.apk com.example.ui.DeviceTest

No Local 'flank.tests' found. It's used to create shards with configurable durations.

Creating bucket: gs://apk-bucket-2017-05-25-12-35

Uploading: app.apk to bucket: gs://apk-bucket-2017-05-25-12-35

Uploading: test.apk to bucket: gs://apk-bucket-2017-05-25-12-35

2 shards will be executed on: Nexus6P

Executing shard 0:  com.example.ui.DeviceTest#displaysPageObjects

Executing shard 1:  com.example.ui.DeviceTest#checkAnimationsOff


Fetching results to: ~/code/tmp/results

Combined test execution time: 0 seconds

End time: 2017-05-25-12-35

FTL | Orchestrator breaks code coverage

ATSL doesn't collect code coverage properly when orchestrator is enabled. A work around is to disable orchestrator. The ATSL team may fix this issue in an upcoming release.

https://issuetracker.google.com/issues/72758547

---

As Firebase test lab lacks a public issue tracker, I'll track the known issues here since they also impact Flank.

Fetch coverage.ec to results directory

Coverage is fetched as bucket artifact, but it can take significant time to fetch all the artifacts just to get coverage results. It'll save some build time if coverage is fetched along with test_results

Flank is unable to fetch results after test run.

We have two projects where we use Flank, one of them works fine, but another fails to fetch test run results. Projects have identical Flank configuration except gcloud-bucket.

After all tests pass Flank crashes with following log:

Created folder: /project/instrumentation-tmp/results

Fetching results to: /project/instrumentation-tmp/results
java.lang.RuntimeException: CommandException: No URLs matched: flank.tests
	at com.walmart.otto.tools.Tool.executeCommand(Tool.java:60)
	at com.walmart.otto.tools.GsutilTool.uploadTestTimeFile(GsutilTool.java:81)
	at com.walmart.otto.Flank.uploadTestTimeFile(Flank.java:151)
	at com.walmart.otto.Flank.start(Flank.java:65)
	at com.walmart.otto.Flank.main(Flank.java:77)

Environment:

• Flank 1.4.1
• Google Cloud SDK 149.0.0-linux-x86_64

I've also double checked gcloud-bucket and gcloud --key-file, it seems to be a Flank issue since it's able to run tests but unable to download results.

Crash on valid apks

I have a configuration like this:

deviceIds=Nexus6
os-version-ids=25
orientations=portrait
locales=pt_BR
shard-timeout=60
shard-duration=2400

numShards=8
shardIndex=
debug-prints=true
fetch-xml-files=false
gcloud-path=
gsutil-path=

and two valid apks (just ran the tests with them on my machine and all passed), but when trying to use them to shard with flank, a crash happens:

Exception in thread "main" java.util.EmptyStackException
	at java.util.Stack.peek(Stack.java:102)
	at java.util.Stack.pop(Stack.java:84)
	at com.walmart.otto.shards.ShardCreator.addToShard(ShardCreator.java:56)
	at com.walmart.otto.shards.ShardCreator.getShards(ShardCreator.java:39)
	at com.walmart.otto.shards.ShardExecutor.executeShards(ShardExecutor.java:43)
	at com.walmart.otto.shards.ShardExecutor.execute(ShardExecutor.java:32)
	at com.walmart.otto.Flank.main(Flank.java:48)

Note that the test APK has near 900 instrumentation tests (don't know if that can contribute to the problem somehow)

Enable user supplied shards.json

We'd love to be able to specify a json file of shards and have Flank use that. Motivation:

1. supports teams who want deterministic test groups and orders
2. allows for teams to implement their own sharding algorithms which can write this file out in a previous step

This could be configured via a shardManifest=path/to/shards.json in the config.properties. Example test JSON file:

[
  [
    "com.example.android.app.androidapp.ApplicationTest",
    "com.example.android.app.view.activity.LoginFlowTest",
    "com.example.android.app.view.activity.LogoutFlowTest"
  ],
  [
    "com.example.android.app.view.activity.SplashScreenTimeoutTest",
    "com.example.android.app.view.activity.HomeFlowTest"
  ]
]

If you would be interested in this feature, I am happy to attempt a pull request or work with an existing flank contributor to do so :)

After flank 1.5.0 tests fail to execute on gcloud

I've recently upgraded to flank 1.5.0 and started seeing the following output

Executing shard 107:  com.godaddy.gdm.telephony.WebSocketTest#testIncomingSmsWebSocket


Fetching bucket to: /Users/smalugu/code/gdm-android-telephony/bucket

Billable time: 0.0 min(s)

Estimated cost:

[2017-12-02_09:13] Total time: 0 min, 25 sec

When I check google firebase console, I see no tests. If I change to flank 1.4.0 tests run just fine.

My config.properties look like this

deviceIds=NexusLowRes
os-version-ids=25
orientations=portrait  
locales=en
environment-variables=^:^coverage=true:coverageFile=\"/sdcard/coverage.ec\":notAnnotation=android.support.test.filters.FlakyTest,com.godaddy.gdm.telephony.uitests.testcategories.DeviceOnlyTests
directories-to-pull=/sdcard
shard-timeout=5 
shard-duration=-1

numShards=  
shardIndex= 
debug-prints=false
fetch-xml-files=false
fetch-bucket=true
gcloud-path=
gsutil-path=

I also tried changing to new syntax for devices and that didn't help. Is there something I am missing?

[Feature] Retry failed tests automatically

It looks like there's no option to retry failed tests? That'd be helpful when 1 test runs in each VM (shard-duration: -1).

Failed gcloud commands do not end the run

With debug = false, all the gcloud commands can fail and flank will happily report success. We should probably check to see if there's anything in the error stream, and if so end the process with a non-zero exit code.

Specify or calculate default test time.

Right now it's a bit of a pain to have to update the flank.tests every time a new test is added. If new tests would automatically be added to other shards based on an average test duration time, or a default value specified in config.properties, it would save a bit of time having to update the test times every time a new test is added.

Choose buckets

Why is it necessary to upload the apks to a gs bucket instead of using the ones on the provided path directly?

Another thing, are there plans to support other kind of buckets? My current CI saves the apks on a s3 bucket on aws, it would be nice to fetch than from there and not needing to replicate them to gs://<>

The last thing, is there a CONTRIBUTING.md guide somewhere or is just to fork and make a PR with the desired feature?

Aggregated report throws a runtime error

Can't process test artifacts for: testSurfaceLogging
Can't process test artifacts for: testAppQuitViaActivityFinish
Can't process test artifacts for: testSaveObstruction
Can't process test artifacts for: testOverflowObstruction
Can't process test artifacts for: testTriedItEndsImpressions
Can't process test artifacts for: testSaveObstruction
Can't process test artifacts for: testSaveObstruction
java.lang.RuntimeException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /project/bucket/2018-01-24_18:01:44.378000_Mlhf/41/NexusLowRes-25-en-portrait/logcat

The xml aggregated and uploaded fine. I, unfortunately, don't know if the not found file was ever created. I believe that the can't process test artifacts were all shards that partially failed.

cost estimate on Jenkins error

When set config.properties like deviceIds=Nexus5X,Nexus6P os-version-ids=23,24 orientations=portrait locales=en,sv
The cost on Jenkins will looks like 00:41:10 Billable time: 0.0 min(s) 00:41:10 00:41:10 Estimated cost:

StringIndexOutOfBoundsException

java -jar Flank-1.4.1.jar walmart-android-app-dev-debug.apk walmart-android-app-dev-debug-androidTest.apk

Downloading 'flank.tests'. It contains test execution times used to create shards with configurable durations. Default shard duration is 120 seconds.

java.lang.StringIndexOutOfBoundsException: String index out of range: 6
at java.lang.String.substring(String.java:1963)
at com.walmart.otto.tools.GsutilTool.uniqueObjectName(GsutilTool.java:33)
at com.walmart.otto.tools.GsutilTool.uploadAPKsToBucket(GsutilTool.java:64)
at com.walmart.otto.Flank.start(Flank.java:59)
at com.walmart.otto.Flank.main(Flank.java:77)

config.properties should be a parameter

In my setup, I will have a different cfg for ci and a different cfg for regression testing. Example, I want ci to run in a single locale / single device.

What do you think of removing all the parameters we have right now and force a config.

so instead of:

java -jar Flank.jar <app-apk> <test-apk> [package-name]

it would be:

java -jar Flank.jar myConfig.properties

properties file should have new keys:
app_apk_path, tests_apk_path, use_tests_in_packages to map the previous parameters

ServiceException: 409 bucket already exists

$ gsutil mb gs://delta-essence-114723-flank/
Creating gs://delta-essence-114723-flank/...
ServiceException: 409 Bucket delta-essence-114723-flank already exists.

We should probably check to see if the bucket exists before running the make bucket command.

[Feature] Print command duration

Gradle builds print time info:

 BUILD SUCCESSFUL
[22:54:52]: ▸ Total time: 4 mins 58.471 secs

It'd be nice to have a Total time that represents the amount of time from the start of flank until the end. Currently I have to wrap flank in a script to figure out how long we're waiting for flank to run the tests.

project premission?

It looks like I'm not able to move cards across the board. 😭

• https://github.com/TestArmada/flank/projects/1

What is the reason to require paid Firebase plan?

TL;TR: we want sometimes run our instrumentation tests in Firebase, but most of the time run them on internal emulators pool.

So we're completely fine with free plan quotas but Flank requires paid plan (for Google Cloud Storage bucket I guess).

What is the reason to require paid plan and can this be fixed to allow test runs on free plan? Thanks!

Skipping tests in certain package

I have some tests used by fastlane+screengrab which should not be ran by flank.

I would like to have a skip-tests in the config file specifying the package of the tests which shouldn't be run. What do you think?

Also, is there a reason why the package which should be run are not part of the config file instead of the command line args?

Thanks!

Does not scale to large test suites

I have a test suite with 112 tests. These tests run on Firebase test lab on every pull request. Currently, we're using a single VM. The configuration is defined in fastlane. Our apps, tests, and build automation are all open source.

gcloud test runs directly on a single VM and takes 0:09:39 (112 tests).
flank with 112 VMs takes 0:13:24.

Flank increased our cost by 18x and is 3m 45s slower.

$ cat config.properties
debug-prints:true
deviceIds: NexusLowRes
os-version-ids: 25
orientations: portrait
locales: en
shard-timeout: 5
shard-duration: -1

The stdout is filled with / [102/113 files][ 2.0 MiB/ 2.1 MiB] 93% Done. It looks like more commands require the quiet flag. The performance loss is because we're:

• fetching XML results from each VM
• spinning up 1 python process per VM
• trying to access the Walmart only flank bucket

In the 1 test per VM model, everything we need is already in stdout. The test name, success/failure and a link to the full report. Skipping XML fetching would be a huge speed win.

In terms of avoiding spinning up the processes, you can check out this proof of concept:

• https://gist.github.com/bootstraponline/0b8fe4907c36ebb7521538e83a1ed91a#file-shard-py

I'm spinning up multiple test VMs from a single python process by extending the SDK with a new shard command.

[Feature] Cost estimate

Combined test execution time: 26 seconds

seconds are round up to minutes by Firebase test lab

thoughts on having flank output estimated cost for the test run? The combined test execution time isn't sufficient to figure out the expense. The output would look something like this:

Estimated cost: $0.033 (virtual), $0.166 (physical)

2 tests at <1 min each

• 1/60.0 * 2 = $0.033 virtual device
• 5/60.0 * 2 = $0.166 physical device

per test sharding?

Instead of creating one shard per test case Flank can create shards that contain tests that add up to a given duration

Is it not possible to have one shard per test case? In terms of fault isolation/video/logs, there are some use cases where this is desirable.

CLI command parsing

You can read more about this in #89

Requirements:

• Add https://pholser.github.io/jopt-simple to parse commands
• new CLI should work like this:
  java -jar Flank.jar -a <app-apk> -t <test-apk> -c <config-path> -p [package-name]

a, t are required, c and p are optional.

add option to exit non-zero if tests failed

This would be a helpful configuration option that many tools provide, and would allow for more easily performing certain logic in CI environments if tests fail (without globbing and grepping xml files). Thanks for your consideration!

Flank tries to run tests with @Ignore and thus crashes AndroidJUnitRunner.

We had to @Ignore couple tests and that turned out to be problem for Flank, because it tries to run them anyways (by passing -e class class1#test1,class1#test2,class2#test1) and that crashes android.support.test.runner.AndroidJUnitRunner with following stacktrace:

java.lang.NullPointerException: Attempt to invoke interface method 'int java.lang.CharSequence.length()' on a null object reference
	at java.util.regex.Matcher.reset(Matcher.java:995)
	at java.util.regex.Matcher.<init>(Matcher.java:174)
	at java.util.regex.Pattern.matcher(Pattern.java:1006)
	at android.support.test.internal.runner.TestRequestBuilder$MethodFilter.stripParameterizedSuffix(TestRequestBuilder.java:523)
	at android.support.test.internal.runner.TestRequestBuilder$MethodFilter.evaluateTest(TestRequestBuilder.java:509)
	at android.support.test.internal.runner.TestRequestBuilder$ParentFilter.shouldRun(TestRequestBuilder.java:145)
	at android.support.test.internal.runner.TestRequestBuilder$ClassAndMethodFilter.evaluateTest(TestRequestBuilder.java:451)
	at android.support.test.internal.runner.TestRequestBuilder$ParentFilter.shouldRun(TestRequestBuilder.java:145)
	at org.junit.runner.manipulation.Filter$3.shouldRun(Filter.java:112)
	at org.junit.runners.ParentRunner.shouldRun(ParentRunner.java:434)
	at org.junit.runners.ParentRunner.filter(ParentRunner.java:382)
	at org.junit.runner.manipulation.Filter.apply(Filter.java:97)
	at android.support.test.internal.runner.TestRequestBuilder$LenientFilterRequest.getRunner(TestRequestBuilder.java:412)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:369)
	at
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1962)

And of course Firebase marks test run as successful even though no tests were run.

Combined Test Reports

In the Medium article about flank (https://medium.com/walmartlabs/flank-smart-test-runner-for-firebase-cf65e1b1eca7) a paragraph describes that flank should output a JUnit report as if all tests were run on a single device.

However when I'm running all tests with flank I'm only getting all the *.xml reports for the testsuite and no combined report.

Is there a flag that I'm missing to generate this report? Or what is the preferred way to combine all the XML reports to one unified HTML report?

Missing full tests results

Due to changes in naming convention on FTL side, after fetching results there is only one file test_results_0.xml and it corresponds to tests executed on the last shard. The same is also applied to coverage.ec.

Occasional NPE stopping builds from succeeding

Version: 1.7.0

java.lang.NullPointerException
	at com.walmart.otto.reporter.PriceReporter.getBillableTime(PriceReporter.java:57)
	at com.walmart.otto.reporter.PriceReporter.getTotalBillableTime(PriceReporter.java:40)
	at com.walmart.otto.Flank.printEstimates(Flank.java:148)
	at com.walmart.otto.Flank.start(Flank.java:83)
	at com.walmart.otto.Flank.main(Flank.java:107)

Could we make it so the build doesn't fail and the billable time was just left unknown?

End time formatting is weird?

End time: 2017-05-25-12-35

Is this a bug? ^

Invalid configuration does not halt run

I'd expect the test run to stop on an exception, not proceed with an invalid configuration.

$ java -jar flank.jar app.apk test.apk com.instructure.teacher.ui.DeviceTest
java.lang.NumberFormatException: For input string: "25m"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:580)
	at java.lang.Integer.parseInt(Integer.java:615)
	at com.walmart.otto.configurator.ConfigReader.setProperty(ConfigReader.java:64)
	at com.walmart.otto.configurator.ConfigReader.setProperties(ConfigReader.java:23)
	at com.walmart.otto.configurator.ConfigReader.<init>(ConfigReader.java:14)
	at com.walmart.otto.Flank.main(Flank.java:26)

Local 'flank.tests' found. It contains test execution times used to create shards with configurable durations. Default shard duration is 120 seconds.

Creating bucket: gs://apk-bucket-2017-05-26-11-34

Creating gs://apk-bucket-2017-05-26-11-34/...
Uploading: app.apk to bucket: gs://apk-bucket-2017-05-26-11-34

$ cat config.properties
debug-prints:true
deviceIds: NexusLowRes
os-version-ids: 25
orientations: portrait
locales: en
shard-timeout: 25m
numShards: -1

Add support for device specification on config.properties

As you can do with gcloud command, you can specify device ids and versions, or you can specify each device with it's own config (version, orientation, locale and model)

The point is to add support for this type of configuration as well on config.properties so you could have a specified device with API 19 and another with API 25.

Today when trying to do this you would end up with four configurations (both devices with API 19 and 25) instead of two.

Build on Travis CI

CI helps make sure that PRs don't break the code. I've submitted a .travis.yml integration. Someone with admin rights will have to enable Travis for this repo. After that, a badge can be added to the readme.

• #6