Using STF for Test Automation

[openstf]

Using STF for Test Automation

[jlhonora]

I recently found out about Square's Spoon tool, which would be a nice add-on for STF. I also saw that @sorccu made some of contribs there, so he's aware of the project at least. I don't think we should re-invent the wheel and use what's already available, any thoughts?

[sorccu]

Hi,

We are of course planning to offer automation, but currently we're pretty much fully booked with other bugs & requests.

We're hoping to have something available (perhaps limited to just 1 testing framework like Espresso) in a few months, but we can't make any promises. At first we're just planning on just creating a fairly simple API to reserve devices with the desired features, with the possibility to enumerate the devices in the response and adb connect to them yourself. We'd then build the real automation on top of this API.

Help would indeed be much appreciated, but our vision regarding this is not clear yet. Right now what would be very useful (aside from a complete implementation) would be a detailed description of how you would like the system to work, including any testing frameworks you need.

[vbanthia-zz]

Thanks for reply,

At first we're just planning on just creating a fairly simple API to reserve devices with the desired features, with the possibility to enumerate the devices in the response and adb connect to them yourself.

This can be a good start.

One limitation with this model which I can see now is, client(using "adb connect" api) will be running the test suites instead of STF. Client will be storing test results. It will be difficult to manage test history on STF in this model. Moreover, this limitation is only when if we want to support test history or project management on STF. User can always use CI tools such as Jenkins to manage project and test history by running test on Jenkins slaves machine using STF apis.

I will start digging up STF source code and try to come up with system requirements and possible designs related to test automation.

[vbanthia-zz]

Hi @sorccu ,

The kind of automation which I currently want to do is as follow:

• Run android monkey test and Appium test on multiple devices parallel from script.

From current STF, I can do this by doing:
1.) Connect all the required devices from STF web app
2.) Get Remote Debug Command string corresponding to each device
3.) Run remote debug command locally and get all devices connected
4.) Run monkey or appium test
5.) Disconnect devices from STF web app

In order to automate this fully, person will require apis which can lock and unlock desired device for him. And he will also need "Remote Debug Command String" which he can run locally to connect remote device.

I checked STF code and found that you have implemented some minimal apis here. By using these apis, person can get list of all devices and device info/status. But there is no api for locking a device.

I am proposing to implement two more end points which are as follow:
1.) /app/api/v1/devices/:serial/lock
2.) /app/api/v1/devices/:serial/unlock

The "lock" endpoint will do the same thing when user click "Use" button for a device and in response it will return device information (/app/api/v1/devices/:serial) + remote debug command string.

The "unlock" endpoint will release the device from user same when a user click "Stop Using" button.

If you like this idea, then I will start designing apis and implementing code. If you have other suggestions please share.

Thanks,

[rseek]

Hi, @vbanthia, @sorccu,
I just wanted to vote for approach offered by @vbanthia, and I think providing API to lock/unlock (maybe lock/unlock are not best words cuz it causes mess with screen locking) - is more or less simple but very useful step in this path to automation, because it makes stf easy-integrable in existing automated testing environments

[vbanthia-zz]

@rseek

lock/unlock can be changed with reserve/release. But lets wait for @sorccu opinion, what he suggests.

[sorccu]

That's more or less what we were planning to do at some point, so it would be very helpful indeed if you could have a go at it.

I prefer reserve/release as they are easier to understand (for me, anyway).

[gunta]

Lock/Unlock can be confused with the screen rotation indeed.
Reserve/Release sounds good.

The next step would be to reserve based on device capabilities, not only serial.

[vbanthia-zz]

Thanks for the opinions. I will start working on it soon :).

[jlhonora]

+1 . If someone is already working on this I'd be glad to help.

[danpe]

👍

[vbanthia-zz]

Well, I stopped in between. Writing my research and one of the approaches to implement this feature.

Authentication

• Every request to stf-app unit (api provider) goes through auth middleware which checks for valid session-id. If your request does not have a session id, it will redirect it to auth server to create a new session. So you need to modify this auth middleware such that it can forward request even it has only jwt token
• Next step will be to develop a feature which can provide users their jwt token. Best way would be to create a section in "Setting" tab and show users their jwt token like github.
• Next, you need to get rid of csrf checking for RESTful apis. One way to do is check csrf after RESTful api section.

After this you will be able to access all STF apis from any http client library using command line.

Implement reserve/release endpoints

• You will be going to implements these new endpoints in stf-app unit
• For this just mimic the websocket unit and do the same things what websocket unit does when it gets reserve and release messages.
• You need to add app side push sockets and push group.join and group.kick messages to processor and inform all the websocket subscriber about the event so that people who are using web browser can also be updated.
• For remote debugging you need to send connect.start message to server and get remote debug url.

Implement client

Last part being implement a good stf command line client which can use these endpoints easily.

I hope this will help.

[gunta]

@vbanthia you can use the preliminary UI for the access tokens in res/app/settings/keys/access-tokens (you'll need to uncomment it on settings/keys/keys.jade).

[gunta]

@jlhonora yes, we want to integrate any of the great tools available.
However we need first an API that its platform/tool agnostic first.

[vbanthia-zz]

you can use the preliminary UI for the access tokens in res/app/settings/keys/access-tokens

That's great. One less task to do.

[chibamap]

Good conversation. I had to read this issue first.
I'm making similar requirements on "booking" PR. Actually The feature aimed to combinate other Test Automation tool.
This is deferent approach. Very interesting.

[sorccu]

About the connect.start thing, I'm actually not very happy about having to retrieve the URL this way even in the current UI. Ideally I'd like to store it in the database, but the reason why it's not there right now is due to a complex dependency graph for the device identity module, which would otherwise be the prime candidate for storing the URL (since it currently also stores the screen WebSocket URL). Currently it has to be accessible before most of the services start. Although I guess we could create a new message for it so that it doesn't have to rely on identity.

[vbanthia-zz]

Feature will be available in next major release.

#200

[yashasvimakin]

Hi @vbanthia i wanted to run Espresso test test are you planning to do that soon as well ?
And Are you planning to use spoon as well?
Thanks and Regards,
Yashasvi.

[vbanthia-zz]

I haven't tried these tools yet but theoretically they should also work. All you need to do is,

1. Reserve the device
2. Remote connect
3. Start your test assuming device is connected to your own machine.

[yashasvimakin]

Thanks @vbanthia :) for replying so soon,
i will try running Espresso test. I wanted to edit code and add few UI and Functionality to run test.
Basic Idea:

1. Select Devices to run tests ,
2. Upload test Apk and ,
3. run test through a UI Button in web interface,
4. And display the execution of the devices selected

Also,
I am new to this can you please tell me how to run the code after editing.

Thanks,
Yashasvi.

[PrabhatPandey]

Using Jenkins , I was able to set up a fully functional Test Automation structure and running it successfully now .
Here is how I am doing this :

1. Run jenkins in a docker container. Here is the link to docker file.
   https://github.com/PrabhatPandey/Jenkins-docker
   2.Run STF in a docker container.
   3.Occupy all the devices for automation in jenkins container through STF-apis ie. remote connect.
2. Use Spoon runner in Jenkins to create the reports of the automation results.
   "java -jar spoon-runner-1.3.1-jar-with-dependencies.jar
   --apk ExampleApp-debug.apk
   --test-apk ExampleApp-debug-androidTest-unaligned.apk --grant-all"

[trix0]

@PrabhatPandey thank you for you solution I would like to ask how fast are you able to execute tests on real devices ? With Appium it Took me around 1 minute to init() real device. My boss expects around 15 seconds. I cant try it I am out of space on this linux machine 👯‍♂️

[PrabhatPandey]

@trix0 I allocate the devices using open-stf apis. Allocation happens instantly. After allocation is done ie 'adb devices' shows you the list of devices then it's pretty straightforward from there.