appetizerio/replaykit

[DEPRECATED] Command line tools for recording, replaying and mirroring touchscreen events for Android

replaykit

中文说明

replaykit is a command line tool intended for mobile continuous integration, which provides the ability to:

• record touchscreen events from one device and replay to many (aspect ratios must be the same)
• control multiple devices concurrently (e.g., run a shell command, install an app, etc.)
• mirror the touchscreen events from one device to many in real time (aspect ratios must be the same)

replaykit can work on Windows 7+, MacOS Mavericks+ and Linux, available for Android API 9-27(9.0 aka. P), emulators, VMs (like Genymotion) and real devices. To integrate replaykit with existing CI infrastructure, consider the Python wrapper.

Please submit issues for bug reports, enhancements and feature requests. Pull requests for documentation improvement are welcomed too.

Installation

The executable file can be found in the darwin/ for MacOS, win32/ for Windows and linux/ for Ubuntu, Fedora and CentOS. You might want to add to PATH to use the tool globally. Linux and MacOS build are both 64-bit and Windows build is 32-bit (compatible with 64-bit as WOW). replaykit requires adb to be present in PATH. You can install the Android SDK and ensure <sdk>/platform-tools is in your PATH.

Control Multiple Devices

Suppose you have several devices attached to the development machine via USB and you want them to perform the same action.

First, make sure replaykit recognizes all of them devices:

./darwin/replaykit devices list

You should see a JSON output. Note the serialno field, which is the same as the adb devices output. A serialno uniquely identifies a connected device.

Next, you can try these:

DEVICE=serialno1,serialno2 # device serialnos, comma separated, no whitespace!
replaykit devices control $DEVICE uninstall com.helloworld # uninstall an app on those devices
replaykit devices control $DEVICE install hellworld.apk # install an app
replaykit devices control $DEVICE kill_all # kill all background apps
replaykit devices control $DEVICE launch_pkg com.helloworld # launch an app
replaykit devices control $DEVICE shell input keyevent KEYCODE_HOME # simulate the home button
replaykit devices control $DEVICE shell pm grant com.helloworld android.permission.SOME_PERMISSION  # grant a permission to an app 
replaykit devices control $DEVICE shell "input keyboard text 'Paste%stext%son%sAndroid%sDevice'"  # input text (pre Android 6)
replaykit devices control $DEVICE shell "input keyboard text 'Paste text on Android Device'"  # input text (Android 6+)
replaykit devices control $DEVICE shell screenrecord /sdcard/demo.mp4  # record screen into a video（Android 4.4+）

Basically everything you did with adb shell can now be replaced with replaykit devices control serialno1,serialno2 shell to perform that on multiple devices simultaneously.

And, just in case you need them:

• Key code cheatsheet
• Common ADB shell commands

Record and Replay touchscreen events

Replaykit can record touchscreen events from one device and replay back to many to automate stuff.

replaykit trace record --device serailno mytrace.trace

Now find the device and start touching. The touchsceen events will be captured and saved to mytrace.trace. If you are done, type in exit to stop. You are all set if seeing "Recorder stopped, trace saved to mytrace.trace". Try not to use Ctrl+C.

Check if the trace is good:

replaykit trace info mytrace.trace

Again, it shows a JSON about the recorded trace, showing the screen resolution, time length, how many fingers, etc. To modify the description of this trace, use replaykit trace describe mytrace.trace "the description to be added".

Next replay the trace back to see how it goes:

replaykit trace replay mytrace.trace serialno

You will see something like the video below:

If you want to replay to more than one devices, simply replace serialno with a comma-separated serialno list like serialno1, serialno2.

Mirror Touchscreen Events

With replaykit, you can mirror the touchscreen events from one device to other devices in real time.

replaykit devices mirror <from_device> <to_device>

The <from_device> is a single serialno while <to_device> can be a comma-separated list of device serialnos. Make sure all involving devices have the same aspect ratio. Otherwise the command will fail. For devices with the same aspect ratio, the touchscreen events will automatically scale to adapt to the screen resolution.

Here is a demo:

ADB related

Finally, replaykit provides some commands to check the ADB environment.

replaykit adb start-server  # start the ADB server
replaykit adb kill-server  # kill the ADB server
replaykit adb detectadb  # this tells where the adb binary locates. If not found, tune your PATH
replaykit adb check-server  # check if the adb server is running properly now

License

Apache License v2

replaykit's Issues

Mirroring over Wifi

This requires adbclient to work fully with the LAN and some basic network detection shall be added.

modify records.

Can I modify the recorded x,y parameters after record and then replay ?

华为为主设备镜像操作问题

镜像操作主设备是华为的时候，子设备每一个动作结束后都保持按着屏幕不放的状态

Support nox player?

I have an issue running this on Nox app player.

Record replaykit trace record --device serailno mytrace.trace works fine,

But when I try to replay the recorded file back, it isn't replaying it right.
Kindly check the video here: Video

I recorded my character to run in a circle and tried to replay it back.
But when I replayed the recorded file - it seems to click the top of the screen.
You can see the android settings bar at the top highlighted.

Seems like the mapping location between the screen and the touch is off for nox?

I tried out different resolution and aspect ratio. But I get the same result.
Nox app player uses Android 5 (API 21)

Used WIN32 version 1.0.8
And windows 10.

Many thanks in advance!

Crash in main thread on trace record

Me again.

On Ubuntu, when I attempt to record a trace, I get an error. Specifically,

When I issue this command:
./appetizer trace record --device abc123 mytrace.trace

I get this error:
]Crash in main_thread Exception AttributeError: "GeteventReader instance has no attribute 'proc'" in <bound method GeteventReader.__del__ of <droidrep.GeteventTools.GeteventReader instance at 0x7f5f8af28098>> ignored

I should mention the context here. I am trying to run the appetizer executable without it on my PATH. Also, I am using the adb standalone program (available here), not the Android SDK, which is in the same directory and also not on the PATH.

This error does not seem to exist on Windows and I do not know about Mac yet, will update. Help is appreciated.

Mac 10.10 Yosemite兼容性问题

code signature invalid
pyinstaller/pyinstaller#2564

Android 11 crash in main_thread on replay

Hey,

I'm trying to run this on my Pixel 4 running Android 11 and I'm running into an Crash in main_thread error when trying to replay.

Here is the full log of what happened

theo@mbp ~/Desktop/replaykit (master) $ ./darwin/replaykit devices list
[{"uid": "99181FFAZ005BC", "serialno": "99181FFAZ005BC", "brand": "google", "screenheight": 2280, "device": "flame", "manufacturer": "Google", "sdk": "30", "name": "flame", "heapsize": "512m", "model": "Pixel 4", "screenwidth": 1080, "os": "11"}]
theo@mbp ~/Desktop/replaykit (master) $ ./darwin/replaykit trace record --device 99181FFAZ005BC mytrace.trace
appetizer> /var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.679207, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 0)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.750403, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 2)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.767937, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 3)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.785726, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 4)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.803497, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 5)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.866235, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 4)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.875182, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 5)
/var/folders/ff/tdk78rk553zdd46dcgsnd_wh0000gn/T/_MEI0Vqc0M/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(21707.893104, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 4)
exit
Current version is 1.0.8, latest version is :1.0.9, please update
theo@mbp ~/Desktop/replaykit (master) $ ./darwin/replaykit trace info mytrace.trace
{"contacts": 1, "description": "", "height": 2280, "length": 10.942165000000386, "records": 199, "valid": true, "width": 1080}
theo@mbp ~/Desktop/replaykit (master) $ ./darwin/replaykit trace replay mytrace.trace 99181FFAZ005BC
Crash in main_thread
Current version is 1.0.8, latest version is :1.0.9, please update

Any ideas ?

Cheers! :)

Not working for Android x86 VirtualBox cases: need to locate emulated touchscreen device

Hi,

Getting the error on running the trace command.

F:\www\android\win32>adb devices
List of devices attached
192.168.1.6:5555        device

F:\www\android\win32>appetizer devices list
[{"uid": "192.168.1.6:5555", "serialno": "192.168.1.6:5555", "brand": "Android-x86", "screenheight": 600, "device": "x86", "manufacturer": "innotek GmbH", "sdk": "19", "name": "android_x86", "heapsize": "512m", "model": "VirtualBox", "screenwidth": 800, "os": "4.4.4"}]

F:\www\android\win32>appetizer trace record --device 192.168.1.6:5555 mytrace.trace
The device does not have proper permissions for getevent

F:\www\android\win32>adb shell getevent -l
add device 1: /dev/input/event7
  name:     "Android Power Button"
add device 2: /dev/input/event1
  name:     "Sleep Button"
add device 3: /dev/input/event0
  name:     "Power Button"
add device 4: /dev/input/event2
  name:     "Video Bus"
add device 5: /dev/input/event4
  name:     "PC Speaker"
could not get driver version for /dev/input/mouse1, Not a typewriter
add device 6: /dev/input/event6
  name:     "ImExPS/2 Generic Explorer Mouse"
add device 7: /dev/input/event5
  name:     "AT Translated Set 2 keyboard"
could not get driver version for /dev/input/mice, Not a typewriter
could not get driver version for /dev/input/mouse0, Not a typewriter
add device 8: /dev/input/event3
  name:     "VirtualBox USB Tablet"

OS: Windows 8.1
adb in C:\adb\platform-tools

How to start replaykit on another port of adb

I would like to run both stf and replaykit, can replaykit run on a separate port ex: 5038
adb start-server -P 5038

Record and playback not working in Moto phone

The command "appetizer trace info mytrace.trace" returns {"contacts": 0, "description": "", "height": 1280, "length": 0.0, "records": 0, "valid": true, "width": 720}

Though, it works on samsung mobiles!

Logo Design

Hello!

I'm a graphic designer and I like to support open source projects. I would like to design a logo for your project if you interested, I will be happy to work with you! :)

Best Regards

Baran Pirinçal

Graphic Designer

是否对Genymotion模拟器的输入框定位和支持有问题？

用的Genymotion模拟器 ，模拟镜像Samsung Galaxy S6 -6.0.0-API23 为。
操作一APP 到注册页面，然后输入用户名和密码。录制过程中，报出来了一些警告。

再回放时候，发现到注册页面，就中止了，没有进行输入用户名和密码操作。
`C:\Users\cmd>appetizer trace info sg6.trace
{"contacts": 1, "description": "", "height": 800, "length": 14.967434000000026, "records": 48, "valid": true, "width": 480}

C:\Users\cmd>appetizer trace replay -h
usage: appetizer trace replay [-h] [--port PORT] [--host HOST]
trace_file device_list

positional arguments:
trace_file Path to the trace file
device_list Comma-separated device serial numbers

optional arguments:
-h, --help show this help message and exit
--port PORT, -P PORT ADB server port
--host HOST, -H HOST ADB server host

C:\Users\cmd>appetizer trace replay sg6.trace 192.168.60.101:5555
appetizer>`

回放并没有给什么日志 放完就自动退出命令行了。

是不是 对Genymotion模拟器的输入框定位有问题 。 哦 模拟器都是用鼠标和键盘操作的，是不是应该用真机USB链接 更好些？

Device hangs when running appetizer trace replay

Hi,
I record the trace on system A and when I run replay on system B, the device hangs, I have to reboot the device manaully, because the device appears offline when I run adb devices

Can you suggest what is wrong?

Source code

Hello,
where can I find the source code of the binaries pushed on this repo?

Thanks!

Android10 回放崩溃

执行回放 提示
crash main thread

replay 不响应.

hyadeiMac:darwin hya$ appetizer trace replay test.trace 0
preparing daemons for 0 devices
appetizer> Replay started
Replay finished
All done

我用这个方法重放录屏触摸，但是手机上没有响应，而且重放后终端无法输入，只能按control+d退出，使用的是mac版本。
启动好像也有错误:

hyadeiMac:~ hya$ appetizer -h
usage: appetizer [-h] {version,trace,adb,devices,plan} ...

positional arguments:
  {version,trace,adb,devices,plan}
                        commands
    version             Print the version information and exit
    trace               Record and replay touchscreen events
    adb                 Control the local ADB
    devices             Command devices
    plan                Compose and execute a test plan file

optional arguments:
  -h, --help            show this help message and exit
Failed to execute script starfish

Replaykit Breaks console on Unix Systems

When using replaykit on Ubuntu or Mac,

Frequently, playing a trace with the command ./appetizer trace replay mytrace.trace $myPhone breaks the console session. Nothing further prints to the console and input is no longer accepted. The trace plays correctly, but the console needs to be restarted, losing session data.

Has this been observed by others? Is it just me?

镜像操作在华为上出现动作延迟或者一个手势需要下一个手势开始后才能镜像

使用的版本：Appetizer1.1.1
使用的设备：
Huawei Mate9、Huawei Mate8，两个都是16:9的比例。
无论主设备是Mate9还是Mate8，镜像操作感觉都不是特别灵敏。

主要的现象如下：
1、如果是主设备按住屏幕，上下滑动的时候，从设备基本还是能够跟上操作，不过还是存在一些延迟。
2、如果是主设备上进行点击操作，从设备会有很大延迟，甚至很长时间从设备都接收不到点击事件。比如点击Mate9桌面左下角的的一个图标，Mate8相同位置的图标没有被点击。
3、快速滑动的体验不是很好，比如我在Mate9的桌面向左滑动，Mate8就滑不过去，界面经常卡在一半不动。

关于第2点，后经过观察，好像是主设备点击之后，如果再进行一些操作，那么从设备上才会收到这个点击事件。如主设备点击返回键，此时从设备没反应，如果再在主设备上进行一些操作，那么从设备就会收到点击事件了。对于点击事件，感觉有遗漏。

镜像操作是个很新颖的功能，想法很棒，建议看下有没优化空间？

record getevent as input for appetizer

Hi,
I would like to record adb shell getevents and pass that as input to appetizer. If yes how do I pass it as input ?

Thanks
Dexter

Can we do it wireless ?

can it be used by wifi ?

可执行文件的实现源码没有开源么？

如题，可执行文件的实现源码没有开源么？

录制报错Record command report error

1.首次运行如上报错。exit后在录制一遍，又正常了。

2.如上。查看录制的情况的命令报错
3.此外发现录制的最后一个命令，如果是点击，有可能被吃掉，就是replay的时候没有执行，exit太早了？或者最后一个点击操作会变成长按，等一会再exit就会如此。

Cannot Start Trepn Profiler with replaykit

Hi,

I am trying to use replaykit to trigger Trepn Profiler to start profiling. Generally, ADB shell commands work fine. Like this one (my replaykit executable is called starfish):

starfish devices control $target shell input keyevent KEYCODE_WAKEUP

However, this command does not work:

starfish devices control $target shell am broadcast -a com.quicinc.trepn.start_profiling -e com.quicinc.trepn.database_file

And gives me this error:

usage: starfish.exe [-h] [-v VERBOSE] {version,update,trace,adb,devices,plan,insight} ... starfish.exe: error: unrecognized arguments: -a com.quicinc.trepn.start_profiling -e com.quicinc.trepn.database_file trepnTemp

But this command works just fine over adb:
adb shell am broadcast -a com.quicinc.trepn.start_profiling -e com.quicinc.trepn.database_file "trepnTemp"

This has been tested and confirmed a problem on Windows and Mac. We can work around this problem by looping through our target list and calling adb commands in stead but this is ugly and time consuming.

Thank you for spending your time on this helpful tool. It has been of use in our research.

Replay is not precise: pixel difference exists

This can be reproduced as follows:

• Open Google Maps
• Set some kind of reference on the map
• Record yourself swipe/move around the map and then going back to the reference
• Go back to the reference
• Replay the actions

What you'll find is that the position at the end of the replay is not always the same. In other words, replaying doesn't always yield the same results.

source code

You said earlier that we can see the source code. It would be possible? I'd be grateful for.

Why replaykey ask firewall to access internet ?

如何为trace文件添加描述？

使用appetizer trace info mytrace.trace来查看trace文件，发现describe为空，但是也没有找到相关的指令来输入描述；请问有相关指令吗？谢谢！

Error replaying on Android8 for Oneplus and Samsung S8

Hi
Great project, but I have some bugs with certain devices,
not all events are recorded, and I'm getting this error in stderr:

[stderr_data]:  /var/folders/t7/wykzbl_x2gqb4yx6l_fz04zh0000gn/T/_MEIBUqnr0/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(989318.934083, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 10)
/var/folders/t7/wykzbl_x2gqb4yx6l_fz04zh0000gn/T/_MEIBUqnr0/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(989318.942262, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 8)

[stderr_data]:  /var/folders/t7/wykzbl_x2gqb4yx6l_fz04zh0000gn/T/_MEIBUqnr0/droidrep/GeteventTools.py:179: UserWarning: TypeB MT meets unknown evCmd(989318.949994, 'EV_KEY', 'BTN_TOOL_FINGER', 0)

[stderr_data]:  /var/folders/t7/wykzbl_x2gqb4yx6l_fz04zh0000gn/T/_MEIBUqnr0/droidrep/GeteventTools.py:179: UserWarning: TypeB MT meets unknown evCmd(989319.074845, 'EV_KEY', 'BTN_TOOL_FINGER', 1)

[stderr_data]:  /var/folders/t7/wykzbl_x2gqb4yx6l_fz04zh0000gn/T/_MEIBUqnr0/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(989319.074845, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 11)

[stderr_data]:  /var/folders/t7/wykzbl_x2gqb4yx6l_fz04zh0000gn/T/_MEIBUqnr0/droidrep/GeteventTools.py:172: UserWarning: Type B MT meets unknown evCmd(989319.099495, 'EV_ABS', 'ABS_MT_TOUCH_MINOR', 10)

the side effect is that when pressing the same button lets say: 10 times
only the first 1-2-3 are recorded.

Once the above error comes, not all events are recorded.

Happens mainly on devices having android >= 8
such as OnePlus 6 / Xiaomi Mi 5 / etc....

On the Xiaomi when trying to record I get:

[stderr_data]:  The Linux device file for touchscreen cannot be auto-detected. The device might not have proper permissions for getevent

When trying to play getting:

[stdout_line]:  Crash in main_thread

How to identify application activity or element id on click

I would like to know which element or button was clicked, I require this for a custom application.
when I execute the command

./appetizer trace info mytrace.trace

I get this result {"contacts": 1, "description": "", "height": 1280, "length": 5.08226399999694, "records": 4, "valid": true, "width": 720}
Is it possible to capture/display the element id or application activity that was launched ?