rhapsod/sapienz

A Prototype of Sapienz (Out-of-date and no longer supported)

Sapienz Prototype (Out-of-date)

Sapienz: Multi-objective Automated Testing for Android Applications

This is a prototype of the Sapienz approach for multi-objective automated testing for Android applications. Please note that this repository is no longer maintained.

Publication:

@InProceedings{mao:sapienz:16,
    author = "Ke Mao and Mark Harman and Yue Jia",
    title = "Sapienz: Multi-objective Automated Testing for {Android} Applications",
    booktitle = "Proc. of ISSTA'16",
    year = "2016", 
    pages = {94--105}
} 

Installation

Simply download the artefact here and apply the following environment configuration:

Environment Configration

• Python: 2.7

• Android SDK: API 19

• Linux: sudo apt-get install libfreetype6-dev libxml2-dev libxslt1-dev python-dev

• Mac OS: brew install coreutils for gtimeout

Install Python dependencies:

sudo pip install -r requirements.txt

Usage

python main.py <apk_path | source_folder_path>

where apk_path is path to the subject apk under test
or you can specify source_folder_path for the subject app with source code

Subject Requirement:

• instrumented apk should be compiled and named with suffix "-debug.apk"
• closed-source/non-instrumented apk name should end with ".apk"

Settings

• ANDROID_HOME and WORKING_DIR in settings.py should be set before starting Sapeinz.

Output

• for open-sourced apps, outputs are stored under the given source folder
• for closed-sourced apps, output are stored under <apk_file_path>_output

Output content:

/coverages - Coverage reports are stored here

/crashes - Crash reports and corresponding test cases that lead to the crashes 
(and also recorded videos files when using real devices)

/intermediate - Generated test event sequences for each generation; logbook of the genetic evolution; 
and line charts showing the variation trend for each objectives.

Notes

• This implementation has been tested with Android 4.4, running on Ubuntu 14.04 and Mac OS 10.10
• If measure statement coverage for open-sourced apps, the subjects need to be processed to support EMMA instrumentation: (Please refer to Dynodroid https://code.google.com/archive/p/dyno-droid/)
• This version is only for emulators running on a physical PC.

Contact

[email protected]

sapienz's Issues

Missing python dependencies

When running sapienz against an open-source app, it throws the following errors:

Traceback (most recent call last):
  File "main.py", line 18, in <module>
    from plot import two_d_line
  File "/home/vagrant/issta16_artefact/plot/two_d_line.py", line 6, in <module>
    import matplotlib.pyplot as plt
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_tkagg.py", line 6, in <module>
    from six.moves import tkinter as Tk
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 203, in load_module
    mod = mod._resolve()
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 82, in _import_module
    __import__(name)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 42, in <module>
    raise ImportError, str(msg) + ', please install the python-tk package'
ImportError: No module named _tkinter, please install the python-tk package

Can you let me know how to resolve this issue?

Thanks!

Can 5.0 above supported?

It seems that the motifcore.jar in my nexus5 with 5.0.1 Lolipop cannot properly run.

The printout shows that monkey Switch action has been generated successfully, but the action is not executed properly.

:Switch: #Intent;action=android.intent.action.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10200000;component=org.jtb.alogcat/.LogActivity;end
:Sending rotation degree=0, persist=false

A successful switch action should be as follows:

:Switch: #Intent;action=android.intent.action.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10200000;component=org.jtb.alogcat/.LogActivity;end
    // Allowing start of Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=org.jtb.alogcat/.LogActivity } in package org.jtb.alogcat

Do you know where the problem is ?

brut.androlib.err.UndefinedResObject: resource spec: 0x01010543

Exception in thread "main" brut.androlib.err.UndefinedResObject: resource spec: 0x01010543
at brut.androlib.res.data.ResPackage.getResSpec(ResPackage.java:59)
at brut.androlib.res.data.ResTable.getResSpec(ResTable.java:65)
at brut.androlib.res.data.ResTable.getResSpec(ResTable.java:61)
at brut.androlib.res.data.value.ResReferenceValue.getReferent(ResReferenceValue.java:57)
at brut.androlib.res.data.value.ResReferenceValue.encodeAsResXml(ResReferenceValue.java:47)
at brut.androlib.res.data.value.ResScalarValue.encodeAsResXmlValue(ResScalarValue.java:58)
at brut.androlib.res.data.value.ResStyleValue.serializeToResValuesXml(ResStyleValue.java:71)
at brut.androlib.res.AndrolibResources.generateValuesFile(AndrolibResources.java:502)
at brut.androlib.res.AndrolibResources.decode(AndrolibResources.java:252)
at brut.androlib.Androlib.decodeResourcesFull(Androlib.java:135)
at brut.androlib.ApkDecoder.decode(ApkDecoder.java:102)
at brut.apktool.Main.cmdDecode(Main.java:165)
at brut.apktool.Main.main(Main.java:81)
rm failed for /mnt/sdcard/_strings.xml, No such file or directory

Motifcore files not found after copy

Motifcore files weren't copied over to the emulator by motifcore_installer.install(). Removing the read-only remount at the end of that function fixed the problem for me, see PR #3 for a patch.

AssertionError

Hello,

I'm trying to run Sapienz on Ubuntu 16.04 and using the aard2 as software under test (https://github.com/itkach/aard2-android/releases)

I made all the configurations showed in the tutorial, but when I try to run the Sapienz, this message is displayed:

=================

Running on Linux
/bin/sh: 1: aapt: not found
mkdir: it is impossible to create directory “.../aard2-android-0.38.apk_output”: File or directory not found

Working on apk:

Preparing devices ...
Booting Device: api19_0
Waiting 30 seconds

Traceback (most recent call last):
File "main.py", line 380, in
main(app_dir)
File "main.py", line 282, in main
emulator.prepare_motifcore()
File ".../sapienz/devices/emulator.py", line 97, in prepare_motifcore
for device in get_devices():
File ".../sapienz/devices/emulator.py", line 63, in get_devices
assert len(ret) > 0
AssertionError

=================

I'd like to know if anyone had the same error and if anyone know how do I overcome this problem.

Thanks.

device name convention & read-only emulator

HI,
thanks for making this great work public!

Here are a few fixes:
(1) in boot_devices() in emulator.py, you used the following to get the device name.
However, this naming convention may not always be correct.
for i in range(0, settings.DEVICE_NUM):
device_name = settings.AVD_SERIES + str(i)
One simple fix is to let users specify the device_name in settings.py

(2) in boot_devices() in emulator.py, we need the option "-writable-system", otherwise, the mounting of "/system" may fail. Also, please remove "-no-audio" since it is no longer a valid option.
sub.Popen('emulator -avd ' + device_name + " -wipe-data -writable-system",
stdout=sub.PIPE, stderr=sub.PIPE, shell=True)

/mnt/sdcard/motifcore.script no such file

After running main.py,the program is stuck here.And I found /mnt folder is empty.

Path:/mnt/sdcard/motifcore.script

:Monkey: seed=1561307463543 count=21
:AllowPackage: --ignore-crashes
:IncludeCategory: android.intent.category.LAUNCHER
:IncludeCategory: android.intent.category.MONKEY

Number of Activities in mainApps: 22

Number of Activities in mainApps: 3

** No activities found to run, monkey aborted.
... Finish generating a sequence
1 KB/s (52 bytes in 0.045s)
rm failed for /mnt/sdcard/bugreport.crash, No such file or directory

How can I solve this problem and how is motifcore.script generated @Rhapsod

ValueError: Number of processes must be at least 1

Execution halts right after "### Initialising population ...." and gives this:
Traceback (most recent call last): File "main.py", line 380, in <module> main(app_dir) File "main.py", line 323, in main package_name=package_name) File "/Users/account/Favorite/sapienz/init/initRepeatParallel.py", line 69, in initPop pool = mp.Pool(processes=len(idle_devices)) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool return Pool(processes, initializer, initargs, maxtasksperchild) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 152, in __init__ raise ValueError("Number of processes must be at least 1") ValueError: Number of processes must be at least 1

Anyone have any idea how to solve this?

rm failed for /mnt/sdcard/_strings.xml, No such file or directory

List of devices attached
emulator-5554 device
370fe160 device

I: Using Apktool 2.0.2 on CJmall.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /root/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values / XMLs...
I: Baksmaling classes.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...
rm failed for /mnt/sdcard/_strings.xml, No such file or directory
370 KB/s (38880 bytes in 0.102s)
rm failed for /mnt/sdcard/bugreport.crash, No such file or directory

java.lang.exception

Hi,
when i run the tools, the system print the below info:
// 10: 3.0% // 11: 10.0% :Switch: #Intent;action=android.intent.action.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10200000;component=com.example.android.notepad/.NotesList;end :Sending rotation degree=0, persist=false java.lang.Exception at android.os.Process.killProcess(Process.java:1255) at com.android.internal.os.RuntimeInit$UncaughtHandler.uncaughtException(RuntimeInit.java:125) at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:693) at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:690) ... Finish generating a sequence
what happened?could you pls help me..

How can users adjust speed of test input generation?

I try to modify the speed in setting.py/MOTIFCORE_SCRIPT_HEADER, but it doen't work.
Is the speed parameter is packaged within motifcore.jar? How can I configure it manually?
Thanks for any suggestion!

** Error: Count is not a number

I was having this issue #3 and was deleted the line for recover permission But now I have this erro. How can fix it?
`... Start generating a sequence

Connecting to Motif Core ...

** Error: Count is not a number
oops.
... Finish generating a sequence
remote object '/storage/sdcard/motifcore.script' does not exist

`