ARES is a black-box tool that uses Deep Reinforcement Learning to test and explore Android applications.
ARES does not instrument any application, however supports emma code coverage (for legacy apps) and JaCoCo (to instruement Android apps from source code go to https://github.com/H2SO4T/COSMO)
More details about ARES can be found in the TOSEM paper "Deep Reinforcement Learning for Black-Box Testing of Android Apps" Please use the following bibtex entry to cite our work:
@article{10.1145/3502868,
author = {Romdhana, Andrea and Merlo, Alessio and Ceccato, Mariano and Tonella, Paolo},
title = {Deep Reinforcement Learning for Black-Box Testing of Android Apps}, year = {2021}, publisher = {Association for Computing Machinery},
issn = {1049-331X},
url = {https://doi.org/10.1145/3502868}, doi = {10.1145/3502868},
journal = {ACM Trans. Softw. Eng. Methodol.},
keywords = {Deep reinforcement learning, Android testing}}
- Android emulator or Android smartphone (more stable)
- MacOS or Ubuntu or Windows
- Python 3.7+
- Android from 6.0 to 12.0
- OpenAI Gym
- Install Appium from http://appium.io/docs/en/about-appium/getting-started/; please make sure to set up the environment variables. $ANDROID_HOME and $JAVA_HOME
- Use appium-doctor to check that everything is working correctly
- Create a virtualenv named
venvin folder ARES (not in rl_interaction):virtualenv -p python3 venvand source itsource venv/bin/activate - Go to rl_interaction:
cd rl_interaction - Install the requirements
requirements.txtusing the commandpip3 install -r requirements.txt
- Export PYTHONPATH:
export PYTHONPATH="path/to/ares" - Move yourself to
ares/rl_interaction - Generate a folder for the apks, and put them inside
- Activate the venv
- Start testing
- Create one or more Android emulators.
- Run the tests using
parallel_exec.py:python3 parallel_exec.py --instr_jacoco --list_devices "avd-name0 avd-name1 ..." --appium_ports "4270 4277 ..." --android_ports "5554 5556 ..." --path "apps" --timer 60 --rotation --internet --emu headless --platform_version 8.1 --iterations 10 --algo SAC --timesteps 4000 --trials_per_app 3
The flag --instr_jacoco is not useful if you are not interested in code coverage.
You don't need to boot the emulators; ARES will do it for you.
To see the avd names of your emulators, you can run emulator -list-avds.
There is no need to specify udids using emulated devices since ARES will manage them automatically.
Read Available Flags for more information.
- Buy one or more Android devices.
- Activate ADB and usb debug.
- Run the testing using
parallel_exec.py:python3 parallel_exec.py --instr_jacoco --real_device --udids "HG*****9 PO********NA" --list_device "lenovo1 levecchio2" --appium_ports "4270 4290" --android_ports "5554 5556" --path "apps" --timer 70 --rotation --internet --platform_version 7.0 --iterations 2 --algo SAC --timesteps 5000 --trials_per_app 3
With real devices the flag --list_devices can contain arbitrary names, while the flag --udids must contain
the real udids of your devices.
You can find the udids using the command adb devices
Flag --real_device is required.
--instr_emma, If you want to collect code coverage with EMMA.--instr_jacoco, If you want to collect code coverage using JaCoCo.--save_policy, You can save an exploration policy of your app and use it in new explorations.--reload_policy, Tell ARES to reload a previous policy.--real_device, If you are using a real device you must specify it.--timer: [time_in_minutes], You can specify the time to test the app, required=True.--platform_version [android_version], You have to specify the android version, default = 10.0 .--iterations [number_of_iterations], How many times you want to repeat the test, default=10 .--algo [algo], Choose one between SAC random and Q-Learning (SAC is the algorithm used in the paper).--timesteps, Number of time steps of each testing, (--timerhas higher priority ), required=True.--rotation, If you want to enable rotation.--internet, If you want to toggle data during testing.--emu, If you are using an emulator, you need to specify in what mode (normal or headless).--max_timesteps, You can specify the duration of an episode, default = 250 .--pool_strings, Name of the file to pick the strings from, default = strings.txt .--list_devices [emulators], A list of the device_names (avd_names), required = True.--appium_ports [ports], A list of the ports you want to use, required = True.--android_ports [ports], A list of the adb-ports you want to use, required = True.--udids [strings], A list of the udids of the real devices, in case you are using emulators don't use this flag (ARES will assign udids for you).--trials_per_app, How many times ARES attempts to launch an app.--path [folders],The folder containing all apks, ARES will equally subdivide the apps between the devices available.
During the testing phase several files are generated:
- In a folder named
coverageyou will find all .ec files associated to each app tested organized by algorithm and number of executions. - In a folder named
logsyou will find the trace of the entire execution (the time, the activity and the operation generated), it is useful to recreate a bug or a specific set of actions. You will also find all the stack traces associated to the generated bugs. - In a folder named
policiesyou will find the policies saved by ARES of your apps.
To automatically instrument apps from source code, you can use COSMO: https://github.com/H2SO4T/COSMO
FATE: https://github.com/H2SO4T/FATE
We strongly suggest using Android 8.1 (to the best of our knowledge, the most stable). Google emulators are not meant to run for multiple days, and they can have unexpected behaviors due to this. ARES integrates many protection systems that save the current session and restart the emulators. However, sometimes the emulators are irremediably broken, and you need to delete and recreate them.
Connection errors usually depend on the emulator, using a real devices can reduce these errors
- At first try to delete and recreate the emulator
- If it does not work, try to reinstall the packages
seleniumandAppium-Python-Client
If you are interested in the project, please feel free to suggest new features!
Using ARES on Apple Silicon is possible, but the configuration is a bit longer. The configuration steps are identical, except for the Python part.
We tested this procedure on a Mac Mini 16Gb, Python3.9 and a real device.
- At first install
minicondafrom https://github.com/conda-forge/miniforge. - Create a venv using
conda create --name venvand activate itconda activate venv. - Run
conda install pandas,conda install numpyandconda install scipy, andconda install pyyaml,conda install typing_extensions.
- Run
brew install openblas - Then clone pytorch:
git clone --recursive https://github.com/pytorch/pytorch cd pytorch- run
python setup.py build - run
python setup.py develop - Install the missing packages using pip:
stable_baselines3,loguru==0.5.0,androguard==3.3.5,Appium-Python-Client==1.0.2,cloudpickle==1.2.2,future==0.18.2andgym==0.18.0 - At last, modify il
parallel_exec.pyat line 99: insert a string with the path to the venv python (usewhich pythonwhen the venv is activated)
Now the environt is ready!
- Use the flags
save-policyandreload-policyto save or reload previous policies in SAC. The policies are saved under the folderpolicieswith the same name of the apk file in apps folder. WARNING: Ifsave-policyis True, then at the end of the testing the previous policy will be overwritten.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent