project-artist/meta

Meta repository for matters related to more than one repository or even the whole project as such.

meta's Issues

Versioning

We currently do not employ any kind of versioning in our projects, but having projects that are not up to date easily leads to incompatibilities, in particular since we are heavily working on all parts. Also, figuring out later which versions in the projects matched at an earlier point in time boils down to comparing commit times, which is a mess.

Introducing proper versioning can solve this problem. It is important to have a versioning scheme that reflects the compatibility between the main parts. I am thinking along the lines of semantic versioning, for example same major and minor version imply perfect compatibility where the patch level only applies to non-breaking changes such as, e.g., small fixes. Finding our what exactly can be derived from, e.g., common major and diverging minor or other combinations is part of this task.

This in general requires some careful consideration since connecting and unifying releases of so many projects with different scopes is an involved task.

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• other

Arena allocation crashes

Unfortunately, in some cases our modules crash because of what I believe to be inconsistent memory management on our side. More precisely, the trace module for example crashes on x86 if its artist pass is created using a placement new operator with the current arena allocator. The following quick-fix solves the problems for now for trace:

HArtist *TraceModule::createPass(const MethodInfo &method_info) const {
//  return new(method_info.GetGraph()->GetArena()) HTraceArtist(method_info);
  return new HTraceArtist(method_info);
}

From what we learned from the ART code, classes prefixed with an H are meant to be arena allocated. However, artist passes extend HOptimizations and therefore should be treated accordingly. But actually using arena allocation there triggers weird crashes such as this one here untangled by ndk-stack from a trace crash without the patch above:

********** Crash dump: **********
  29   │ Build fingerprint: 'Android/sdk_phone_x86/generic_x86:7.1.1/NYC/4931657:userdebug/test-keys'
  30   │ pid: 13209, tid: 13209, name: main  >>> /data/user/0/saarland.cispa.artist.artistgui/files/artist/dex2oat <<<
  31   │ signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x4
  32   │ Stack frame #00 pc 001f0345  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine HOptimization at /proc/self/cwd/art/compiler/optimizing/optimization.h:34
  33   │ Stack frame #01 pc 001f1bb1  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::OptimizingCompiler::Compile(art::DexFile::CodeItem const*, unsigned int, art::InvokeType, unsigned short, unsigned int, _jobject*, art::DexFile const&,
       │  art::Handle<art::mirror::DexCache>) const at /proc/self/cwd/art/compiler/optimizing/optimizing_compiler.cc:857
  34   │ Stack frame #02 pc 0014ab2c  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompileMethod(art::Thread*, art::CompilerDriver*, art::DexFile::CodeItem const*, unsigned int, art::InvokeType, unsigned short, unsigned int, _jobject*
       │ , art::DexFile const&, art::optimizer::DexToDexCompilationLevel, bool, art::Handle<art::mirror::DexCache>) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:649
  35   │ Stack frame #03 pc 0015e19c  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompileClassVisitor::Visit(unsigned int) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:2710
  36   │ Stack frame #04 pc 001595fe  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::ParallelCompilationManager::ForAllClosure::Run(art::Thread*) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:1957
  37   │ Stack frame #05 pc 0055e360  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart.so: Routine art::ThreadPool::Wait(art::Thread*, bool, bool) at /proc/self/cwd/art/runtime/thread_pool.cc:215
  38   │ Stack frame #06 pc 0015260f  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::ParallelCompilationManager::ForAll(unsigned int, unsigned int, art::CompilationVisitor*, unsigned int) at /proc/self/cwd/art/compiler/driver/compiler_d
       │ river.cc:1933
  39   │ Stack frame #07 pc 00147b3d  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompilerDriver::CompileDexFile(_jobject*, art::DexFile const&, std::__1::vector<art::DexFile const*, std::__1::allocator<art::DexFile const*> > const&,
       │  art::ThreadPool*, unsigned int, art::TimingLogger*) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:2750 (discriminator 1)
  40   │ Stack frame #08 pc 001463b1  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompilerDriver::CompileAll(_jobject*, std::__1::vector<art::DexFile const*, std::__1::allocator<art::DexFile const*> > const&, art::TimingLogger*) at /
       │ proc/self/cwd/art/compiler/driver/compiler_driver.cc:506
  41   │ Stack frame #09 pc 000251d9  /data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat: Routine art::Dex2Oat::Compile() at /proc/self/cwd/art/dex2oat/dex2oat.cc:1710 (discriminator 1)
  42   │ Stack frame #10 pc 0000b4eb  /data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat: Routine art::CompileApp(art::Dex2Oat&) at /proc/self/cwd/art/dex2oat/dex2oat.cc:2725
  43   │ Stack frame #11 pc 0001532c  /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libc.so: Routine __libc_init at /proc/self/cwd/bionic/libc/bionic/libc_init_dynamic.cpp:109
  44   │ Stack frame #12 pc 0000a272  /data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat: Routine _start at art/dex2oat/dex2oat.cc:?
  45   │ Stack frame #13 pc 0000000d  <unknown>

So what can we do? I assume that we have some inconsistency in where we use arena allocation and where we use other means of memory management (e.g., smart pointers). Fixing that might solve the problem. For example, there is an arena version of a vector and I am pretty sure we use regular vectors to store H-prefixed objects, such as HInstructions, but this is just a guess.

Anyway, at some point we need to better understand the arena allocation implementation and make sure we use it correctly within artist and the modules to avoid crashes.

Affected Projects

• ARTist
• art
• template-module (actually, any module)

Remove ARTist as art's submodule and place ARTist in local_manifest

Remove ARTist as art's submodule and place ARTist in local_manifest

This allows us to use repo to efficiently switch branches and manage the remotes and our sourcecode.

API for controlling modules from outside the compiler

Currently, the artist modules used are determined at compile-time by hardwiring them in optimizing_compiler.cc.

A better approach is to allow for activation and deactivation without requiring recompilation of ARTist by, e.g., controlling modules through either configurations (files?) or command line arguments to the compiler.

Depends on #1.

(Moved here from Project-ARTist/ARTist#4)

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• other

Module: Frida

Frida is a well-known dynamic hooking framework that also supports Android. There are two possibilities: Either run the frida server on your device so that you can attach to all processes (including apps) or add the frida gadget library into a particular app/binary/process so that only this one is exposed.
Since the latter either required repackaging apps, which breaks the signature, or adding this in a system-centric manner, which would require OS changes, ARTist might provide an alternative deployment path. By making sure the gadget library is loaded in a target process, we can selectively expose apps without requiring the big server (which people often try to detect as part of their anti-debugging strategy) and by avoiding the downsides of the gadget approach.

Affected Projects

• new project: frida-module

Proper READMEs

README.md files are the landing pages of repositories, hence these are the first thing new users see when they visit one of our repositories. Currently, however, those READMEs are not very helpful or even (almost) non-existent. Since we want to make it easier for new users to join our project and work with ARTist, we should have helpful README files that provide some basic information and point at further resources where they can find help. Beside information about the actual repository, it should also give a basic introduction into the overall ARTist project and put the repository in context. Also, a pointer to the documentation and Gitter group would help.

Tasklist

Check a task as soon as a proper README.md file has been committed in the corresponding project.

• meta
• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen

Enabling and disabling Modules

Implement enabling and disabling functionality for modules so that, e.g., modules are simply disabled if preconditions are not met, instead of terminating the whole compilation.

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• other

Update READMEs

Since we officially announced the beta, it is time to update all the README files.
This includes the Beta remark in almost all README files, as well as adding READMEs to the new projects, such as the modules and codelibs

Affected Projects

• meta
• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• template-module
• template-codelib
• trace-module
• trace-codelib
• logtimization-module
• module-sdk-gen

Module: method "filter engine"

While our current showcase modules simply run on all methods of an application, some modules aim at a particular set of methods, e.g., from a single class (package name prefix), with
particular parameter or return types, or other method properties.

Currently, modules could achieve this by checking for those properties themselves in the beginning and immediately return/terminate the execution for this method if it does not match the filter. There are 2 problems with this approach:

1. Most of the time, the "filter functionality" can be implemented by means of a simple whitelist or blacklist. Repeating this over and over for different modules leads to code duplication and increased risk of introducing bugs.
2. it is very inefficient to construct all the objects just to abort immediately. A superior solution checks for those conditions before the module classes are initialized. Remember that this is done for each single method visible to the compiler.

It would be nice to have built-in functionality so that declaring those filters is straightforward. Ready-made implementations for whitelist and blacklist will cover a lot of cases and modules with special requirements can implement their own filters by, e.g., extending them.

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• other

Android 8.0 Oreo Support

Implementation in:

• art-branch: artist_oreo_master
• artist-branch feature_oreo_support.

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous

Android 8.0 Oreo: Untouched base.apk

ARTist's injection only works when replacing the original /data/app/<package>/base.apk with the injected and signed base.apk..

On Android 6, 7 & 7.1 it works, when rewriting / faking the checksums.

Affected Projects

• ARTist
• art
• ArtistGui (potential)
• dexterous (potential)

The only log message indicating the problem is: zygote64: expected 1 dex files but found 2
Then the compiled oat fiel is thrown away and the original non-injected app is executed

Context Logs

10-10 15:37:28.066  1081  1784 I ActivityManager: START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=de.heise.android.heiseonlineapp/.ui.ChannelActivity bnds=[255,245][512,440]} from uid 10035
10-10 15:37:28.069   803  4088 I ACDB-LOADER: ACDB AFE returned = -19
10-10 15:37:28.069   803  4088 D hardware_info: hw_info_append_hw_type : device_name = speaker
10-10 15:37:28.069   803  4088 D audio_hw_primary: enable_snd_device: snd_device(2: speaker)
10-10 15:37:28.069   803  4088 D audio_hw_primary: enable_audio_route: usecase(1) apply and update mixer path: low-latency-playback speaker
10-10 15:37:28.069   872   997 E ANDR-PERF-OPTSHANDLER: Warning: Resource [2, 0] not supported for core 1. Instead use resource for core 0
10-10 15:37:28.069   872   997 E ANDR-PERF-RESOURCEQS: Failed to apply optimization [2, 2, 0]
10-10 15:37:28.101  1081  2426 I ActivityManager: Start proc 17944:de.heise.android.heiseonlineapp/u0a150 for activity de.heise.android.heiseonlineapp/.ui.ChannelActivity
10-10 15:37:28.165 17944 17944 E zygote64: expected 1 dex files but found 2

Error Research

Log(Fatal) terminates here:

bool OatFileAssistant::DexChecksumUpToDate(const OatFile& file, std::string* error_msg) {
  (...)
  // art::OatFile
  uint32_t number_of_dex_files = file.GetOatHeader().GetDexFileCount();
  if (required_dex_checksums->size() != number_of_dex_files) {
  (...)
}
bool OatFileAssistant::DexChecksumUpToDate(const VdexFile& file, std::string* error_msg) {
  (...)
  // art::VdexFile
  uint32_t number_of_dex_files = file.GetHeader().GetNumberOfDexFiles();
  if (required_dex_checksums->size() != number_of_dex_files) {
  (...)
}

Ideas

Experiments with rewriting the dexfile count have been done here:

Project-ARTist/art@e3b1e22 via:

OatHeader::ArtistFixDexFileCount(){ ... }

w/o success. Injected apps didn't even log an error, before ARTist's compiled oat file was thrown away.

Website: Better CodeLib explanation

Better explain what is or what should be done in the CodeLib. We have the CodeLib website, but it mainly focuses on the "How to compile" part.

Module SDK and dynamic loading

We want to be able to write modules without requiring to compile them with/into the compiler and load them dynamically.

One possible approach is to create a module sdk that is used to create and compile modules as regular c++ projects without the need to ever touch the compiler. The idea is that the resulting compiled module registers itself with the compiler so that we can not only control externally which of the modules in the compiler are executed (see Project-ARTist/ARTist#4 ) but actually move the module code out of the compiler and only have module-agnostic code in there.

There are multiple advantages:

1. Developing modules does NOT require building (in the context of) AOSP anymore, which lowers the gap to use artist tremendously.
2. We can load or unload modules from the GUI (see also Project-ARTist/ARTist#4)
3. Modules can be separate projects and do not bloat the ARTist codebase.

(moved here from Project-ARTist/ARTist#5)

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• other: new repo for module sdk, I guess

X86_64 emulator support: ARTist / ARTistGui cannot instrument apps on the Nougat 7.1.1 x86_64 emulator images

ARTist / ARTistGui cannot instrument apps on the Nougat 7.1.1 x86_64 emulator images

Everything works find on the Nougat 7.1.1 x86 images.

Cause:

/data/dalvik-cache/x86_64/ is empty on those images, but dex2oat looks for data/dalvik-cache/x86_64/system@[email protected] and possible more files there.
dex2oat also tries to copy them from /system/framework/x86_64/ but is not allowed to do so.

Things I tried:

• Populating them manually from /system/framework/x86_64/*
  • compilation failed anyways and the newly copied files got deleted from /data/dalvik-cache/x86_64/ again, when executing dex2oat.
• Forcing dex2oat to use --instruction-set=x86 leads to a working compilation/instrumentation process, but the instrumented oat-file won't get used afterwards when executing the APP.

Relevant Error Section

dex2artist: Could not create image space with image file '/system/framework/boot.art'. Attempting to fall back to imageless running. Error was: Unable to relocate image '/system/framework/boot.art' from '/system/framework/x86_64/boot.art' to '/data/dalvik-cache/x86_64/system@[email protected]': Only the zygote can create the global boot image.
dex2artist: Attempted image: /system/framework/boot.art
dex2artist: Dex file fallback disabled, cannot continue without image.
dex2artist: Failed to create runtime

Error Log (long)

ArtistProcExecutor: execute() SU [`export LD_LIBRARY_PATH=/data/app/saarland.cispa.artist.artistgui-2/lib/x86_64:/data/user/0/saarland.cispa.artist.artistgui/files/artist/lib/;/data/user/0/saarland.cispa.artist.artistgui/files/artist/dex2oat --oat-file=/data/app/de.heise.android.heiseonlineapp-1/oat/x86_64/base.odex --compiler-backend=Optimizing --compiler-filter=everything --generate-debug-info --compile-pic --dex-file=/data/app/de.heise.android.heiseonlineapp-1/base.apk --checksum-rewriting --dex-location=/data/app/de.heise.android.heiseonlineapp-1/base.apk --instruction-set=x86_64 --instruction-set-variant=atom --instruction-set-features=smp --instruction-set-features=default`]
ArtistProcExecutor: > execute() ProcessInfos: [Process[pid=24267, hasExited=false]]
ArtistProcExecutor: > execute() Waiting: de.heise.android.heiseonlineapp:dex2artist
art     : Dex location /system/framework/am.jar does not seem to include dex file. Allow oat file use. This is potentially dangerous.
tmp-mksh: type=1400 audit(0.0:979): avc: denied { execute } for name="dex2oat" dev="dm-0" ino=22057 scontext=u:r:init:s0 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=file permissive=1
tmp-mksh: type=1400 audit(0.0:980): avc: denied { execute_no_trans } for path="/data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat" dev="dm-0" ino=22057 scontext=u:r:init:s0 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=file permissive=1
memtrack: Couldn't load memtrack module (No such file or directory)
android.os.Debug: failed to load memtrack module: -2
Radio-JNI: register_android_hardware_Radio DONE
dex2artist: /data/user/0/saarland.cispa.artist.artistgui/files/artist/dex2oat --oat-file=/data/app/de.heise.android.heiseonlineapp-1/oat/x86_64/base.odex --compiler-backend=Optimizing --compiler-filter=everything --generate-debug-info --compile-pic --dex-file=/data/app/de.heise.android.heiseonlineapp-1/base.apk --checksum-rewriting --dex-location=/data/app/de.heise.android.heiseonlineapp-1/base.apk
dex2artist: Could not create image space with image file '/system/framework/boot.art'. Attempting to fall back to imageless running. Error was: Unable to relocate image '/system/framework/boot.art' from '/system/framework/x86_64/boot.art' to '/data/dalvik-cache/x86_64/system@[email protected]': Only the zygote can create the global boot image.
dex2artist: Attempted image: /system/framework/boot.art
dex2artist: Dex file fallback disabled, cannot continue without image.
dex2artist: Failed to create runtime
dex2artist: dex2oat took 4.737ms (threads: 4) 
AndroidRuntime: Calling main entry com.android.commands.am.Am
AndroidRuntime: Shutting down VM
EGL_emulation: eglMakeCurrent: 0x766803437320: ver 2 0 (tinfo 0x76680340c340)
WindowManager: Attempted to remove non-existing token: android.os.Binder@db980df
AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<<
AndroidRuntime: CheckJNI is ON
art     : Could not create image space with image file '/system/framework/boot.art'. Attempting to fall back to imageless running. Error was: Unable to relocate image '/system/framework/boot.art' from '/system/framework/x86_64/boot.art' to '/data/dalvik-cache/x86_64/system@[email protected]': Only the zygote can create the global boot image.
art     : Attempted image: /system/framework/boot.art
art     : Unexpected CPU variant for X86 using defaults: x86_64
ArtistProcExecutor: > execute() Waiting: de.heise.android.heiseonlineapp:dex2artist DONE

Affected Projects

• ArtistGui (configuring commandline / config parameter when calling dex2oat)
• ARTist (perhaps we need to adapt this as well)
• art (perhaps we need to adapt this as well)

Module configuration

Currently, there is no way to configure a module. For example, changing the proxy's IP in a traffic interceptor module requires either a new module or ugly hacks in the module. It would be more comfortable to configure IPs, ports, etc. in Artist Gui. The module could use those values during compilation.

Affected Projects

• ARTist
• ArtistGui

Distinguish between usage during AOSP build, as system compiler or as an app asset

Distinguishing whether the current compilation is happening as a part of the AOSP build (creating the initial boot.oat and boot.art), during system boot (first & after OS upgrade) or at regular system runtime for app compilation (the ArtistGui path) is required for the filesystem helper to provide the appropriate paths (see TODO in this commit that closed Project-ARTist/ARTist#16). While it is obvious that the paths on an AOSP build machine and on the actual Android device differ, we also need to differentiate between system compiler and ArtistGui compilation because at boot time not all directories are accessible and we cannot assume that ArtistGui and its folder structure is created already.

Affected Projects

• ARTist
• art

Branching & Release Scheme

While #6 focuses on how to decide for a versioning scheme (e.g., semantic versioning), this issue focuses on the branching and release policy, hence what branches to create when, how to merge them, when to release, what can and should be automated etc.

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen

Module: Method Tracing (traceArtist) Improvements

Perhaps this is a good beginner's task?

We could improve the traceModule with the following features:

• Indent each subsequent method call, to better display the call hierarchy:
  • methodCall1()
    • methodCall1.1()
    • methodCall1.2()
      • methodCall1.2.1()
  • methodCall2()
    • methodCall2.1()
  • methodCall3()
• Enable File tracing mode, that logs the tracelogs to a file (CSV file), in a machine-readable way, to be able to further process it and get more insights.

System Server Instrumentation Support

Prepare dex2oat and artist to instrument the system server during (first) boot.

This requires, e.g., that we can disable modules for all other compilations (#1), merge the codelib with those parts that we want to instrument, etc.

Affected Projects

• ARTist
• art
• ArtistGui
• dexterous
• monkey-troop
• Codelib
• codelib-gen
• other

Auto deploy module into ArtistGui (from IDE) while developing modules

The introduction of modules made the process of developing a module for ARTist a lot easier. It's not necessary to checkout AOSP anymore. However it is still messy to develop modules. The developer has to build the module, push it to the device, import the module and instrument the desired app using the module. A lot of room for improvement.
What we want is to press a button in the IDE of the user which builds the module, pushes it to the device, instruments the desired app and starts the app. We have a CMakeLists.txt file for IDE support (syntax highlighting, auto completion, ...).

It's possible to let CMake build the module automatically¹ and push the module onto a device. ArtistGui can import it, instrument an app and start it automatically using a custom broadcast. We can trigger it then after pushing the module to the device.

¹ https://cmake.org/cmake/help/v3.6/manual/cmake-toolchains.7.html

Affected Projects (ticked = done)

• ArtistGui
• module-sdk-gen
• template-module

Website: Documentation 2.0

Our documentation at https://artist.cispa.saarland is very bare-bones and not up to date. While we are working towards entering beta, the website needs to be adapted as well.

First, when we officially enter beta, the documentation needs to be updated to reflect the new workflows for, e.g., setting up your environment, creating modules, build the ARTist core components and so on.

Second, the current scope of the documentation is way too tight. Instead of just giving step-by-step introductions for a small set of workflows, we should in addition have more high-level descriptions of what we want to achieve and why, the different components in the ARTist ecosystem, the future roadmap and a lot more. In general, the website should be a rich source of information about the project that users frequently visit to dig further into the material.

Affected Projects

The project website at https://artist.cispa.saarland

`boot.oat` compilation support (AOSP build time)

This issue aims for supporting the compilation of the boot.oat and boot.art files that resemble the Zygote pre-loaded classes since the introduction of ART.

To the best of my knowledge, the two image files are compiled in two different scenarios:

1. when compiling AOSP, the initial boot files are created
2. after a system upgrade

In this issue, we are focusing on the system-centric deployment path here where we compile AOSP with art and ARTist replacing the original art repository at build time (scenario 1). Chances are scenario 2 will follow naturally and is omitted here (harder to test). If you feel this is something you require for using ARTist for your project(s), feel free to open an own issue for that.

Distinguishing whether dex2oat is invoked at compile time or on the device (#16) is a blocker because of the filesystem helper that requires proper paths to be set up.

Affected Projects

• ARTist
• art