project-artist / meta Goto Github PK
View Code? Open in Web Editor NEWMeta repository for matters related to more than one repository or even the whole project as such.
Meta repository for matters related to more than one repository or even the whole project as such.
We currently do not employ any kind of versioning in our projects, but having projects that are not up to date easily leads to incompatibilities, in particular since we are heavily working on all parts. Also, figuring out later which versions in the projects matched at an earlier point in time boils down to comparing commit times, which is a mess.
Introducing proper versioning can solve this problem. It is important to have a versioning scheme that reflects the compatibility between the main parts. I am thinking along the lines of semantic versioning, for example same major and minor version imply perfect compatibility where the patch level only applies to non-breaking changes such as, e.g., small fixes. Finding our what exactly can be derived from, e.g., common major and diverging minor or other combinations is part of this task.
This in general requires some careful consideration since connecting and unifying releases of so many projects with different scopes is an involved task.
Unfortunately, in some cases our modules crash because of what I believe to be inconsistent memory management on our side. More precisely, the trace module for example crashes on x86 if its artist pass is created using a placement new
operator with the current arena allocator. The following quick-fix solves the problems for now for trace:
HArtist *TraceModule::createPass(const MethodInfo &method_info) const {
// return new(method_info.GetGraph()->GetArena()) HTraceArtist(method_info);
return new HTraceArtist(method_info);
}
From what we learned from the ART code, classes prefixed with an H
are meant to be arena allocated. However, artist passes extend HOptimization
s and therefore should be treated accordingly. But actually using arena allocation there triggers weird crashes such as this one here untangled by ndk-stack
from a trace crash without the patch above:
********** Crash dump: **********
29 │ Build fingerprint: 'Android/sdk_phone_x86/generic_x86:7.1.1/NYC/4931657:userdebug/test-keys'
30 │ pid: 13209, tid: 13209, name: main >>> /data/user/0/saarland.cispa.artist.artistgui/files/artist/dex2oat <<<
31 │ signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x4
32 │ Stack frame #00 pc 001f0345 /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine HOptimization at /proc/self/cwd/art/compiler/optimizing/optimization.h:34
33 │ Stack frame #01 pc 001f1bb1 /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::OptimizingCompiler::Compile(art::DexFile::CodeItem const*, unsigned int, art::InvokeType, unsigned short, unsigned int, _jobject*, art::DexFile const&,
│ art::Handle<art::mirror::DexCache>) const at /proc/self/cwd/art/compiler/optimizing/optimizing_compiler.cc:857
34 │ Stack frame #02 pc 0014ab2c /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompileMethod(art::Thread*, art::CompilerDriver*, art::DexFile::CodeItem const*, unsigned int, art::InvokeType, unsigned short, unsigned int, _jobject*
│ , art::DexFile const&, art::optimizer::DexToDexCompilationLevel, bool, art::Handle<art::mirror::DexCache>) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:649
35 │ Stack frame #03 pc 0015e19c /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompileClassVisitor::Visit(unsigned int) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:2710
36 │ Stack frame #04 pc 001595fe /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::ParallelCompilationManager::ForAllClosure::Run(art::Thread*) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:1957
37 │ Stack frame #05 pc 0055e360 /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart.so: Routine art::ThreadPool::Wait(art::Thread*, bool, bool) at /proc/self/cwd/art/runtime/thread_pool.cc:215
38 │ Stack frame #06 pc 0015260f /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::ParallelCompilationManager::ForAll(unsigned int, unsigned int, art::CompilationVisitor*, unsigned int) at /proc/self/cwd/art/compiler/driver/compiler_d
│ river.cc:1933
39 │ Stack frame #07 pc 00147b3d /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompilerDriver::CompileDexFile(_jobject*, art::DexFile const&, std::__1::vector<art::DexFile const*, std::__1::allocator<art::DexFile const*> > const&,
│ art::ThreadPool*, unsigned int, art::TimingLogger*) at /proc/self/cwd/art/compiler/driver/compiler_driver.cc:2750 (discriminator 1)
40 │ Stack frame #08 pc 001463b1 /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libart-compiler.so: Routine art::CompilerDriver::CompileAll(_jobject*, std::__1::vector<art::DexFile const*, std::__1::allocator<art::DexFile const*> > const&, art::TimingLogger*) at /
│ proc/self/cwd/art/compiler/driver/compiler_driver.cc:506
41 │ Stack frame #09 pc 000251d9 /data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat: Routine art::Dex2Oat::Compile() at /proc/self/cwd/art/dex2oat/dex2oat.cc:1710 (discriminator 1)
42 │ Stack frame #10 pc 0000b4eb /data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat: Routine art::CompileApp(art::Dex2Oat&) at /proc/self/cwd/art/dex2oat/dex2oat.cc:2725
43 │ Stack frame #11 pc 0001532c /data/data/saarland.cispa.artist.artistgui/files/artist/lib/libc.so: Routine __libc_init at /proc/self/cwd/bionic/libc/bionic/libc_init_dynamic.cpp:109
44 │ Stack frame #12 pc 0000a272 /data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat: Routine _start at art/dex2oat/dex2oat.cc:?
45 │ Stack frame #13 pc 0000000d <unknown>
So what can we do? I assume that we have some inconsistency in where we use arena allocation and where we use other means of memory management (e.g., smart pointers). Fixing that might solve the problem. For example, there is an arena version of a vector and I am pretty sure we use regular vectors to store H
-prefixed objects, such as HInstructions
, but this is just a guess.
Anyway, at some point we need to better understand the arena allocation implementation and make sure we use it correctly within artist and the modules to avoid crashes.
Remove ARTist as art's submodule and place ARTist in local_manifest
This allows us to use repo to efficiently switch branches and manage the remotes and our sourcecode.
Currently, the artist modules used are determined at compile-time by hardwiring them in optimizing_compiler.cc.
A better approach is to allow for activation and deactivation without requiring recompilation of ARTist by, e.g., controlling modules through either configurations (files?) or command line arguments to the compiler.
Depends on #1.
(Moved here from Project-ARTist/ARTist#4)
Frida is a well-known dynamic hooking framework that also supports Android. There are two possibilities: Either run the frida server on your device so that you can attach to all processes (including apps) or add the frida gadget library into a particular app/binary/process so that only this one is exposed.
Since the latter either required repackaging apps, which breaks the signature, or adding this in a system-centric manner, which would require OS changes, ARTist might provide an alternative deployment path. By making sure the gadget library is loaded in a target process, we can selectively expose apps without requiring the big server (which people often try to detect as part of their anti-debugging strategy) and by avoiding the downsides of the gadget approach.
README.md files are the landing pages of repositories, hence these are the first thing new users see when they visit one of our repositories. Currently, however, those READMEs are not very helpful or even (almost) non-existent. Since we want to make it easier for new users to join our project and work with ARTist, we should have helpful README files that provide some basic information and point at further resources where they can find help. Beside information about the actual repository, it should also give a basic introduction into the overall ARTist project and put the repository in context. Also, a pointer to the documentation and Gitter group would help.
Check a task as soon as a proper README.md file has been committed in the corresponding project.
Implement enabling and disabling functionality for modules so that, e.g., modules are simply disabled if preconditions are not met, instead of terminating the whole compilation.
Since we officially announced the beta, it is time to update all the README files.
This includes the Beta remark in almost all README files, as well as adding READMEs to the new projects, such as the modules and codelibs
While our current showcase modules simply run on all methods of an application, some modules aim at a particular set of methods, e.g., from a single class (package name prefix), with
particular parameter or return types, or other method properties.
Currently, modules could achieve this by checking for those properties themselves in the beginning and immediately return/terminate the execution for this method if it does not match the filter. There are 2 problems with this approach:
It would be nice to have built-in functionality so that declaring those filters is straightforward. Ready-made implementations for whitelist and blacklist will cover a lot of cases and modules with special requirements can implement their own filters by, e.g., extending them.
Implementation in:
ARTist's injection only works when replacing the original /data/app/<package>/base.apk
with the injected and signed base.apk..
On Android 6, 7 & 7.1 it works, when rewriting / faking the checksums.
The only log message indicating the problem is: zygote64: expected 1 dex files but found 2
Then the compiled oat fiel is thrown away and the original non-injected app is executed
10-10 15:37:28.066 1081 1784 I ActivityManager: START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=de.heise.android.heiseonlineapp/.ui.ChannelActivity bnds=[255,245][512,440]} from uid 10035
10-10 15:37:28.069 803 4088 I ACDB-LOADER: ACDB AFE returned = -19
10-10 15:37:28.069 803 4088 D hardware_info: hw_info_append_hw_type : device_name = speaker
10-10 15:37:28.069 803 4088 D audio_hw_primary: enable_snd_device: snd_device(2: speaker)
10-10 15:37:28.069 803 4088 D audio_hw_primary: enable_audio_route: usecase(1) apply and update mixer path: low-latency-playback speaker
10-10 15:37:28.069 872 997 E ANDR-PERF-OPTSHANDLER: Warning: Resource [2, 0] not supported for core 1. Instead use resource for core 0
10-10 15:37:28.069 872 997 E ANDR-PERF-RESOURCEQS: Failed to apply optimization [2, 2, 0]
10-10 15:37:28.101 1081 2426 I ActivityManager: Start proc 17944:de.heise.android.heiseonlineapp/u0a150 for activity de.heise.android.heiseonlineapp/.ui.ChannelActivity
10-10 15:37:28.165 17944 17944 E zygote64: expected 1 dex files but found 2
Log(Fatal)
terminates here:
bool OatFileAssistant::DexChecksumUpToDate(const OatFile& file, std::string* error_msg) {
(...)
// art::OatFile
uint32_t number_of_dex_files = file.GetOatHeader().GetDexFileCount();
if (required_dex_checksums->size() != number_of_dex_files) {
(...)
}
bool OatFileAssistant::DexChecksumUpToDate(const VdexFile& file, std::string* error_msg) {
(...)
// art::VdexFile
uint32_t number_of_dex_files = file.GetHeader().GetNumberOfDexFiles();
if (required_dex_checksums->size() != number_of_dex_files) {
(...)
}
Experiments with rewriting the dexfile count have been done here:
Project-ARTist/art@e3b1e22 via:
OatHeader::ArtistFixDexFileCount(){ ... }
w/o success. Injected apps didn't even log an error, before ARTist's compiled oat file was thrown away.
Better explain what is or what should be done in the CodeLib. We have the CodeLib website, but it mainly focuses on the "How to compile" part.
We want to be able to write modules without requiring to compile them with/into the compiler and load them dynamically.
One possible approach is to create a module sdk that is used to create and compile modules as regular c++ projects without the need to ever touch the compiler. The idea is that the resulting compiled module registers itself with the compiler so that we can not only control externally which of the modules in the compiler are executed (see Project-ARTist/ARTist#4 ) but actually move the module code out of the compiler and only have module-agnostic code in there.
There are multiple advantages:
(moved here from Project-ARTist/ARTist#5)
ARTist / ARTistGui cannot instrument apps on the Nougat 7.1.1 x86_64 emulator images
Everything works find on the Nougat 7.1.1 x86 images.
/data/dalvik-cache/x86_64/
is empty on those images, but dex2oat looks for data/dalvik-cache/x86_64/system@[email protected]
and possible more files there.
dex2oat also tries to copy them from /system/framework/x86_64/
but is not allowed to do so.
/data/dalvik-cache/x86_64/
again, when executing dex2oat. --instruction-set=x86
leads to a working compilation/instrumentation process, but the instrumented oat-file won't get used afterwards when executing the APP.dex2artist: Could not create image space with image file '/system/framework/boot.art'. Attempting to fall back to imageless running. Error was: Unable to relocate image '/system/framework/boot.art' from '/system/framework/x86_64/boot.art' to '/data/dalvik-cache/x86_64/system@[email protected]': Only the zygote can create the global boot image.
dex2artist: Attempted image: /system/framework/boot.art
dex2artist: Dex file fallback disabled, cannot continue without image.
dex2artist: Failed to create runtime
ArtistProcExecutor: execute() SU [`export LD_LIBRARY_PATH=/data/app/saarland.cispa.artist.artistgui-2/lib/x86_64:/data/user/0/saarland.cispa.artist.artistgui/files/artist/lib/;/data/user/0/saarland.cispa.artist.artistgui/files/artist/dex2oat --oat-file=/data/app/de.heise.android.heiseonlineapp-1/oat/x86_64/base.odex --compiler-backend=Optimizing --compiler-filter=everything --generate-debug-info --compile-pic --dex-file=/data/app/de.heise.android.heiseonlineapp-1/base.apk --checksum-rewriting --dex-location=/data/app/de.heise.android.heiseonlineapp-1/base.apk --instruction-set=x86_64 --instruction-set-variant=atom --instruction-set-features=smp --instruction-set-features=default`]
ArtistProcExecutor: > execute() ProcessInfos: [Process[pid=24267, hasExited=false]]
ArtistProcExecutor: > execute() Waiting: de.heise.android.heiseonlineapp:dex2artist
art : Dex location /system/framework/am.jar does not seem to include dex file. Allow oat file use. This is potentially dangerous.
tmp-mksh: type=1400 audit(0.0:979): avc: denied { execute } for name="dex2oat" dev="dm-0" ino=22057 scontext=u:r:init:s0 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=file permissive=1
tmp-mksh: type=1400 audit(0.0:980): avc: denied { execute_no_trans } for path="/data/data/saarland.cispa.artist.artistgui/files/artist/dex2oat" dev="dm-0" ino=22057 scontext=u:r:init:s0 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=file permissive=1
memtrack: Couldn't load memtrack module (No such file or directory)
android.os.Debug: failed to load memtrack module: -2
Radio-JNI: register_android_hardware_Radio DONE
dex2artist: /data/user/0/saarland.cispa.artist.artistgui/files/artist/dex2oat --oat-file=/data/app/de.heise.android.heiseonlineapp-1/oat/x86_64/base.odex --compiler-backend=Optimizing --compiler-filter=everything --generate-debug-info --compile-pic --dex-file=/data/app/de.heise.android.heiseonlineapp-1/base.apk --checksum-rewriting --dex-location=/data/app/de.heise.android.heiseonlineapp-1/base.apk
dex2artist: Could not create image space with image file '/system/framework/boot.art'. Attempting to fall back to imageless running. Error was: Unable to relocate image '/system/framework/boot.art' from '/system/framework/x86_64/boot.art' to '/data/dalvik-cache/x86_64/system@[email protected]': Only the zygote can create the global boot image.
dex2artist: Attempted image: /system/framework/boot.art
dex2artist: Dex file fallback disabled, cannot continue without image.
dex2artist: Failed to create runtime
dex2artist: dex2oat took 4.737ms (threads: 4)
AndroidRuntime: Calling main entry com.android.commands.am.Am
AndroidRuntime: Shutting down VM
EGL_emulation: eglMakeCurrent: 0x766803437320: ver 2 0 (tinfo 0x76680340c340)
WindowManager: Attempted to remove non-existing token: android.os.Binder@db980df
AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<<
AndroidRuntime: CheckJNI is ON
art : Could not create image space with image file '/system/framework/boot.art'. Attempting to fall back to imageless running. Error was: Unable to relocate image '/system/framework/boot.art' from '/system/framework/x86_64/boot.art' to '/data/dalvik-cache/x86_64/system@[email protected]': Only the zygote can create the global boot image.
art : Attempted image: /system/framework/boot.art
art : Unexpected CPU variant for X86 using defaults: x86_64
ArtistProcExecutor: > execute() Waiting: de.heise.android.heiseonlineapp:dex2artist DONE
Currently, there is no way to configure a module. For example, changing the proxy's IP in a traffic interceptor module requires either a new module or ugly hacks in the module. It would be more comfortable to configure IPs, ports, etc. in Artist Gui. The module could use those values during compilation.
Distinguishing whether the current compilation is happening as a part of the AOSP build (creating the initial boot.oat
and boot.art
), during system boot (first & after OS upgrade) or at regular system runtime for app compilation (the ArtistGui path) is required for the filesystem helper to provide the appropriate paths (see TODO in this commit that closed Project-ARTist/ARTist#16). While it is obvious that the paths on an AOSP build machine and on the actual Android device differ, we also need to differentiate between system compiler and ArtistGui compilation because at boot time not all directories are accessible and we cannot assume that ArtistGui and its folder structure is created already.
While #6 focuses on how to decide for a versioning scheme (e.g., semantic versioning), this issue focuses on the branching and release policy, hence what branches to create when, how to merge them, when to release, what can and should be automated etc.
Perhaps this is a good beginner's task?
We could improve the traceModule with the following features:
methodCall1()
methodCall1.1()
methodCall1.2()
methodCall1.2.1()
methodCall2()
methodCall2.1()
methodCall3()
Prepare dex2oat and artist to instrument the system server during (first) boot.
This requires, e.g., that we can disable modules for all other compilations (#1), merge the codelib with those parts that we want to instrument, etc.
The introduction of modules made the process of developing a module for ARTist a lot easier. It's not necessary to checkout AOSP anymore. However it is still messy to develop modules. The developer has to build the module, push it to the device, import the module and instrument the desired app using the module. A lot of room for improvement.
What we want is to press a button in the IDE of the user which builds the module, pushes it to the device, instruments the desired app and starts the app. We have a CMakeLists.txt file for IDE support (syntax highlighting, auto completion, ...).
It's possible to let CMake build the module automatically¹ and push the module onto a device. ArtistGui can import it, instrument an app and start it automatically using a custom broadcast. We can trigger it then after pushing the module to the device.
¹ https://cmake.org/cmake/help/v3.6/manual/cmake-toolchains.7.html
Our documentation at https://artist.cispa.saarland is very bare-bones and not up to date. While we are working towards entering beta, the website needs to be adapted as well.
First, when we officially enter beta, the documentation needs to be updated to reflect the new workflows for, e.g., setting up your environment, creating modules, build the ARTist core components and so on.
Second, the current scope of the documentation is way too tight. Instead of just giving step-by-step introductions for a small set of workflows, we should in addition have more high-level descriptions of what we want to achieve and why, the different components in the ARTist ecosystem, the future roadmap and a lot more. In general, the website should be a rich source of information about the project that users frequently visit to dig further into the material.
The project website at https://artist.cispa.saarland
This issue aims for supporting the compilation of the boot.oat
and boot.art
files that resemble the Zygote pre-loaded classes since the introduction of ART.
To the best of my knowledge, the two image files are compiled in two different scenarios:
In this issue, we are focusing on the system-centric deployment path here where we compile AOSP with art and ARTist replacing the original art repository at build time (scenario 1). Chances are scenario 2 will follow naturally and is omitted here (harder to test). If you feel this is something you require for using ARTist for your project(s), feel free to open an own issue for that.
Distinguishing whether dex2oat is invoked at compile time or on the device (#16) is a blocker because of the filesystem helper that requires proper paths to be set up.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.