Giter Site home page Giter Site logo

project-artist / dexterous Goto Github PK

View Code? Open in Web Editor NEW
46.0 5.0 16.0 7.83 MB

Library and standalone CLI tool for apk/dex merging, repackaging and signing. Can also get used as a dex analyzer framework.

License: Other

Java 99.91% HTML 0.06% Python 0.03%
dex analysis tool android artist merge

dexterous's Introduction

dexterous

Build Status Gitter

Dexterous is a commandline interface to ARTist's partial dex merging, apk-packaging and apk-signing. It is used as a preprocessing step before handing apk and zip files to ARTist, where its main task is to partially merge a CodeLib so that ARTist modules can, e.g., inject calls to CodeLib methods into the target code. Partial merge means that the CodeLib dex files are added to the target and their methods are registered with the existing target dex files. Essentially, the CodeLib's symbols (dex method identifiers) are added to the target dex files to make it possible for the target code to call into the CodeLib. For more information about the ARTist ecosystem, see the dedicated section below.

In addition to partially merging dex files, dexterous can also be used for analyzing dex files: Take a look at the class saarland.cispa.dexterous.cli.Dexterously to get started with implementing your own analysis.

Build

You can build a fat jar with all depedencies with the following command:

./gradlew DexterousJar

Build files

desktop.gradle is the build file for the desktop build.

build.gradle is the build file for the android-library build.

It implicitly uses the bundled debug keystore: res/artist-debug.keystore.

dexterous usage

user@host ~/dexterous $ java -jar build/libs/dexterous.jar --help

usage: Dexterously <options>
 -a,--analyze          Analyze APK
 -b,--build-apk        Build partially merged APK
 -c,--codelib <file>   Path to codelib.apk (name doesn't matter).
 -h,--help             Prints this message.
 -m,--merge            Build merged APK
 -s,--sign-apk         Build and sign partialy merged APK

Merge codelib partially w/o signing the apk

java -jar dexterous.jar my_application.apk --codelib codelib.apk --build-apk

Merge codelib partially and resign apk

java -jar dexterous.jar my_application.apk --codelib codelib.apk --build-apk --sign-apk

Merge two dex files completely

java -jar dexterous.jar --merge my_application.dex library.dex

Analyze apk only

Execute dexterous without the build / sign flags

java -jar dexterous.jar --analyze my_application.apk

Third-party code usage:

dexterous uses a lot of third-party code

Dependency libraries:

The following code is used as depedency libraries:

Included code

The following code is included, but moved to different packages, modified and fixed:

ARTist - The Android Runtime Instrumentation and Security Toolkit

ARTist is a flexible open source instrumentation framework for Android's apps and Java middleware. It is based on the Android Runtime’s (ART) compiler and modifies code during on-device compilation. In contrast to existing instrumentation frameworks, it preserves the application's original signature and operates on the instruction level.

ARTist can be deployed in two different ways: First, as a regular application using our ArtistGui project (this repository) that allows for non-invasive app instrumentation on rooted devices, or second, as a system compiler for custom ROMs where it can additionally instrument the system server (Package Manager Service, Activity Manager Service, ...) and the Android framework classes (boot.oat). It supports Android versions after (and including) Marshmallow 6.0.

For detailed tutorials and more in-depth information on the ARTist ecosystem, have a look at our official documentation and join our Gitter chat.

Upcoming Beta Release

We are about to enter the beta phase soon, which will bring a lot of changes to the whole ARTist ecosystem, including a dedicated ARTist SDK for simplified Module development, a semantic versioning-inspired release and versioning scheme, an improved and updated version of our online documentation, great new Modules, and a lot more improvements. However, in particular during the transition phase, some information like the one in the repositories' README.md files and the documentation at https://artist.cispa.saarland might be slightly out of sync. We apologize for the inconvenience and happily take feedback at Gitter. To keep up with the current progress, keep an eye on the beta milestones of the Project: ARTist repositories and check for new blog posts at https://artist.cispa.saarland .

Contribution

We hope to create an active community of developers, researchers and users around Project ARTist and hence are happy about contributions and feedback of any kind. There are plenty of ways to get involved and help the project, such as testing and writing Modules, providing feedback on which functionality is key or missing, reporting bugs and other issues, or in general talk about your experiences. The team is actively monitoring Gitter and of course the repositories, and we are happy to get in touch and discuss. We do not have a full-fledged contribution guide, yet, but it will follow soon (see beta announcement above).

Academia

ARTist is based on a paper called ARTist - The Android Runtime Instrumentation and Security Toolkit, published at the 2nd IEEE European Symposium on Security and Privacy (EuroS&P'17). The full paper is available here. If you are citing ARTist in your research, please use the following bibliography entry:

@inproceedings{artist,
  title={ARTist: The Android runtime instrumentation and security toolkit},
  author={Backes, Michael and Bugiel, Sven and Schranz, Oliver and von Styp-Rekowsky, Philipp and Weisgerber, Sebastian},
  booktitle={2017 IEEE European Symposium on Security and Privacy (EuroS\&P)},
  pages={481--495},
  year={2017},
  organization={IEEE}
}

There is a follow-up paper where we utilized ARTist to cut out advertisement libraries from third-party applications, move the library to a dedicated app (own security principal) and reconnect both using a custom Binder IPC protocol, all while preserving visual fidelity by displaying the remote advertisements as floating views on top of the now ad-cleaned application. The full paper The ART of App Compartmentalization: Compiler-based Library Privilege Separation on Stock Android, as it was published at the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS'17), is available here.

dexterous's People

Contributors

alfink avatar schrnz avatar sweisgerber-dev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

dexterous's Issues

Fine-grained merging

Instead of merging all classes, types and methods, it would be preferable to allow for selective, fine-grained merging. This avoids wasting IDs on Android helper classes that are not really used and allows for merging codelib packages that feature bigger libraries. With this approach, a codelib class could define a small interface to the large library and only those few class methods are assigned IDs in target dex files (see 64k method ID limit).
This could, e.g., be controlled with blacklists or whitelists provided via commandline arguments.

Cannot merge codelib into systemserver

When I try to use the current dexterous version @ fd5a845 to merge a codelib into the systemserver (services.jar), the following error occurs:

Partially merging CodeLib into services.jar using dexterous
I/Dexterous: Loading DexFile: services.apk:classes.dex
I/Dexterous: Loading DexFile: services.apk:classes2.dex
I/Dexterous: DONE Loading 01 Dexfiles (services.apk)
I/Dexterous: Loading DexFile: app-debug.apk:classes.dex
I/Dexterous: Loading DexFile: app-debug.apk:classes2.dex
I/Dexterous: DONE Loading 01 Dexfiles (app-debug.apk)
I/Dexterously: # MULTIDEX File. DexFile Count: 2
I/Dexterously: 
I/Dexterously: # DEXFILE: services.apk:classes.dex #############################################
I/Dexterously: 
I/Loggy: ----------------------------------------------------------------------------
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                     HEADER | OFF:        0 | SIZE:        1 |
I/Loggy: Section |TYPE:                 STRING_IDS | OFF:      112 | SIZE:    63478 |
I/Loggy: Section |TYPE:                   TYPE_IDS | OFF:   254024 | SIZE:     4637 |
I/Loggy: Section |TYPE:                  PROTO_IDS | OFF:   272572 | SIZE:     9860 |
I/Loggy: Section |TYPE:                  FIELD_IDS | OFF:   390892 | SIZE:    19115 |
I/Loggy: Section |TYPE:                 METHOD_IDS | OFF:   543812 | SIZE:    34535 |
I/Loggy: Section |TYPE:                 CLASS_DEFS | OFF:   820092 | SIZE:     2616 |
I/Loggy: Section |TYPE:   ANNOTATION_SET_REF_LISTS | OFF:   903804 | SIZE:      352 |
I/Loggy: Section |TYPE:            ANNOTATION_SETS | OFF:   908676 | SIZE:     3517 |
I/Loggy: Section |TYPE:                 CODE_ITEMS | OFF:   945024 | SIZE:    24583 |
I/Loggy: Section |TYPE: ANNOTATION_DIRECTORY_ITEMS | OFF:  4128108 | SIZE:     2239 |
I/Loggy: Section |TYPE:                 TYPE_LISTS | OFF:  4193532 | SIZE:     6329 |
I/Loggy: Section |TYPE:          STRING_DATA_ITEMS | OFF:  4262876 | SIZE:    63478 |
I/Loggy: Section |TYPE:            DEBUG_INFO_ITEM | OFF:  5763692 | SIZE:    22550 |
I/Loggy: Section |TYPE:            ANNOTATION_ITEM | OFF:  6587462 | SIZE:     3781 |
I/Loggy: Section |TYPE:        ENCODED_ARRAY_ITEMS | OFF:  6656830 | SIZE:      539 |
I/Loggy: Section |TYPE:           CLASS_DATA_ITEMS | OFF:  6670105 | SIZE:     2604 |
I/Loggy: Section |TYPE:                   MAP_LIST | OFF:  6880792 | SIZE:        1 |
I/Loggy: ----------------------------------------------------------------------------
I/Dexterously: 
I/Dexterously: # DEXFILE: app-debug.apk:classes.dex #############################################
I/Dexterously: 
I/Loggy: ----------------------------------------------------------------------------
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                     HEADER | OFF:        0 | SIZE:        1 |
I/Loggy: Section |TYPE:                 STRING_IDS | OFF:      112 | SIZE:    21234 |
I/Loggy: Section |TYPE:                   TYPE_IDS | OFF:    85048 | SIZE:     2352 |
I/Loggy: Section |TYPE:                  PROTO_IDS | OFF:    94456 | SIZE:     3489 |
I/Loggy: Section |TYPE:                  FIELD_IDS | OFF:   136324 | SIZE:    10394 |
I/Loggy: Section |TYPE:                 METHOD_IDS | OFF:   219476 | SIZE:    17438 |
I/Loggy: Section |TYPE:                 CLASS_DEFS | OFF:   358980 | SIZE:     1626 |
I/Loggy: Section |TYPE:                 CODE_ITEMS | OFF:   411012 | SIZE:    11730 |
I/Loggy: Section |TYPE:            DEBUG_INFO_ITEM | OFF:  1237269 | SIZE:    10561 |
I/Loggy: Section |TYPE:                 TYPE_LISTS | OFF:  1463400 | SIZE:     2079 |
I/Loggy: Section |TYPE:          STRING_DATA_ITEMS | OFF:  1484594 | SIZE:    21234 |
I/Loggy: Section |TYPE:            ANNOTATION_ITEM | OFF:  2021194 | SIZE:     2040 |
I/Loggy: Section |TYPE:           CLASS_DATA_ITEMS | OFF:  2054433 | SIZE:     1516 |
I/Loggy: Section |TYPE:        ENCODED_ARRAY_ITEMS | OFF:  2151369 | SIZE:      291 |
I/Loggy: Section |TYPE:            ANNOTATION_SETS | OFF:  2167192 | SIZE:     1916 |
I/Loggy: Section |TYPE:   ANNOTATION_SET_REF_LISTS | OFF:  2189004 | SIZE:      144 |
I/Loggy: Section |TYPE: ANNOTATION_DIRECTORY_ITEMS | OFF:  2191632 | SIZE:     1436 |
I/Loggy: Section |TYPE:                   MAP_LIST | OFF:  2237768 | SIZE:        1 |
I/Loggy: ----------------------------------------------------------------------------
I/MethodFilter: Annotationtype used for whitelisting:Lsaarland/cispa/artist/codelib/CodeLib$Inject;
Exception in thread "main" java.lang.IllegalArgumentException: position=88867372 length=2237988
	at comm.android.dex.Dex.open(Dex.java:191)
	at comm.android.dx.merge.MethodFilter.getString(MethodFilter.java:136)
	at comm.android.dx.merge.MethodFilter.getMethodString(MethodFilter.java:150)
	at comm.android.dx.merge.MethodFilter.initializeWhitelist(MethodFilter.java:230)
	at comm.android.dx.merge.MethodFilter.<init>(MethodFilter.java:387)
	at comm.android.dex.Dex.setWhitelistedAnnotation(Dex.java:358)
	at saarland.cispa.dexterous.cli.Dexterously.mergeCodeLib(Dexterously.java:617)
	at saarland.cispa.dexterous.cli.Main.main(Main.java:49)

It does NOT occur when I use the dexterous version from before the annotation-based whitelisting patch (tested: 3253508).

@alfink do you have an intuition what is going wrong here?

Merging fields

Merging fails when whitelisting class fields.

Logcat:

11-03 21:20:19.688  7541 12227 D Dexterous: Dexterous() Init:
11-03 21:20:19.688  7541 12227 D Dexterous: > App:            /data/app/de.alfink.regbug-1/base.apk
11-03 21:20:19.688  7541 12227 D Dexterous: > CodeLib:        /data/user/0/saarland.cispa.artist.artistgui/app_modules/saarland.cispa.artist.module.regbug/codelib.apk
11-03 21:20:19.689  7541 12227 D Dexterous: > CodeLibName:    codelib.apk
11-03 21:20:19.689  7541 12227 D Dexterous: > CodeLibDexName: codelib.apk:classes.dex
11-03 21:20:19.689  7541 12227 D Dexterous:   > DexFile:        base.apk:classes.dex
11-03 21:20:19.689  7541 12227 D Dexterous:   > DexFile:        codelib.apk:classes.dex
11-03 21:20:19.690  7541 12227 I MethodFilter: Annotationtype used for whitelisting:Lsaarland/cispa/artist/codelib/CodeLib$Inject;
11-03 21:20:19.690  7541 12227 D MethodFilter: Field whitelisted:10
11-03 21:20:19.690  7541 12227 D MethodFilter: Field whitelisted:11
11-03 21:20:19.691  7541 12227 D MethodFilter: Field whitelisted:12
11-03 21:20:19.691  7541 12227 D MethodFilter: Field whitelisted:13
11-03 21:20:19.691  7541 12227 D MethodFilter: Field whitelisted:14
11-03 21:20:19.692  7541 12227 D MethodFilter: Field whitelisted:15
11-03 21:20:19.692  7541 12227 D MethodFilter: Field whitelisted:16
11-03 21:20:19.692  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:fiveargs
11-03 21:20:19.693  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:fourargs
11-03 21:20:19.693  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:onearg
11-03 21:20:19.693  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:sevenargs
11-03 21:20:19.694  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:sixargs
11-03 21:20:19.694  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:threeargs
11-03 21:20:19.694  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:twoargs
11-03 21:20:19.695  7541 12227 I MethodFilter: CodelibWhitelisting	Class whitelisted: 17 (Lsaarland/cispa/artist/codelib/CodeLib;)
11-03 21:20:19.695  7541 12227 I MethodFilter: class_annotations_off: 3720
11-03 21:20:19.695  7541 12227 I MethodFilter: f/m/p:7/7/0
11-03 21:20:19.696  7541 12227 I Dexterous: MERGING DEX codelib.apk:classes.dex into base.apk:classes.dex
11-03 21:20:19.697  7541 12227 D Dexterous: DexMerger	mergeStringIds...
11-03 21:20:19.698  7541 12227 W DexMerger: Whitelisted: 20 - LL
11-03 21:20:19.749  7541 12227 W DexMerger: Whitelisted: 21 - LLL
11-03 21:20:19.751  7541 12227 W DexMerger: Whitelisted: 22 - LLLL
11-03 21:20:19.751  7541 12227 W DexMerger: Whitelisted: 23 - LLLLL
11-03 21:20:19.752  7541 12227 W DexMerger: Whitelisted: 24 - LLLLLL
11-03 21:20:19.753  7541 12227 W DexMerger: Whitelisted: 25 - LLLLLLL
11-03 21:20:19.754  7541 12227 W DexMerger: Whitelisted: 26 - LLLLLLLL
11-03 21:20:19.755  7541 12227 W DexMerger: Whitelisted: 30 - Ldalvik/annotation/MemberClasses;
11-03 21:20:19.795  7541 12227 W DexMerger: Whitelisted: 32 - Ljava/lang/Object;
11-03 21:20:19.797  7541 12227 W DexMerger: Whitelisted: 43 - Lsaarland/cispa/artist/codelib/CodeLib;
11-03 21:20:19.803  7541 12227 W DexMerger: Whitelisted: 57 - argument1
11-03 21:20:19.843  7541 12227 W DexMerger: Whitelisted: 58 - argument2
11-03 21:20:19.844  7541 12227 W DexMerger: Whitelisted: 59 - argument3
11-03 21:20:19.845  7541 12227 W DexMerger: Whitelisted: 60 - argument4
11-03 21:20:19.845  7541 12227 W DexMerger: Whitelisted: 61 - argument5
11-03 21:20:19.846  7541 12227 W DexMerger: Whitelisted: 62 - argument6
11-03 21:20:19.847  7541 12227 W DexMerger: Whitelisted: 63 - argument7
11-03 21:20:19.847  7541 12227 W DexMerger: Whitelisted: 71 - fiveargs
11-03 21:20:19.868  7541 12227 W DexMerger: Whitelisted: 72 - fourargs
11-03 21:20:19.870  7541 12227 W DexMerger: Whitelisted: 78 - onearg
11-03 21:20:19.925  7541 12227 W DexMerger: Whitelisted: 85 - sevenargs
11-03 21:20:19.950  7541 12227 W DexMerger: Whitelisted: 86 - sixargs
11-03 21:20:19.953  7541 12227 W DexMerger: Whitelisted: 88 - threeargs
11-03 21:20:19.958  7541 12227 W DexMerger: Whitelisted: 89 - twoargs
11-03 21:20:19.962  7541 12227 W DexMerger: Whitelisted: 91 - value
11-03 21:20:19.968  7541 12227 D DexMerger$IdMerger: Blacklisted: 49
11-03 21:20:19.968  7541 12227 D Dexterous: DexMerger	mergeTypeIds...
11-03 21:20:19.980  7541 12227 D DexMerger$IdMerger: Blacklisted: 14
11-03 21:20:19.980  7541 12227 D Dexterous: DexMerger	mergeTypeLists...
11-03 21:20:20.001  7541 12227 D Dexterous: DexMerger	mergeProtoIds...
11-03 21:20:20.025  7541 12227 D DexMerger$IdMerger: Blacklisted: 6
11-03 21:20:20.025  7541 12227 D Dexterous: DexMerger	mergeFieldIds...
11-03 21:20:20.029  2228  2228 D wpa_supplicant: wlan0: Control interface command 'SIGNAL_POLL'
11-03 21:20:20.044  2228  2228 D wpa_supplicant: CTRL-DEBUG: global_ctrl_sock-sendto: sock=9 sndbuf=163840 outq=0 send_len=48
11-03 21:20:20.092  7541 12227 W System.err: java.lang.IllegalArgumentException: Expected an unsigned short: -1
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dex.Dex$Section.writeUnsignedShort(Dex.java:673)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dex.FieldId.writeTo(FieldId.java:59)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger$5.write(DexMerger.java:633)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger$5.write(DexMerger.java:607)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger$IdMerger.mergeSorted(DexMerger.java:353)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger.mergeFieldIds(DexMerger.java:635)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger.mergeDexesMethods(DexMerger.java:201)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger.mergeMethodsOnly(DexMerger.java:244)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeCodeLibReference(Dexterous.java:234)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeCodeLibReference(Dexterous.java:220)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeMethodIds(Dexterous.java:107)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeCodeLib(Dexterous.java:121)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.artist.artistgui.instrumentation.stages.InstrumentationStagesImpl.mergeCodeLib(InstrumentationStagesImpl.java:182)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.artist.artistgui.instrumentation.InstrumentationTask.run(InstrumentationTask.java:69)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.lang.Thread.run(Thread.java:761)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.