App data backup and restore itself is not a difficult task but it is difficult to do it in a proper way. In this issue, I shall describe how App Manager is going to handle app backups.
Backup Format
All data is backed up using the standard linux shell commands and is stored as tarball individually based on their directory types. Then the tarball will be compressed with one of the compression methods (gzip for greater compatibility, bz2 might be considered in future). The advantage of this method is that the size of the compressed file will be little to largely less than the size of a zip file and more importantly, the permissions will be kept as is.
Metadata
It is important to preserve some metadata in order to display user information about the package as well as for signature verification and to decide which folder goes to where during the restore process. Every metadata is a key-value pair and is stored as a single json file.
Metadata Keys
label: (String) App Labelpackage_name: (String) Name of the packageversion_name: (String) Version name of the package being storedversion_code: (long) Version code of the package being storedsource_dir: (String) Source directorydata_dirs: (String[]) Data directoriesis_system: (boolean) Whether the app is a system appis_split_apk: (boolean) Whether the app consists of multiple apkssplit_names: (String[]) Name of each splitsplit_sources: (String[]) Source of each split (same order as above)has_rules: (boolean) Whether there is a rules file for this app to apply after restoring the backupbackup_time: (long) Backup time of the package being storedcert_sha256_checksum: (String[]) SHA-256 checksum of the signing certificatesource_dir_sha256_checksum: (String) SHA-256 checksum of the compressed source filedata_dirs_sha256_checksum: (String[]) SHA-256 checksum of the compressed data file (same order as data_dirs)mode: (int) Backup mode
0 - Unencrypted backup1 - PGP encrypted backup
version: (int) Metadata version (preserved for future, current value is 1)apk_name: (String) Name of the apk (usually base.apk)instruction_set: (String) Name of the instruction set (architecture) where the backup is made (e.g. arm64, x86_64, etc.)flags: (Integer) Backup flags
0 - Do nothing (nothing is backed up/nothing to be restored)1 << 0 - Backup up source files1 << 1 - Backup data directories (/data/user and /data/user_de)1 << 2 - Backup external data directories (e.g., /sdcard/Android/data)1 << 3 - Exclude cache (from all data directories)1 << 4 - Backup rules (blocking rules)1 << 5 - Skip signature verifications (applicable only for restore operation)1 << 6 - Backup only apk files (instead of the whole source). See #641 << 7 - Backup obb and media files (e.g., /sdcard/Android/obb, /sdcard/Android/media). See #651 << 8 - Enable multiple backups (the backup will result in a new backup separate from the base backup)1 << 9 - Enable backups for all users
user_handle: (Integer) The user (internally known as user handle) to whom the backup belong to. Additional options are only available in the App Info tabtar_type: (String) Compression method: z for GZip and j for BZip2key_store: (boolean) Whether the app uses Android KeyStore
Storage
Backup data is stored at the user's preferred location (in future) or at /sdcard/AppManager. Each package has its own dedicated folder named after its package name. The folder structure for each package is given below:
.
|-- package.name
| |-- 0
| | |-- data0.tar.gz (data dir at zeroth index)
| | |-- source0.tar.gz (source dir at zeroth index)
| | |-- keystore.tar.gz (optional KeyStore files)
| | |-- rules.am.tsv (optional rules file to be applied after restoring the app)
| | |-- perms.am.tsv (permissions)
| | `-- meta.am.v1
| `-- 10
| |-- data0.tar.bz2
| |-- source0.tar.bz2
| |-- rules.am.tsv
| |-- perms.am.tsv
| `- meta.am.v1
|-- another.package
Backup Process
Backup Process involves creation of the metadata based on user's preferences (described in the design section below). This metadata is then used to backup files. It is probably important to distinguish between system applications as the sources directory for these apps can be /system/app or /system/priv-app in which case typical restoring process may not work as expected. For sources directories, the whole source directory is backed up instead of just apk files. This is helpful for people who use patched odex file, for example. Split apks have a bunch of apk files which requires special attentions (and possibly, workarounds).
Restoring
Restoring is performed only after the verification of the file contents as well as signature verification (if the app is currently installed). In future, the sensitive information such as checksums will be encrypted using the encryption key. Restore process is a bit complicated than the backup process due to several factors mostly involving source files. First of all, it is essential to install/update or even reinstall an app based on the current state of the application: 1) If the app in not installed, it has to be installed first, 2) If it is already installed and the installed version is less than or equal to the app version to be restored, it can be simply updated, 3) Otherwise, the app has to be uninstalled (without deleting data if only apk restore is requested) and then installed again. After applying any one of these operations (the app should be forced closed), the source file has to be directly restored to source directory and data files to data directories. At the same time, pm or cmd package install can easily be utilised for split apks.
UI Design
Backup/restore feature is part of batch ops as well as available individually in the App Info tab. Apps with existing backups have a backup sign at the bottom of their app icon in the Main window as well as the version info. In case of batch ops, a menu item called backup/restore data is added which if clicked opens a new alert dialog consisting of a multichoice list consisting of apk, data, external data, exclude cache and blocking rules (all but external data is enabled by default), and three actions: backup (positive), restore (negative), delete backup (neutral). In case of App Info tab, a single backup/restore option is added in the menu which opens an alert dialog similar to the former. A tag is also displayed in this tab describing the backup app version.
Multiple user support (#70)
App Manager currently doesn't support multiple user. If devices have multiple users, it will simply fail to restore (#69). Support for multiple user will be added in future.
How multiple user support works (out of order)
Currently, only user 0 is supported (it is also hard coded, unfortunately), and the backups are stored at /AppManager/package.name/ as explained above. The idea is that data from other users will be saved in the same way as described but in another directory whose name is the user id. An optional boolean parameter user_handle will be added in the meta file to identify it. At the same time, a flag Multiple users will be added (disabled by default) in the backup/restore dialog (if multiple users are available) so that user can choose to backup/restore data for multiple users.
Limitations & workarounds
In case, the user id doesn't match, the restore process will fail. To solve this issue, an option could be added in settings so that user can choose the replacement for the user. But this is only for Android phones with more than 2 profiles. If there are only two profiles and backup data are also available for two profiles, then the unmatched profile will be automatically marked as matched and data will be restored for that particular user. This is useful for devices with work profile, for example.
Multiple backup support (#87)
This implementation is similar to above except that the folders will begin with non-digit letter/symbols to distinguish between the two. If user chooses to do multiple backup via batch operation, they will be offered to pick a name for the backups (all packages have the same name inside their respective directory). If the name begins with a digit (despite the warning), an underscore will be added as a prefix and/or if the name already exists, a number will be added at the end. Unlike the current behaviour (which is still experimental), each package have folders inside them where the backups will be kept. The 0 folder will be used to keep the backups of the user 0 (this is decided from the current user handle), for example. However, when restoring backups via batch operations, only backups from the user 0 will be restored. In routine operation, user can choose which backups to restore by specifying a name in the backup profile (if the named backup doesn't exist, the restoring will be ignored for that package). Other backups can only be restored from the respective App Info tab.
Misc
Handling large backup files (#60)
FAT32 partition has 4GB file size limit which is a problem for apps with large chunks of data. Although it is highly unlikely for a single app to have such an huge amount of storage, it cannot be ignored. Besides, having a max archive size limit increases portability as a user might be interested in transferring files to computer, cloud or even to different device as part of upgrading or migrating. To achieve that, each archive can be at most 1GB in size. Since we're using tarball, it can be done liek this:
tar -czf - directory/to/backup | split -b 1G - directory/with/backup/prefix.
Extracting this tar is easy:
cat /directory/with/backup/prefix* | tar -xzf - -C /
Since we're already using .tar.gz, during restore, the * will fall back to nothing.
Checksum will be the combination of all of these files.
Platform dependency
Since the whole source directory is being backed up, the app restoration process is largely platform dependent if not handled properly. As stated in #58, oat, lib and lib-32 are largely platform dependant (doesn't matter if the app is bundled or not). Hence, if the platform of the backup doesn't match the current platform, only the app will be restored instead of the entire source directory. To circumvent this problem, two more keys (namely instruction_set and apk_name) are added (see Metadata keys section above for description).
Encryption
Encryption support will be added shortly. Since the backup data are stored in the external directory, it's not secure to store sensitive files without encryption. Therefore, once support for encryption is added, unencrypted backup option will be removed (unencrypted backups can still be restored). Support for encryption may include OpenKeychain (the most secure option), RSA-4096) and AES-256. Passwords have to be set for the latter encryption methods and will not be remembered by App Manager (but will be remembered for a single session ie. as long as the app isn't destroyed if the user chooses to do so).
Data Integrity (fixed)
It is essential to preserve data integrity. When an app backup is requested, the existing backup is deleted before the new backup takes place which is directly written to the same directory. If the new backup fails, there is no way to recover the old backups. The alternative way would be to write backups to a separate directory first and after the new backup is complete, replace the old backup with the new one. This could be done in many ways but an efficient way would be to write the new backups inside the backup directory (for an app package.name the new backup directory is /sdcard/AppManager/package.name~) and upon completion delete the old backup and rename the new directory (ie. mv /sdcard/AppManager/package.name{~,}). This way old backup will not be lost if the new backup fails. Another similar but quite fatal issue occurs when restoring fails as there's no way to recover the original data if this happens. Sadly, there's no easy way to solve it (I can take backup of the current app and its data but it might result in an infinite loop is it fails continuously).
Scheduling (part of #61)
Backup scheduling is considered an important option by many and will be added in a future release.
Split apk support (related to #49)
Support for split apk is added in 5f190a4.
App crashing after restore
Due to several reasons, an app may crash such as:
- If the app is newly installed, uid and guid have to be set properly (fixed in eea79a8 and 38b3747)
- For older platforms, permissions may need to restore despite the fact that permissions are preserved within a tar file (fixed in 3faff88)
- If the app uses Android KeyStore, special care is needed (see #82)
Compression methods
gzip and bz2 (xz isn't supported by toybox and therefore won't be added).
Android Lollipop (API < 23) support
Since toybox is introduced in API 23, tar command is not available before API 23 and needs additional support. A possible solution is to ask users to install busybox tools at preferable location. App Manager is going to supply toybox within App Manager (see #84).
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent