Crash trying to import

[tmo1]

Crash trying to import

[tmo1]

Hi,

The problem described in #11 that I fixed should arise only when the Android version of the system from which the messages were exported is different from the version of the system to which they are being imported. Are all your systems Android 11? And yes, a logcat would be very helpful.

[mmorais-idealsumario]

Are all your systems Android 11?

No, in fact. Probably best to explain what I'm trying to do.
I have two mmssms.db: a very old one, probably Android 6 or 7, can't remember; and another one from Android 8. Meanwhile, I have a fresh A11 to which I'm trying to merge and import both dbs. Now, the reason I'm here is that, as you know, the database schema varies between the three. A quick search didn't give me any info on the schema changes, so before diving into the sql myself to merge the dbs directly, I searched for apps to do that or to bypass it entirely, and found yours.

What I was trying to do with your app is then to copy both mmssms.db backups, in turn, to the new A11 and use your app to export them to .json. This worked, as A11 seems to be able to parse different schemas. So now I have the two .json to match each database. I then removed A11's mmssms.db so it would recreate a new one at boot, which it did, and then tried to use your app to import both .json. This is where it failed.

And yes, a logcat would be very helpful.

Ok so, here's the relevant logcat on importing the .json of the oldest db:

01-04 12:23:29.598 2440 2750 E SQLiteLog: (1) table sms has no column named phone_id in "INSERT INTO sms(thread_id,phone_id,priority,address,locked,protocol,status,sub_id,date_sent,pri,body,date,read,seen,type,creator,error_code,reply_path_present,service

01-04 12:23:29.599 2440 2750 E SQLiteDatabase: Error inserting thread_id=1 phone_id=0 priority=-1 address=xxx locked=0 protocol=67 status=-1 sub_id=1 date_sent=1514810995000 pri=-1 body=xxx date=1514812883074 read=1 seen=1 type=1 creator=com.github.tmo1.sms_ie error_code=0 reply_path_present=0 service_center=xxx

01-04 12:23:29.599 2440 2750 E SQLiteDatabase: android.database.sqlite.SQLiteException: table sms has no column named phone_id (code 1 SQLITE_ERROR): , while compiling: INSERT INTO sms(thread_id,phone_id,priority,address,locked,protocol,status,sub_id,date_sent,pri,body,date,read,seen,type,creator,error_code,reply_path_present,service_center) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)

And from the other:

01-04 12:25:29.195 2440 2879 E SQLiteLog: (1) table sms has no column named priority in "INSERT INTO sms(thread_id,priority,address,locked,protocol,status,sub_id,date_sent,body,date,read,seen,type,creator,error_code,reply_path_present,service_center) VALU

01-04 12:25:29.196 2440 2879 E SQLiteDatabase: Error inserting thread_id=2 priority=-1 address=xxx locked=0 protocol=0 status=-1 sub_id=1 date_sent=1640772008000 body=xxx date=1640772010145 read=1 seen=1 type=1 creator=com.github.tmo1.sms_ie error_code=0 reply_path_present=0 service_center=xxx

01-04 12:25:29.196 2440 2879 E SQLiteDatabase: android.database.sqlite.SQLiteException: table sms has no column named priority (code 1 SQLITE_ERROR): , while compiling: INSERT INTO sms(thread_id,priority,address,locked,protocol,status,sub_id,date_sent,body,date,read,seen,type,creator,error_code,reply_path_present,service_center) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)

I've replaced private data with 'xxx'

Further info that may be helpful:

mmssms.db-1 (A6 or A7):

schema version/user version: 112/66

mmssms.db-2 (A8):

schema version/user version: 44/68

mmssms.db-new (A11):

schema version/user version: 49/68

If you could upload the new apk with the fix, I'd appreciate it :)
Thanks again and good work!

[mmorais-idealsumario]

Just to add another test I've done, I've edited the .json files and removed fields not present in A11's db schema -- essentially 3 fields, "phone_id", "priority" and "pri" -- so the app doesn't crash on import.

The thing is, since I can't find any documentation on the db schema and its changelog, I don't know the relevance of the data on those fields, so this isn't a proper migration. I wonder what's the proper way to do it...

[tmo1]

Yes, it looks like you're seeing the problem I just fixed. I, too, could not easily find documentation of the evolution of the database schemas, so my solution was to modify the import code to look up the list of columns in the local versions of the databases, and then to just drop all fields in the JSON that aren't present in those local versions.

A fixed version for testing is available here.

Update: I realize that my fix is just a programmatic way of doing what you did manually, and that a more sophisticated solution would entail actually figuring out what to do with the values in those fields. That would take a lot of detective and coding work, however, especially in light of the aforementioned lack of clear documentation of the evolution of the database structure, so at least for now we're just going to have to assume that those fields aren't critical and ignore them.

[tmo1]

Duplicate of #11

[mmorais-idealsumario]

Update: I realize that my fix is just a programmatic way of doing what you did manually, and that a more sophisticated solution would entail actually figuring out what to do with the values in those fields. That would take a lot of detective and coding work, however, especially in light of the aforementioned lack of clear documentation of the evolution of the database structure, so at least for now we're just going to have to assume that those fields aren't critical and ignore them.

Yeah, guess so :(

Oh just one more thing which may be of interest to you. Since I was working on backups of the .db alone, I didn't have the mms files (dir app_parts if I'm not mistaken?). This is fine for me, since I don't care about them. However, when using your app to export to json with the mms bits included, it crashes because the parts are missing and leaves the json file half done.

Also, when importing an invalid json such as that one, the app will crash too when it gets to the invalid objects.

A fixed version for testing is available here.

Thanks!

[tmo1]

Yeah, guess so :(

Note that the current situation does not seem to be too bad: the fields in which I would assume people are most interested, such as addresses / phone numbers, dates, and message body / text and data parts, have been fairly stable - the current ones were mostly introduced in API 19 (Android 4.4, released in 2013). The fields that have been removed seem to be less interesting / important. If we ever encounter fields that are important to preserve, we can always add code to do some sort of migration.

Oh just one more thing which may be of interest to you. Since I was working on backups of the .db alone, I didn't have the mms files (dir app_parts if I'm not mistaken?). This is fine for me, since I don't care about them. However, when using your app to export to json with the mms bits included, it crashes because the parts are missing and leaves the json file half done.

Yes, the app is designed to run in a coherent / consistent Android environment, and will crash if the parts database is not present (the app does not look in the parts directory directly - it queries the appropriate Content Provider to get the parts).

Note that working directly with the databases themselves is the goal of SMS Import / Export's sibling project, sms-db. That program runs on linux systems (and perhaps in other Perl environments?), and extracts messages from Android databases (of various sorts) into its own standardized database format, from which they can then be exported, currently only as XML. It's supposed to have SMS Import / Export-compatible JSON export capability as well, but I wrote sms-db before writing sms-ie, and I haven't gotten around to implementing JSON export yet. I really should do it, though - it shouldn't be too difficult.

Also, when importing an invalid json such as that one, the app will crash too when it gets to the invalid objects.

Yes - sms-ie as written is rather fragile - it does little error checking / exception handling, and will crash if things are not as it expects. It would be nice to improve its robustness, but this has not been a priority heretofore :/