SMSs are not imported in chronological order

[jayb-g]

SMSs are not imported in chronological order

[tom93]

@jayb-g: I built an APK that can be used to fix the conversation timestamps. Unfortunately, because it gets installed as a separate app and doesn't have access to the official app's data, the fix will only propagate to the official app if you clear its data. This shouldn't delete the messages or the blocked number list, but it will wipe the blocked keyword list, custom conversation titles, and the other settings (and maybe additional things that I missed). I recommend backing up the messages beforehand just in case. Alternatively, you can wait until the devs release a version with a proper fix.
The code is in branch "build/timestamp-fixer" of my fork (commit), and here is the APK:
unofficial-messages-timestamp-fixer.apk.zip
(again, note that I'm not a project member so you shouldn't trust me; consider building from source yourself with that patch).
To use this build, unzip and install the .apk, launch the app, make it the default SMS app, then tap Settings > Import messages > Back (without selecting any file).
It should show a message saying "Fixing conversation timestamps..." and then "Finished processing N conversations". You can now close and uninstall the app.
Next, reset the data of the official app (long-tap the launcher > tap (i) > Storage > Clear Data). Now you can open it and make it the default SMS app again.
The conversation timestamps should be fixed (except for threads that only contain MMS).

[jayb-g]

@tom93 I can confirm that this works as expected. Awesome work.

The code is in branch "build/timestamp-fixer" of my fork (commit), and here is the APK:
unofficial-messages-timestamp-fixer.apk.zip

[jayb-g]

@Aga-C done.

It happens again and again. I have tried wiping all the messages and deleting cache for simple SMS app and then reimport messages to default app then again installing the simple SMS app but result is same.

Upon wiping and reimporting to default app, it shows correct order and timestamps. So when simple SMS imports messages from default SMS app, it should load in correct order with all timestamps but that doesn't happen.

So what I want to try is export SMSs to json first, then make it compatible with the schema supported by simple SMS app and then try importing from

[jayb-g]

Anyone can point out to the schema used by simple SMS import/export?

So it would be easier to create something that can be used by the community (temporarily until issue is looked into and fixed by devs,)which exports/converts exported jsons to the schema supported and importable by simple SMS.

[jayb-g]

@Aga-C Thanks, I'll try to use this for a conversion script meanwhile its fixed in the app.

[sudwhiwdh]

Unfortunately, the last version of Simple SMS Messenger only produced binary files for me when exporting my messages for an operating system migration.
That's why I used https://f-droid.org/packages/com.github.tmo1.sms_ie/ and now have the data as a .zip file.
Due to this issue, I have just ended up back with the standard SMS messaging app from Android, because the error does not occur there when re-importing.
I would be very happy if I could use Fossify Messages in the future.
How can this issue be solved technically?

[sudwhiwdh]

Support for another import format would be a small additional concern for me.

My main concern for this issue is a different one. Namely, that when I import messages through Fossify Messages and have selected this app as the new default application for messages, the messages should be sorted by date and time received/sent and not by import time.

So far, I assume that the messages land in the system memory after an import, for example via SMS import / export.
The standard Messages app can correctly read this information, whereas Fossify Messages cannot yet do this.

And this is what my question refers to. What does it take to make this possible in Fossify Messages as well?

[tom93]

@jayb-g I haven't opened a PR yet (still haven't done the cleanup); also blocked on #149 which needs to be merged first.

[Aga-C]

This was issue 802 of Simple Messages
Please refer to it for further details and discussions.

Simple SMS Messenger repository can be deleted any time, so please copy these details here.

[Aga-C]

Anyone can point out to the schema used by simple SMS import/export?

Here are Kotlin files with classes used during export to JSON:

• https://github.com/FossifyOrg/Messages/blob/master/app/src/main/kotlin/org/fossify/messages/models/ExportedMessage.kt
• https://github.com/FossifyOrg/Messages/blob/master/app/src/main/kotlin/org/fossify/messages/models/MmsBackup.kt
• https://github.com/FossifyOrg/Messages/blob/master/app/src/main/kotlin/org/fossify/messages/models/SmsBackup.kt

They are serialized with Gson, so schema is the same as these types.

[ikanakova]

Note for those switching from SMT: Export and Import between Simply Messages and Fossify Messages works without this issue.

[chesio]

Note for those switching from SMT: Export and Import between Simply Messages and Fossify Messages works without this issue.

If you don't switch to another phone as well, then there's no need to run export and import. Just install Fossify Messages, set it as default SMS app and remove Simply SMS Messagenger. All messages will be available in Fossify Messages as they are stored in phone SMS storage anyway.

[jayb-g]

@Aga-C Thanks, I'll try to use this for a conversion script meanwhile its fixed in the app.

@sudwhiwdh this is the only temporary solution until its fixed by devs. A script that converts the exported SMSs(by SMS Import Export) to a json format compatible with Fossify Messages Import.

[jayb-g]

@sudwhiwdh Probably some bug in the way fossify messages (mis)reads the messages db.

Hopefully someone who can go through the code of SMS Import Export and fossify messages can compare and pinpoint the difference.

[jayb-g]

I opened the ndjson and saw that SMSs that have this problem of incorrect date(ie date of import) are all first exported, then wiped, then restored using sms-ie; all have entries sub_id: -1 and creator: com.github....

Whereas other messages that appear to show correct date time, all have sub_id: 1 and creator: com.android.messaging

These are the only differences in metadata.

I think it probably has to do something with sub_id: -1 not being addressed in the right way in fossify messages, although its fine in default stock SMS app.

Will try changing all sub_id to 1 and see if it works.

[jayb-g]

Worth mentioning, even though timestamps are shown to be of the time imported in the list view; when you perform a search and list is filtered it shows message times correctly! Maybe this will give devs a hint what is wrong

[Abcd1234-dot]

As you can check in my open issue (not sure if it is similar or different fundamentally than yours), @tom93 has fixed the timestamp issue for importing backups for SMS (in PR). @Aga-C correct me if I am wrong.

[Aga-C]

@Abcd1234-dot Fix from e006a91 could fix it (I haven't tested it), but I don't see it in any open PR.

[tom93]

WARNING: If you are going to build from source, don't try to use commit e006a91 in isolation -- it would cause havoc unless you also include #149.

[tom93]

To add some details, the issue we came across was that, after importing messages from a SMT/Fossify .json backup, the conversation dates shown in the main view were the date of the import rather than the date of the last message in the conversation (details). It appears to be the same as this issue.

The root cause is that when a message is inserted, Android sets the conversation date to the current time rather than the time of the message (source). I found a workaround, and I'm planning to open a PR but I want to do some more work first (make it faster and document it).

Note that the way I intend to implement the workaround will only apply to newly-imported messages. If you already imported a .json backup (using the current release), and you try to re-import the same .json backup using a future release with my planned workaround, then it won't update the conversation dates because existing messages are skipped during import. So if you want to fix the dates on your existing conversations, you can use my build from #88 (comment) and re-import the .json file (but note that I'm not a project member so you shouldn't trust me). I can also make a build that will update all conversations (without having to import anything), let me know if that would be useful.

Finally, note that Android has another bug that breaks my workaround on conversations containing only MMS (details).

[jayb-g]

The root cause is that when a message is inserted, Android sets the conversation date to the current time rather than the time of the message.

@tom93
Even though the default AOSP shows conversation timestamps correctly after import. So I highly doubt that's the case.

I can also make a build that will update all conversations (without having to import anything), let me know if that would be useful.

That would be super useful. Or even better if devs can just include it to fix existing conversations in the next release.

[tom93]

Even though the default AOSP shows conversation timestamps correctly after import. So I highly doubt that's the case.

I believe the AOSP Messages app calculates the conversation date & snippet itself instead of using the values reported by the Telephony provider. Here is the code in the Telephony provider that sets the date to the current time: https://android.googlesource.com/platform/packages/providers/TelephonyProvider/+/android14-release/src/com/android/providers/telephony/MmsSmsDatabaseHelper.java#134

Or even better if devs can just include it to fix existing conversations in the next release.

My thinking was that updating all conversations can be slower than updating just the conversations affected by the import, so ideally the import should only update the affected conversations. But we'll see what the devs think. One option is to update all conversations as a one-off using a migration, for the benefit of users who imported using an older version of the app.

[jayb-g]

I believe the AOSP Messages app calculates the conversation date & snippet itself instead of using the values reported by the Telephony provider.

@tom93 Agree, then why not fossify messages can do the same? AOSP messages is also open source.

One option is to update all conversations as a one-off using a migration, for the benefit of users who imported using an older version of the app.

Yes. That would be good

[tom93]

then why not fossify messages can do the same?

Calculating the conversation date & snippet is more work (and will probably be less efficient). It may be worth it though, considering the Android bug with MMS-only conversations (also, Android's snippet for MMS is bad and could be improved). We'll have to see what the devs think.

[jayb-g]

@tom93 at least now there's some hope that the issue might be resolved.

[tom93]

Just to be clear, the main issue (incorrect dates on import) can be fixed using the simple workaround that I'm proposing.
Calculating the conversation date & snippet on the app's side is only necessary to work around a second Android bug that only affects MMS-only conversations (and to optionally to improve the snippets for MMS).

[jayb-g]

the main issue (incorrect dates on import) can be fixed using the simple workaround that I'm proposing.

My issue was mainly related to SMSs import only. I'm not sure if its interconnected with the MMS related bug or not. Or if it has to be fixed separately. When I encountered the bug I probably didn't have any MMS in my data. But not 100% sure.

Can you please create a PR for the fix you are proposing for import-timestamp issue? If devs agree then maybe we all get relief.

[Abcd1234-dot]

My issue was mainly related to SMSs import only. I'm not sure if its interconnected with the MMS related bug or not. Or if it has to be fixed separately. When I encountered the bug I probably didn't have any MMS in my data. But not 100% sure.

@jayb-g I had many tests with latest Fossify messages app, you may read the whole conversation on my open issue which also includes discussion about incorrect timestamp. I and tom93 confirmed timestamp issue for SMS and MMS while importing backup. He resolved the issue for SMS already, he is currently working on properly documenting the issue and make workaround more efficient and smooth. He will open PR when he will be ready.

[jayb-g]

@Abcd1234-dot yes of course he can if he wants to and whenever he is ready.

[jayb-g]

@tom93 that sounds good. Will definitely try it

[jayb-g]

Awaiting the fix to be merged/implemented in the main app

[jayb-g]

@tom93

I can confirm that this works as expected. Awesome work.

So I guess this is the only option until it gets committed