Silence XML plaintext backup conversion?

[JoeGretch]

Silence XML plaintext backup conversion?

[tmo1]

I'm not promising to do it, but I'll consider it. The format doesn't look too complicated. Are there MMS messages, or only SMS?

[JoeGretch]

I do not think Silence exports MMS successfully. There certainly are no image or other files associated with these messages.

Thank you. I appreciate your just looking at it.

[tmo1]

A conversion script is now available here (documented here). It works on the sample XML you provided, but please test and report back here.

[JoeGretch]

Thank you!

Unfortunately, emojis, seem to break the Element Tree parser. There weren't any emojis in the sample I sent you; I apologize. This is the result I got:

xml.etree.ElementTree.ParseError: reference to invalid character number: line 12, column 93

and this is line 12 from my file:

<sms protocol="0" address="[phone]" date="1601816103259" type="2" subject="null" body="&#55357;&#56397;" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />

I might just manually strip out the lines with emojis and see if I can make the file work that way, but I wanted to let you know how the initial test went.

Thank you again.

[tmo1]

I looked into this, and it turns out that the problem is that Silence is emitting invalid XML - this was first reported seven years ago! You can try the entityfixer code posted by Calvin-L in that thread.

[DavidHenryThoreau]

@tmo1 Can you implement entityfixer in the https://github.com/tmo1/sms-ie/blob/master/tools/silence-convert.py ?

[DavidHenryThoreau]

./silence-convert.py SilencePlaintextBackup.xml
Traceback (most recent call last):
  File "/tmp/l/./silence-convert.py", line 46, in <module>
    tree = ET.parse(input_file)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 75, column 108

[DavidHenryThoreau]

Nice feature, SMS are imported with emoji 👍

Do you think MMS can be save and restore ?
SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

[tmo1]

Nice feature, SMS are imported with emoji 👍

I'm glad it's working!

Do you think MMS can be save and restore ?
SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

[DavidHenryThoreau]

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

MMS are stored in /data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms

These files are encrypted :

file part1001235047216594581.mms
part1001235047216594581.mms: data

[tmo1]

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

[DavidHenryThoreau]

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

I'd like to decrypt a file first , in order to see what file it's and then post it.

Here's the MMS database structure :
https://git.silence.dev/Silence/Silence-Android/-/blob/master/src/org/smssecure/smssecure/database/MmsDatabase.java

The keys to decrpyt are located in :
org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

[tmo1]

Thanks. I'm not planning to comb through the code to figure out the database structure and encryption scheme, but if you make progress on it, please post your results here.

[dngray]

I created a tool to do this in python to txt format https://github.com/dngray/silence-backup-parse/blob/main/silence_backup_parse.py there were some specifics regarding emojis.

as for MMS those are not exported in the XML file.

[DavidHenryThoreau]

@dngray Nice, emoji and SMS are well exported by using @tmo1 source code
What is missing :

• SMS with more than 2 people (which is not present in the XML)
• MMS see my previous messages

[dngray]

SMS with more than 2 people (which is not present in the XML)

That's probably because those are actually MMS messages and not SMS.

[DavidHenryThoreau]

Yes, I don't have found what kind of algo is used to encrypt MMS files :
/data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms
Do you have any ideas about these files ? How to decrypt them ?
Keys are locate in :
org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

[tmo1]

That's probably because those are actually MMS messages and not SMS.

Correct. SMS messages cannot have more than one recipient - messages sent to more than one recipient are automatically sent as MMS messages, even when they're only simple text messages with no "media" involved.

[Pommede]

Hi
sure i'm a stupid
but i tried
python silence-convert.py SilencePlaintextBackup.xml
and i've got:
Traceback (most recent call last):
File "/home/user/silence-convert.py", line 46, in
tree = ET.parse(input_file)
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
tree.parse(source, parser)
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
self._root = parser.parse_whole(source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 4, column 116

then i did like recommended:

python silence-xml-fixer.py SilencePlaintextBackup.xml SilencePlaintextBackupfixed.xml
and i've got nothing happened..after 15min...i quit..

and
python3 silence_backup_parse.py SilencePlaintextBackup.xml

and i've got a SilencePlaintextBackup1.zip (as i wrote in the python file) but empty file....

somebody can help me ??? :-(
thks !

[DavidHenryThoreau]

@Pommede https://github.com/tmo1/sms-ie/blob/master/tools/Tools.md#silence-convertpy

[Pommede]

that's what i did, (first command) or something wrong on my command?

[DavidHenryThoreau]

xml.etree.ElementTree.ParseError: reference to invalid character number: line nnn, column mmm

If this error is encountered, first use the XML fixer tool to produce valid XML:

silence-xml-fixer.py < silence-xxx.xml > silence-xxx-fixed.xml

then run the converter on the fixed XML:

silence-convert.py <silence-xxx-fixed.xml>

[Pommede]

thank for your response but i did all that (it's my text.. :-()

[DavidHenryThoreau]

You've missed < and >

[Pommede]

yeah the first command worked
python silence-xml-fixer.py <SilencePlaintextBackup.xml> SilencePlaintextBackupfixed.xml
but the second
no <> if i put those i've got
fish: Expected a string, but found end of the input

but python silence-convert.py SilencePlaintextBackupfixed.xml
i've got a zip with something....i'll try to another sms app
thks

[Pommede]

it's workin !! :-)
many thks.
no mms but it's a silence issue ?

[DavidHenryThoreau]

MMS are not located in the XML still working about it

[Pommede]

oki, very good job, i was afraid to be stuck with silence rrrr

[DavidHenryThoreau]

I'm looking at org.smssecure.smssecure/shared_prefs/Secure/SMS-Preferences.xml it mentions curve25519

Rapid search can lead to :
https://github.com/topics/curve25519
The rage project may can decrypt file ??

-d, --decrypt Decrypt the input.
https://github.com/str4d/rage#passphrase-protected-identity-files
https://github.com/str4d/rage

[tmo1]

Silence / SMSSecure uses the Signal encryption protocol, which uses / can use curve25519 in its operation. I think that this may be only applicable to the encryption used for messages in transit, though, and not to the encryption used for the on-disk message databases.

There are various tools for decoding the encypted backups produced by Signal, including signalbackup-tools and signal_for_android_decryption, but they are not designed to be used with the raw database files retrieved from the Android filesystem.

[DavidHenryThoreau]

There's also ; that may be needs to be adapted to Silence
https://github.com/alexlance/signal-sms-mms-importer