Plaintext backup support?

[sequc82]

Hi @bepaald

First, thank you for your work in creating this tool!

Second, is it possible to use this tool on older plaintext backups, from back when Signal still exported them as XML?

[bepaald]

Hi! This tool currently does not work with the old XML backups (at least not as input, it does output them).

The XML format was quite simple if I remember correctly, it should not be too hard to maybe import such a file into an existing backup, after which any other function of this program should work on that backup file. Was there any specific functionality you were looking for?

Do note that the XML backups were (very) lossy. If I'm not mistaken (it's been a long time, I'd have to investigate to be sure) not only did they leave out messages (all outgoing group messages, and all messages with a media attachment (including the text part)), it also permanently altered the context of conversations as incoming group messages are indiscernible from the 1-on-1 messages of that specific contact.

Any work on this will take a little while though, I'm quite busy at the moment and the time I do have I'm spending on fixing existing functionality first (changes in Signal Desktop have broken some functions).

[bepaald]

Any chance #230 has made this feature request obsolete?

[sequc82]

Hi @bepaald,

It may be obsolete for me, if I can get the DB version 23 backup mentioned in #230 to work, but I'm not sure if those two backups overlap yet.

Thanks!

[sequc82]

Hi @bepaald,

To follow up here, it looks like the DB version 23 backup I referenced in #230 did not overlap the old plaintext backups, so I would still find value in this enhancement, though I acknowledge it would be a rather low impact value add to your project overall, and with the breaking changes from Signal Desktop, I completely understand it to be a low priority change.

I know it can be time consuming to provide the background to get a new dev up to speed on an existing project, but I'd be happy to offer support where possible, when I have time!

[bepaald]

Ok, no problem. I will get on this when I have some time. I don't think it'll take too long (famous last words).

I think your support will be most useful by testing what I come up with, I'm sure I won't get it completely right immediately.

Still, an important question:
Is the goal here to just have a nicely formatted version of the XML contents (like --exporthtml), or to actually get a valid backup-file that can be restored by Signal Android?

In case of the latter, this tool will not be able to create any contacts, they need to already exist in the input-backup file into which the XML will be imported. In case of the former, this is not necessary, I can just insert (incomplete) contacts into the database, but the backup will not be valid.

[bepaald]

I've been making steady progress on this. Nothing in the repo yet, but locally I've actually written an XML parser, and a class that imports data from SignalPlaintextBackup.xml into an sqlite database. I'd say I'm making good progress.

Just another quick question. I know the backup file format has changed quite a bit over time, but I don't know if that has also happened with the plaintext backups. I only have one single plaintext backup from around april 2018, but I don't know if it looks the same as yours. Could you confirm the general structure of the XML file is this:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!-- File Created By Signal -->
<smses count="20659">
 <sms protocol="0" address="+[phone-number]" contact_name="[name]" date="1472921043299" readable_date="Sat, 03 Sep 2016 18:44:03 GMT+02:00" type="2" subject="null" body="[message body goes here]" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 [...more sms tags...]
</smses>

Do you have the same two element types (smses (once) and sms (repeated))? Do you see any extra or missing attributes in your file?

Thanks!

[bepaald]

An update:

I think this function is more or less complete (at least until it is tested).

---

• To import a plaintext backup into an existing Android backup:

./signalbackup-tools [android backup] [passphrase] --importplaintextbackup [SignalPlaintextBackup.xml] -o [output]

The import attempts to automatically match the contacts in the XML file to those in the Android backup, first by name, then by phone number.

Note, due to the age of the plaintext backup, it is not unlikely the matching fails. In my testing, I had contacts in the plaintext backup that are no longer on Signal. For these, it's possible the contact has split into two in the Android backup, one with only a phone number and one with a UUID. It is important the contacts are matched correctly (the one with the UUID), or Signal will likely crash when attempting to restore.

The matching can be done manually as well. To list the contacts in the xml file, use ./signalbackup-tools --listxmlcontacts [SignalPlaintextBackup.xml]. To list the contacts in the Android backup, use --listrecipients (note the _id column). Then the matches can be specified manually by adding --mapxmlcontacts "+1234567890=3,+[more_phone_numbers]=[more_ids],..." to the --importplaintextbackup command, where the phone number is from the XML file, and the id is from the Android backup.

If contacts can not be matched, but importing is desired anyway, add --addincompletedataforhtmlexport. As the name implies this will not yield a valid Android backup, but the resulting backup can be passed to --exporthtml (or --exporthtml can be used in the same command, without even using -o [ouput]).

---

• To export a plaintext backup to html (without an Android backup):

Obviously, the output backup from the above procedure can be passed to --exporthtml as desired, but if you want to do this without an existing Android backup, use

./signalbackup-tools --exportplaintextbackuphtml [SignalPlaintextBackup.xml] [HTMLOUTPUTDIR]

This command also supports most of the options that --exporthtml does (see here).

Additionally, both the above options support the options --(no-)xmlmarkdelivered and --(no-)xmlmarkread to mark the imported messages as read (default=false) or delivered (default=true), as this information is not available in the XML backup.

---

Note, in addition to the limitations of the plaintext backup I mentioned in an earlier message, I also noticed Signal-calls are in the backup and not distinguishable from other messages (except the message body "Called Bob", may be rare for real messages).

---

That was a lot to type, I hop it is somewhat clear. Let me know if anything needs clarification. Otherwise test it whenever you find the time.

Thanks!

[bepaald]

ping

[sequc82]

@bepaald

I am so sorry I hadn't gotten back to you sooner. Unfortunately, life caught me by surprise, and now I'm finally able to check back in. Looks like you've been busy while I've been gone! Looking at what I have, it looks like the XML structure is the same as you describe, at least back to 2017.

To answer your earlier question, my original goal was just to format the old XML the same as --exporttohtml but it looks like you've gone above and beyond! I will give it a test as soon as I can.

[bepaald]

No problem, that's the way things go sometimes. Even if I had closed this issue as stale, it could always be re-opened.

Take your time, and let me know whether things work as you would want or if it needs more work. Thanks!