reagentx/imessage-exporter

Export iMessage data + run iMessage Diagnostics

imessage-exporter

This crate provides both a library to interact with iMessage data as well as a binary that can perform some useful read-only operations using that data.

Binary

The imessage-exporter binary exports iMessage data to txt or html formats. It can also run diagnostics to find problems with the iMessage database.

Installation instructions for the binary are located here.

Library

The imessage_database library provides models that allow us to access iMessage information as native data structures.

Documentation for the library is located here.

Supported Features

This crate supports every iMessage feature as of macOS 14.3.1 (23D60) and iOS 17.3.1 (21D61):

• Multi-part messages
• Replies/Threads
• Attachments
• Expressives
• Reactions
• Stickers
• Apple Pay
• URL Previews
• App Integrations
• Edited messages

See more detail about supported features here.

Frequently Asked Questions

The FAQ document is located here.

Special Thanks

• All of my friends, for putting up with me sending them random messages to test things
• SQLiteFlow, the SQL viewer I used to explore and reverse engineer the iMessage database
• Xplist, an invaluable tool for reverse engineering the payload_data plist format
• Compart, an amazing resource for looking up esoteric unicode details

imessage-exporter's Issues

`AuthorizationForStatementDenied` When connecting to `chat.db`

https://old.reddit.com/r/apple/comments/10cp8dh/_/j4kfavh

	Building cache…

	[1/4] Caching chats…

	thread ‘main’ panicked at ‘called  Result::unwrap()  on an  Err  value: SqliteFailure(Error { code: AuthorizationForStatementDenied, extended_code: 23 }, Some(“authorization denied”))’, /Users/username/.cargo/registry/src/github.com-1ecc6299db9ec823/imessage-database-1.0.0/src/tables/chat.rs:34:54 note: run with  RUST_BACKTRACE=1  environment variable to display a backtrace

Support parsing different types of app messages

balloon_bundle_id has some app developer metadata and payload_data has some plist data we can look at for specific apps like Apple Pay that contain information useful to the export.

Choose conversations to export

This came from feature request #51.

Right now, imessage-exporter exports every conversation it can see. It could be useful to specify specific conversation(s).

The easiest way to accomplish this is to know the chat's ID in the table, but even that may not be correct due to the duplication problem. We can of course show this to the user, but selecting from a list in a reasonable manner is an interesting UX problem.

I don't want to emit a huge list and expect the user to read through it, there must be a way to live-search it.

Implement `csv` Dump

Feature request: turn off lazy loading option

Hi, I love this tool - many thanks!

I am using this to generate pdf versions of message threads and didn't have any luck with wkhtml but found with some extra CSS I can get a very good result printing to pdf from Safari - however, in longer threads, many images get skipped unless I remove loading="lazy" so it would be nice to have a --no-lazy switch.

Support SharePlay Messages

These will likely work the same as older app messages.

Support Handwritten messages

Details are here: https://support.apple.com/en-us/HT206894

Null value error on txt export

To preface: I'm not familiar with rust, but will try to provide helpful information!

I'm running imessage-exporter -f txt and seeing the following output:

Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!
Exporting to /Users/tassilo/imessage_export as txt...
⠓ [2s] [################>---] 103216/123664 (39,193.6516/s, ETA: 1s)
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /Users/tassilo/.cargo/registry/src/github.com-1ecc6299db9ec823/imessage-exporter-0.1.2/src/exporters/txt.rs:337:42

Attempted to pull a backtrace by running RUST_BACKTRACE=full imessage-exporter -f txt:

stack backtrace:
   0:        0x1041ba34c - std::backtrace_rs::backtrace::libunwind::trace::hbe97f4303076f99a
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x1041ba34c - std::backtrace_rs::backtrace::trace_unsynchronized::hb462a91ac605b16b
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x1041ba34c - std::sys_common::backtrace::_print_fmt::h555c176cf2344151
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x1041ba34c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he84d5e26d26c914d
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x1041d526c - core::fmt::write::h07c7e4739f78f3b0
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/fmt/mod.rs:1209:17
   5:        0x1041b6f5c - std::io::Write::write_fmt::h4a3c5f094ffc65e8
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/io/mod.rs:1682:15
   6:        0x1041ba160 - std::sys_common::backtrace::_print::h03a1c9ac6001091a
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:47:5
   7:        0x1041ba160 - std::sys_common::backtrace::print::h6fc2cc8be2392a8e
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:34:9
   8:        0x1041bb7b4 - std::panicking::default_hook::{{closure}}::h3bb44ba6ff437e23
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:267:22
   9:        0x1041bb50c - std::panicking::default_hook::h92c6668a1a98ae8c
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:286:9
  10:        0x1041bbdf0 - std::panicking::rust_panic_with_hook::h9b8acc949e1fb807
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:688:13
  11:        0x1041bbbc4 - std::panicking::begin_panic_handler::{{closure}}::hfc38b2cc56e3623c
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:577:13
  12:        0x1041ba7b4 - std::sys_common::backtrace::__rust_end_short_backtrace::hbafd151207fc448f
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:137:18
  13:        0x1041bb954 - rust_begin_unwind
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
  14:        0x1041e8b38 - core::panicking::panic_fmt::h8c7f3d540a75578f
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
  15:        0x1041e8bcc - core::panicking::panic::h2505ca165b8854d6
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:115:5
  16:        0x1040f9764 - <imessage_exporter::exporters::txt::TXT as imessage_exporter::exporters::exporter::Writer>::format_message::h1bcefd6001801411
  17:        0x1040f69c0 - <imessage_exporter::exporters::txt::TXT as imessage_exporter::exporters::exporter::Exporter>::iter_messages::h5ed45f1370a5180d
  18:        0x1041017ec - imessage_exporter::app::runtime::Config::start::h1b7f6ac2978263f9
  19:        0x1040f2908 - imessage_exporter::main::h38ca28207216a674
  20:        0x1040fe79c - std::sys_common::backtrace::__rust_begin_short_backtrace::h85560146c1c1ecb9
  21:        0x104101f00 - std::rt::lang_start::{{closure}}::hb95f493d3c11ff53
  22:        0x1041b223c - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::ha4b10a239e2af884
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:286:13
  23:        0x1041b223c - std::panicking::try::do_call::h6b4bcb7d3635e86a
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:483:40
  24:        0x1041b223c - std::panicking::try::h579c8cca81ff0f69
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:447:19
  25:        0x1041b223c - std::panic::catch_unwind::h4a997c12755a6e33
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panic.rs:137:14
  26:        0x1041b223c - std::rt::lang_start_internal::{{closure}}::hf20057b44f57f87c
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/rt.rs:148:48
  27:        0x1041b223c - std::panicking::try::do_call::hdfca34da16d8863f
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:483:40
  28:        0x1041b223c - std::panicking::try::h4f4022e500de0807
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:447:19
  29:        0x1041b223c - std::panic::catch_unwind::h62721286166676e8
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panic.rs:137:14
  30:        0x1041b223c - std::rt::lang_start_internal::h659a783147314d97
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/rt.rs:148:20
  31:        0x1040f2a3c - _main

Here's the output of imessage-exporter -d:

Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!

iMessage Database Diagnostics

Messages not associated with a chat: 68
Missing attachment data:
    Missing files: 24
    ck_server_change_token_blob: 7
Duplicated contacts: 63
Duplicated chats: 47
Done!

Running imessage-exporter -f html works fine and it looks like the message triggering the error is this one (it's the next message at the point of failure):

Version for reference:
imessage-exporter --version: iMessage Exporter 0.1.2

'Unable to open database file' error when specifying the database path option

I'm getting the following error when specifying the database path option:

$ RUST_BACKTRACE=full ./imessage-exporter-aarch64-apple-darwin -o /tmp/imessage/out/ -f html -p /tmp/imessage/chat.db 
Building cache...
[1/4] Caching chats...
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: SqliteFailure(Error { code: CannotOpen, extended_code: 14 }, Some("unable to open database file"))', /Users/chris/.cargo/registry/src/github.com-1ecc6299db9ec823/imessage-database-1.0.0/src/tables/chat.rs:34:54
stack backtrace:
   0:        0x10415cdf8 - __mh_execute_header
   1:        0x1040c184c - __mh_execute_header
   2:        0x10413c988 - __mh_execute_header
   3:        0x1041600b0 - __mh_execute_header
   4:        0x10415fd10 - __mh_execute_header
   5:        0x104160be0 - __mh_execute_header
   6:        0x104160768 - __mh_execute_header
   7:        0x1041606dc - __mh_execute_header
   8:        0x1041606b8 - __mh_execute_header
   9:        0x1041725c8 - __mh_execute_header
  10:        0x104172798 - __mh_execute_header
  11:        0x1040f7bcc - __mh_execute_header
  12:        0x1040ea5e0 - __mh_execute_header
  13:        0x1040f217c - __mh_execute_header
  14:        0x1040e8560 - __mh_execute_header
Abort trap: 6

I can't see any more information on why the file couldn't be opened; as far as I can see there's nothing in the permissions leading up to and including the file that would deny access:

$ stat /tmp
  File: /tmp -> private/tmp
  Size: 11        	Blocks: 0          IO Block: 4096   symbolic link
Device: 1,16	Inode: 1152921500312421341  Links: 1
Access: (0755/lrwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/   wheel)
...
$ stat /private/tmp
  File: /private/tmp
  Size: 384       	Blocks: 0          IO Block: 4096   directory
Device: 1,16	Inode: 110015026   Links: 12
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/   wheel)
...
$ stat /tmp/imessage
  File: /tmp/imessage
  Size: 224       	Blocks: 0          IO Block: 4096   directory
Device: 1,16	Inode: 110861970   Links: 7
Access: (0755/drwxr-xr-x)  Uid: (  501/joenyland)   Gid: (   20/   staff)
...
$ stat /tmp/imessage/chat.db 
  File: /tmp/imessage/chat.db
  Size: 16932864  	Blocks: 33072      IO Block: 4096   regular file
Device: 1,16	Inode: 110862025   Links: 1
Access: (0644/-rw-r--r--)  Uid: (  501/joenyland)   Gid: (   20/   staff)
...

I also get the same behaviour if I run this inside my home directory. I'm doing this in /tmp just to keep things separate and also in an attempt to reduce the chances of access being denied through some sort of permissions.

If I run the tool without specifying a different DB, it works fine (therefore using ~/Library/Messages/chat.db). It's just that I want to export from a backed up chat.db file.

I've got two backed up chat.db files from differing macOS versions that give the same error. One from macOS 13.1 and the other from 11.7.2.

Implement `Try` for `Cacheable` trait

imessage-exporter/imessage-database/src/tables/table.rs

Lines 15 to 20 in e223e49

	/// Defines behavior for table data that can be cached in memory
	pub trait Cacheable {
	type K;
	type V;
	fn cache(db: &Connection) -> HashMap<Self::K, Self::V>;
	}

[BUG] video type video/quicktime breaks compatibility with Chromium based browsers

A video with the src set to

<video controls> <source src="attachments/71cba48a-707b-4dee-b494-ba43ef5c0f0c.mov" type="video/quicktime"> </video> will not play in Edge or Chrome.

A video with the src set to

<video controls> <source src="attachments/71cba48a-707b-4dee-b494-ba43ef5c0f0c.mov" </video> play just fine.

From a quick google, this seems to have been or is an issue https://secure.phabricator.com/T13135

Support edited iMessages

iOS 16 and MacOS Ventura add a message editing feature: https://www.macrumors.com/2022/06/06/imessage-unsend-edit-15-minutes/

Update hero image for #64

It is a bit misleading to say "Contact 1" given imessage-exporter does not access macOS Contacts.

Add CLI Arg to replace default "Me" value

ME should be overridden by custom text.

imessage-exporter/imessage-exporter/src/app/runtime.rs

Lines 248 to 252 in 9a40b7b

	/// Determine who sent a message
	pub fn who(&self, handle_id: &i32, is_from_me: bool) -> &str {
	if is_from_me {
	ME
	} else {

imessage-exporter/imessage-exporter/src/exporters/html.rs

Lines 465 to 470 in b61493b

	fn format_annoucement(&self, msg: &'a Message) -> String {
	let mut who = self.config.who(&msg.handle_id, msg.is_from_me);
	// Rename yourself so we render the proper grammar here
	if who == ME {
	who = "You"
	}

[BUG] no image attachements in html format

When generating an archive with HTML files the images don't appear into the conversation. But an attachements directory is correctly created in file structure<;

Tried on intel mac. Golden master binary and cloned repo at current state.

Calculate message date offset once, not for each object

imessage-exporter/imessage-database/src/tables/messages.rs

Lines 209 to 210 in e223e49

	// TODO: Calculate once, not for each object
	offset: Utc.ymd(2001, 1, 1).and_hms(0, 0, 0).timestamp(),

Deprecated `xmp` tag in URL metadata

Some webmasters include HTML in their sites' metadata:

https://darknetdiaries.com/transcript/21/

<meta property="og:description" content="Ira Winkler's specialty is assembling elite teams of special forces and intelligence officers to go after companies. Ira shares a story about a time he and his team broke into a global 5 company. A company so large that theft of intellictual property could result in billions of dollars of damage. <br><br>Ira's consulting company: <a href='http://www.securementem.com/'>Secure Mentum</a>.<br><br>His books: <a href='https://www.amazon.com/gp/product/0764584685/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0764584685&amp;linkCode=as2&amp;tag=tunn01-20&amp;linkId=8f8a26b3c5fe68e6c8913ceea8adb0f5'>Spies Among Us</a>, <a href='https://www.amazon.com/gp/product/0128093161/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0128093161&amp;linkCode=as2&amp;tag=tunn01-20&amp;linkId=ac08090c6d15b5ad2c77789fbcf50d56'>Advanced Persistent Security</a>, <a href='https://www.amazon.com/gp/product/0895263904/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0895263904&amp;linkCode=as2&amp;tag=tunn01-20&amp;linkId=63e40c57a8bf8f04f077f55d6ed21455'>Through the Eyes of the Enemy</a>. ">

In the payload_data that ends up looking like this escaped html code:

If we got the whole string that would be fine, but iMessage truncates this data, so we end up getting some unclosed tags.

Rendering inside of <pre><code>…</code></pre> ends up rendering all of the escaped HTML, so I used an xmp tag in lieu of a code tag here to render the text properly. Since xmp is deprecated, we should do something else here. Ideally we can avoid using Javascript, as we want these pages to be printable to a PDF, and Javascript opens the door to non-printable features.

Support Ventura MacOS message storage

It seems like messages sent from a Mac running Ventura don't store data in the text column. Messages sent from iOS 16 still populate the field.

What is `item_type` 4?

There is an int column in the messages table called item_type. Normal messages are 0, channel announcements that update the room name are 2. I have a third item_type in my database, "aptly" named 4, and it seems to be a fully null message (no text or blob data, no associated message, nothing). What does 4 represent? Is there a 1?

Organize messages and attachments by date

Copying all messages to single files in an output directory and all attachments to an output subdirectory can leave a very large number of files in just two directories. This can slow down directory listings, backups, and file loading.

Instead, messages could be broken up by date. Maybe something like: /YYYY/mm/dd/contact/YYYY-mm-dd_contact.format.

Attachments can be referenced by multiple chats, so they should probably keep the same directory format that they start with (/Attachments/xx/xx/uuid-ish/name.format), but they can be aliased or symlinked in to the same directory as the chat file, maybe named according to the timestamp that they were sent (/YYYY/mm/dd/contact/attachment/YYYY-mm-dd_HHMMSS.format)

Implement `txt` Dump

Support "Richer Collaboration" messages

These will likely work the same as older app messages.

FAQ needed?

For example:

Q: Does imessage-exporter export message conversations that are on a user's iPhone but not mirrored on the user's otherwise in-sync Mac?
A: No, the user will have to ensure that iPhone conversations are each mirrored onto their Mac before they attempt export

Q: Can you run expert again and again? How does the exporter handle previously exported messages
A: imessage-exporter merges in missing messages for each conversation.

Q: If an export is done. then message(s) are deleted through Apple's messages apps, does a subsequent export delete the same messages from the export file set?
A: No, the exporter does not replicate deletes onto a previous exported set of files

Split attachments folder into subfolders

This came from feature request #51.

Right now all attachments get copied to output_dir/attachments per:

https://github.com/ReagentX/imessage-exporter/blob/release/imessage-exporter/src/app/runtime.rs/#L77-L82

When we copy to that location, all we do is drop in the file with a unique name:

https://github.com/ReagentX/imessage-exporter/blob/release/imessage-exporter/src/exporters/html.rs/#L411-L421

It would be better to have separate subdirectories for each conversation to make it easier to find attachments manually.

Use `bundled` feature for `rusqlite`

Windows does not bundle sqlite3 like macOS does.

Enabling the bundled crate feature for rusqlite will prevent build issues on Windows:

error: linking with  x86_64-w64-mingw32-gcc failed: exit code: 1
note: ld: cannot find -lsqlite3

Docs: https://docs.rs/crate/rusqlite/latest

bundled causes us to automatically compile and link in an up to date version of SQLite for you. This avoids many common build issues, and avoids depending on the version of SQLite on the users system (or your system), which may be old or missing.

imessage-exporter/imessage-database/Cargo.toml

Line 15 in c5006a3

rusqlite = {version = "0.28.0", features = ["blob"]}

imessage-exporter/imessage-exporter/Cargo.toml

Line 16 in e699285

rusqlite = {version = "0.28.0", features = ["blob"]}

[Linux] Issue saving attachments, `Conversion failed: No such file or directory (os error 2)`

Problem

I ran your tool like so

$ ./target/debug/imessage-exporter -p ~/Downloads/imessage-exporter/2023-02-13.db -o output/ -f html

It seemed to export messages correctly, creating several html files in the output folder, but failed exporting/saving attachments

In the output I see a few hundred lines of just stuff like this:

Unable to create "output/attachments/304/4956db87-9b80-464e-94a5-661a26e86aef.jpeg" from "/home/raleigh/Library/Messages/Attachments/ce/14/213B4C5E-3C9F-4589-91D3-A9401A0412BD/IMG_6907.jpeg"
Unable to create "output/attachments/304/d6ce16c8-2859-4539-8b9a-c791dd0d1257.mov" from "/home/raleigh/Library/Messages/Attachments/a9/09/EB36B8DA-4D56-49D5-B638-7808142DD6B8/IMG_7156.mov"
Conversion failed: No such file or directory (os error 2)
Conversion failed: No such file or directory (os error 2)
Unable to create "output/attachments/304/0ef19b8c-cb68-4517-9fae-c0d63dc86b2a.jpeg" from "/home/raleigh/Library/Messages/Attachments/2a/10/0BDCE0BE-2A5F-4FFA-B9FE-20E0F82E9FF7/IMG_6908.jpeg"
Unable to create "output/attachments/304/1092e964-0a37-402f-9ff7-d49fc4b65b46.jpeg" from "/home/raleigh/Library/Messages/Attachments/2a/10/0BDCE0BE-2A5F-4FFA-B9FE-20E0F82E9FF7/IMG_6908.jpeg"
Conversion failed: No such file or directory (os error 2)
Conversion failed: No such file or directory (os error 2)
Conversion failed: No such file or directory (os error 2)
Unable to create "output/attachments/304/f5dabb53-6324-4d64-9bca-37932d1a1f1b.mov" from "/home/raleigh/Library/Messages/Attachments/33/03/525C1628-1C28-4F3F-8B70-7EC4D36F3C94/IMG_7163.mov"
Conversion failed: No such file or directory (os error 2)
Conversion failed: No such file or directory (os error 2)
Conversion failed: No such file or directory (os error 2)

Here's what it prints at the beginning:

/target/debug/imessage-exporter -p ~/Downloads/imessage-exporter/2023-02-13.db -o output/ -f html
Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!
Exporting to output/ as html...
⠉ [0s] [--------------------] 0/33210 (0/s, ETA: 0s)                                                                                                            Unable to create "output/attachments/456/c2733db6-ef5a-4f1f-a873-3ca86a214b6c.mov" from "/home/raleigh/Library/Messages/Attachments/e9/09/at_0_B7F7762D-C4A7-47BC-AB0D-DE7036577446/ms-lWcHzt.mov"
Unable to create "output/attachments/440/766e262c-15b8-4628-bdb2-4f8931f77dc7.jpeg" from "/home/raleigh/Library/Messages/Attachments/a6/06/at_0_577842B2-E12A-406C-9275-3758F2713AED/jpeg-image-blPDlo.jpeg"
Unable to create "output/attachments/440/0ab52f8d-f492-46fd-9d2b-94d31e4aa097.gif" from "/home/raleigh/Library/Messages/Attachments/c1/01/at_0_0DE2340B-A108-4DD6-8A17-3119BE39C81C/ms-IW4ge1.gif"
Unable to create "output/attachments/440/7a3e7915-386f-4c60-8b7f-4d571694a3e4.jpeg" from "/home/raleigh/Library/Messages/Attachments/01/01/at_0_E5067AAE-C915-4C3C-9C92-7BC5DE05CB54/jpeg-image-xF2BEf.jpeg"
Unable to create "output/attachments/440/6e215a6e-07f8-4b47-8e20-0baac9eec161.jpeg" from "/home/raleigh/Library/Messages/Attachments/06/06/at_0_2DD11F8F-A851-4E2F-9E82-5D8A9CE93D29/jpeg-image-G1dBJT.jpeg"
Conversion failed: No such file or directory (os error 2)

System information

• Ubuntu 22.04.1 LTS x86_64
• 5.15.0-60-generic
• rustc 1.67.1 (d5a82bbd2 2023-02-07)

Contact Details Not Resolving, Phone Number Only

When I run:
imessage-exporter -f html -o /Users/...

The tool successully exports and builds the export, with one exception - All of HTML files, and chat mentions are titled or addressed using individuals phone numbers, and not their contact name in the MacOS Contacts.

Is this a bug? User error? Or a feature that needs to be developed?

Unnecessary `unwrap()` when writing to file

Trace:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', imessage-exporter/src/exporters/html.rs:654:14

stack backtrace:
   0:        0x100ed84e8 - __mh_execute_header
   1:        0x100e400a4 - __mh_execute_header
   2:        0x100eb8ebc - __mh_execute_header
   3:        0x100edb608 - __mh_execute_header
   4:        0x100edb3f0 - __mh_execute_header
   5:        0x100edbb94 - __mh_execute_header
   6:        0x100edb79c - __mh_execute_header
   7:        0x100edb710 - __mh_execute_header
   8:        0x100edb6ec - __mh_execute_header
   9:        0x100eeb7fc - __mh_execute_header
  10:        0x100eeb9cc - __mh_execute_header
  11:        0x100dfcccc - __mh_execute_header
  12:        0x100df704c - __mh_execute_header
  13:        0x100e78428 - __mh_execute_header
  14:        0x100e69820 - __mh_execute_header
  15:        0x100e71804 - __mh_execute_header
  16:        0x100e67588 - __mh_execute_header

Diagnostic:

Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!

iMessage Database Diagnostics

Messages not associated with a chat: 1256
Missing attachment data:
    Missing files: 7
    ck_server_change_token_blob: 8
Chats with no handles: 2
Duplicated contacts: 89
Duplicated chats: 34
Done!

[Request] Adjust attachment creation/modification date to iMessage timestamp

This would be a super helpful way to possibly organize/consume the attachments. If you took the date/time from the message it was sent/received with and tagged that to the photo. I could find all photos sent/received in April 2019 for example if they had date stamps instead of just the day that they're all defaulted to.

Test that the GUID length is correct in `clean_associated_guid()`

imessage-exporter/imessage-database/src/tables/messages.rs

Lines 484 to 500 in 7d98061

	fn clean_associated_guid(&self) -> Option<(usize, &str)> {
	// TODO: Test that the GUID length is correct!
	if let Some(guid) = &self.associated_message_guid {
	if guid.starts_with("p:") {
	let mut split = guid.split('/');
	let index_str = split.next();
	let message_id = split.next();
	let index = str::parse::<usize>(&index_str.unwrap().replace("p:", "")).unwrap_or(0);
	return Some((index, message_id.unwrap()));
	} else if guid.starts_with("bp:") {
	return Some((0, &guid[3..guid.len()]));
	} else {
	return Some((0, guid.as_str()));
	}
	}
	None
	}

Add updating feature

it would be amazing if you could add a feature where the software can update the last export to include new messages, as right now when the software is ran a second time in the same directory, the file has 2 copies of the text inside.
thanks

Full Disk Access warning logic doesn't always work

Per this comment:

	Building cache…

	[1/4] Caching chats…

	thread ‘main’ panicked at ‘called  Result::unwrap()  on an  Err  value: SqliteFailure(Error { code: AuthorizationForStatementDenied, extended_code: 23 }, Some(“authorization denied”))’, /Users/username/.cargo/registry/src/github.com-1ecc6299db9ec823/imessage-database-1.0.0/src/tables/chat.rs:34:54 note: run with  RUST_BACKTRACE=1  environment variable to display a backtrace

This logic is supposed to emit a readable error when Full Disk Access is not available for the imessage-exporter process:

imessage-exporter/imessage-database/src/tables/table.rs

Lines 45 to 51 in c5006a3

	/// Get a connection to the iMessage SQLite database
	pub fn get_connection(path: &str) -> Result<Connection, String> {
	match Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_ONLY) {
	Ok(res) => Ok(res),
	Err(why) => Err(format!("Unable to read from chat database: {}\nEnsure full disk access is enabled for your terminal emulator in System Settings > Security and Privacy > Full Disk Access", why)),
	}
	}

However, the above error suggests we got to the first SQL query instead:

imessage-exporter/imessage-database/src/tables/chat.rs

Lines 33 to 35 in c5006a3

	fn get(db: &Connection) -> Statement {
	db.prepare(&format!("SELECT * from {CHAT}")).unwrap()
	}

Support Apple Music messages

Implement `pdf` Dump

Implement `html` Dump

Support TypedStream fields

Some info on this archaic type: https://mmems.gitbook.io/calepin/formats-encoding-and-protocols-114/typedstream/typedstream

Lack of compatibility with old iMessage chat db's?

I have a chat.db as well as the attachment folder backed up to a drive, I put it on my desktop, and ran the command as follows:
imessage-exporter --db-path "/path/to/chat_folder/" -f html
And I get the full disk access error -- I tried specifying chat.db at the end of the path too (as I'm not sure if that is appended, or whether the .db file needs to be specified.

Great project though! I've been backing up my iMessage chat db's for years, and have been wanting to make this kind of program for years.

Feature Request: Merge (out of sync) conversations from multiple devices

This is great! Thanks for building this.

I think it would be a good feature to merge conversations from multiple devices in the event that they are not synced. I have had a few problems with iCloud Messages and I am not confident that some very long conversations are 100% in sync.

Maybe you could export a conversation from each device to an intermediate file and use a command line tool to merge multiple conversations, removing duplicates and maintaining the timeline based on time stamps. It might need to calculate an offset (from duplicate messages) so it is able to accurately merge the messages in the event that device times are also not in sync.

Implement Diagnostic for `chat_handle.rs`

Needed to determine how many chats do not exist in the join table.

In group conversations, denote senders by color

`transfer_name` column not present in Attachments table

Thanks for making this. I wish I could give you a slice of 🍰

After running:

imessage-exporter -f html
Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!
Exporting to /Users/myusername/imessage_export as html...

I get this error:

Unable to launch: Failed to parse row: Invalid column type Null at index: 12, name: transfer_name

Any other debugging info I can share?

Panic - Filename too long

The program generates the following panic when exporting messages:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 63, kind: InvalidFilename, message: "File name too long" }', imessage-exporter/src/exporters/txt.rs:250:14

This appears to happen when a group chat name is concatenated to too long of a string.

Feature: Import back to the chat.db?

Not sure if possible, but I could see a need for importing. Ie. if you start a new apple account, etc.

FAQ deleted messages item needs wordsmithing, IMO

Q: Are deleted messages simply erased from the database? A: Yes, this tool cannot recover deleted messages.

delete messages where? .. erased from where? Differentiation between canonical Apple DB (Sqlite - multiple locations) and the imessage-exporter quasi database needed.

Support Mentions

iMessage has a feature where you can mention chat participants:

There is only one column in the database that seems related to mentions: has_unseen_mention. It is just a boolean column, though, so it contains no data on what or who was mentioned. The binary blob columns also do not seem to have any information out of the ordinary. Thus, I am not sure how to proceed with this feature.

Better support for Activity/Workout messages

Workout messages are not sent with any payload or special information, but contain text that looks like one of:

$(kIMTranscriptPluginBreadcrumbTextReceiverIdentifier) completed a workout.
$(kIMTranscriptPluginBreadcrumbTextReceiverIdentifier) earned an achievement.
$(kIMTranscriptPluginBreadcrumbTextReceiverIdentifier) closed all three Activity rings.

Where I assume $(kIMTranscriptPluginBreadcrumbTextReceiverIdentifier) is some information about the contact that sent the message. All of the BLOB fields are NULL for these messages, so we need to check them when writing the text and replace the contact's phone number in the correct place.

If we can avoid scanning each message as we write them, we should. Perhaps we should preflight and cache the IDs of messages that contain this kIMTranscriptPluginBreadcrumbTextReceiverIdentifier string and cache them like we do other types of message information.

Feature: Export from date

Feature request. Export from date. Could make it easier to automate backups, just need exact days, etc.

Feature Request: Allow for export of single conversation (or split attachments folder into subfolders)

Don't want to delete my attachments/turn into a PDF, but there isn't a way of going through the attachments folder and deleting all but one conversation.

Doesn't seem to be in crates.io

Running cargo install imessage_exporter I received this error:

Updating crates.io index
error: could not find `imessage_exporter` in registry `crates-io` with version `*` 

Anything special I need to do?

Release to Brew

imessage-exporter.rb:

class ImessageExporter < Formula
  desc "Export MacOS iMessage data + run iMessage Diagnostics"
  homepage "https://github.com/ReagentX/imessage-exporter"
  url "https://github.com/ReagentX/imessage-exporter/archive/refs/tags/0.1.9.tar.gz"
  sha256 "033789e51e3621a74b19e4a5339f0e0b65c2d33b8c3e367a5f7fbd1cdfa642e7"
  license "GPL-3.0-or-later"

  depends_on "rust" => :build

  def install
    system "cargo", "install", *std_cargo_args(path: "imessage-exporter")
  end

  test do
    assert_equal "imessage is not a valid export type! Must be one of <txt, html>\n",
    shell_output("imessage-exporter -f imessage")
    assert_equal "Diagnostics are enabled; format is disallowed\n",
    shell_output("imessage-exporter -f txt -d")
    assert_equal "No export type selected, required by no-copy\n",
    shell_output("imessage-exporter -n")
    assert_equal "No export type selected, required by export-path\n",
    shell_output("imessage-exporter -o imessage")
    assert_equal "",
    shell_output("imessage-exporter -p imessage")
  end
end