gauteh/abunchoftags

(DISCONTINUED, use gmailieer) experimental tag syncing between gmail tags (using X-Keywords) and notmuch tags

keywsync

attempts to synchronize notmuch tags between the X-Keywords header of messages and the notmuch database.

Disclaimer

This is highly experimental. It will modify you emails. It may totally destroy all your email. Use at own risk.

Things to be aware of

Check out keywsync.hh to see which tags are ignored and how the mappings are done. I also replace / with ., so if you got any tags with . in them it is going to become a mess. This can be controlled with --replace-chars and --no-replace-chars.

Also, there is no support for tags with spaces in them (they should probably be wrapped in quotes at some point). Make sure you have a recent version of OfflineIMAP, some issues with multiple occurences of tags should be fixed there.

Usage

Remote to local

Add all tags from remote to notmuch db:

$ ./keywsync -m /path/to/db -k -p -a -q query

Remove all tags from notmuch db that are not in remote:

$ ./keywsync -m /path/to/db -k -p -r -q query

Do both in one go:

$ ./keywsync -m /path/to/db -k -p -q query

Note:

Typically use a query with tag:new, but since notmuch will not detect a change within a message (say a keyword has been changed on an existing message) it is necessary to run this on all messages now and then to ensure all remote tag changes have been caught.

Local to remote

Add all tags from local to remote:

$ ./keywsync -m /path/to/db -t -p -a -q query

Remove all tags from local to remote:

$ ./keywsync -m /path/to/db -t -p -r -q query

Do both in one go:

$ ./keywsync -m /path/to/db -t -p -q query

Strategy:

Assuming you have fully synced database and you want to synchronize your maildir with the remote maildir:

Note: The query needs to filter messages so that only the messages of the GMail maildir are tested.

See this example: fetch_and_sync.sh.

1. Synchronize tags local-to-remote (-t), now all tag changes done in the notmuch db are synchronized with the message files (preferably using a lastmod: query [1] which catches messages where changes have been done after the revision of the db at the time of the last remote-to-local synchronization)

2. Save the current unix time: $ before_offlineimap=$( date +%s )

3. Run offlineimap to synchronize your local maildir and messages with the remote. According to the offlineimap documentation [0] the X-Keywords flags are synchronized in the same way as maildir flags (whatever that means [2]).

4. Run notmuch new to detect any new or deleted files, or and renames.

5. Synchronize tags remote-to-local (-k) using a query that filters out anything but the maildir in question. Use the --mtime flag to only sync messages that match the query and are modified after offlineimap was run: echo $before_offlineimap.

6. Store the current database revision for the next lastmod search in the local-to-remote step of your next search: $ notmuch_get_revision /path/to/db. Alternatively, store the revision from before the local-to-remote sync. In that way it is possible to detect local changes that happened during the sync. This will re-check the messages that were modified as part of the remote-to-local sync.

Note: notmuch new does not detect message changes that do not include a file addition, removal or rename. Therefore simple changes to the X-Keywords header will not be detected. Use the --mtime query to filter out unchanged files.

Initial sync

If you wish to discard all local tags, start with a remote-to-local sync with a query matching the entire directory.

If you wish to discard all remote tags, start with a local-to-remote sync.

You might be able to preserve some stuff using the --only-add and --only-remove flags during sync. Otherwise dumping notmuch tags and restoring at a later time when you have brought your local copy up-to-date with the remote might work.

Timing

Running a full keyword-to-tag sync on a Macbook Pro with around 55k messages on an encfs volume took 1m48s and used about 108MB of memory.

Running a full tag-to-keyword check on the same message base with 3 changed messages took about 1m5s and 100MB of memory.

References

[0] http://offlineimap.readthedocs.org/en/next/MANUAL.html?highlight=keywords#sync-from-gmail-to-a-local-maildir-with-labels
[1] id:[email protected]
[2] Someone have any documentation anywhere?

Other resources

Access to GMail labels via IMAP extensions: https://developers.google.com/gmail/imap_extensions#access_to_gmail_labels_x-gm-labels
Commit in OfflineIMAP for GMail label support: https://github.com/OfflineIMAP/offlineimap/commit/0e4afa913253c43409e6a32a6b6e11e8b03ed3d9
Original patch for OfflineIMAP: http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970
Special tags for notmuch: http://notmuchmail.org/special-tags/

abunchoftags's Issues

--mtime slow (70s) for 400k messages

23:50 YELP-CHARLIE~[[email protected]] % keywsync -m /Users/charlie/Maildir/ -q 'not path:spodder/**' --mtime 1472423730 -k --no-replace-chars -d
** keyword <-> tag sync
replace chars: false
=> db: /Users/charlie/Maildir
=> direction: keyword-to-tag
=> query: not path:spodder/**
=> note: dryrun!
=> remove double x-keywords header: 1
mtime: only operating messages with mtime newer than: 2016-Aug-28 22:35:30
* db: current revision: 11562648
*  messages to check: 409777
*  query time: 654.258 ms.
=> done, checked: 409777 messages and changed: 0 messages (skipped: 409777) in 57847 ms [cpu], 74.9375 s [real time].

The large pause is after query time and then it completes.

The count of messages almost matches up:

0:09 YELP-CHARLIE~[[email protected]] % notmuch count 'not path:spodder/**'
409442

I run this regularly, and I have not modified 409,777 messages in the last 5 minutes.

Meaning of words in the README

What do you mean by

Note: The query needs to filter messages so that only the messages of the GMail maildir are tested.

in the README? Does that mean that it only works with GMail? What I want to do is the following:

ASTROID A <=> NOTMUCH A <=abunchoftags=> MAILDIR A <=offlineimap=> REMOTE (not necessarily GMail) <=offlineimap=> MAILDIR B <=abunchoftags=> NOTMUCH B <=> ASTROID B.

where A and B are different computers. Would it work?

aur package version

AUR shows an old commit hash.

gmailieer does this better

https://github.com/gauteh/gmailieer - but it is new!

segfault on mail parsing

I know the software is experimental but I wanted let you know that I got a segfault by doing:

keywsync -m mail/ -k -p -a -q "*"

The trace is:

open("mail/Sent/cur/1451701521_7.14042.flocon,U=284,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S", O_RDONLY) = 8
lseek(8, 0, SEEK_CUR)                   = 0
fstat(8, {st_mode=S_IFREG|0644, st_size=5063, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17c932b000
lseek(8, 0, SEEK_SET)                   = 0
read(8, "Date: Fri, 13 Aug 2010 15:10:18 "..., 4096) = 4096
open("/gnu/store/rwvf6xqgsyb8bmpi7rwk9fildnwvzrv5-tzdata-2015c/share/zoneinfo/America/New_York", O_RDONLY|O_CLOEXEC) =
 9                                                                                                                   
fstat(9, {st_mode=S_IFREG|0444, st_size=3545, ...}) = 0
fstat(9, {st_mode=S_IFREG|0444, st_size=3545, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17c932a000
read(9, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 3545
lseek(9, -2261, SEEK_CUR)               = 1284
read(9, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 2261
close(9)                                = 0
munmap(0x7f17c932a000, 4096)            = 0
open("/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/lib/gconv/ISO8859-1.so", O_RDONLY|O_CLOEXEC) = 9
read(9, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\5\0\0\0\0\0\0"..., 832) = 832
fstat(9, {st_mode=S_IFREG|0555, st_size=12600, ...}) = 0
mmap(NULL, 2105432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 9, 0) = 0x7f17c4bd1000
mprotect(0x7f17c4bd3000, 2093056, PROT_NONE) = 0
mmap(0x7f17c4dd2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 9, 0x1000) = 0x7f17c4dd2000
close(9)                                = 0
mprotect(0x7f17c4dd2000, 4096, PROT_READ) = 0
lseek(8, 4096, SEEK_SET)                = 4096
read(8, " une id=E9e comment cela a  p=FB"..., 4096) = 967
read(8, "", 4096)                       = 0
lseek(8, 4096, SEEK_SET)                = 4096
read(8, " une id=E9e comment cela a  p=FB"..., 967) = 967
read(8, "", 4096)                       = 0
close(8)                                = 0
munmap(0x7f17c932b000, 4096)            = 0
stat("mail/Sent/cur/1451701521_7.14042.flocon,U=284,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S", {st_mode=S_IFREG|0644,
 st_size=5063, ...}) = 0                                                                                             
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x48} ---
+++ killed by SIGSEGV +++
Segmentation fault

AUR package not online?

Hi,

I can't find the AUR package with the search.
Is it not online, if so is that intentional?

Regards