gauteh/lieer

Fast email-fetching and sending and two-way tag synchronization between notmuch and GMail

lieer's Issues

After first pull, all emails have the inbox and unread labels

Other labels are fine though. Did I do something wrong? Only 2 emails do not have the inbox and unread labels.

README.md -- Installation instructions need help

I found the README.md insufficient to help me do a normal install of gmailieer.

Here are the problems I encountered, along with some suggestions for solutions. I don't have all the solutions, though, because I simply got blocked, and don't know the answer.

1. In the requirements section of the README.me, it is unclear what these names are. Are they pip packages? .deb's or .rpm's? Also, setup.py will not run without setuptools, which is not mentioned. So:
   a. add setuptools to the requirements list as the 6th item:

• setuptools

b. Give explicit examples as to how these requirements may actually be met. For instance, add the following text:

In Debian 9 ("Stretch"), for instance, these prerequisites can be obtained this way:

$ sudo apt-get -y install python3-tqdm python3-googleapi python3-oauth2client python3-notmuch python3-setuptools

2. "cart before horse"? -- notmuch seems to be required before running.
   In the installation section, item 2 seems to assume that one already has a ~/.mail/[account].gmail/.notmuch directory set up. (I only this having worked with notmuch already. But I would have been greatly confused as a newbie, if I had not. I would have had NO idea what this even means.) But the instructions to create a .notmuch subdir do not exist here, and are only implied. Also, the instructions do not say that what they are really referring to is ~/.mail/[account].gmail/.notmuch/hooks/post-new . They also apparently refer to ~/.notmuch-config , and assume the ~/.notmuch-config is already set up and pointing to ~/.mail/[account].gmail -- But again, this is not stated, and if I had not already worked with notmuch, I would not have known this. I would have been completely lost.

3. Even setting up a ~/.notmuch-config and pointing it to ~/.mail/[account].gmail , and setting up a ~/.mail/[account].gmail/.notmuch/hooks/post-new script: When I cd to ~/.mail/[account].gmail, and run,

gmi init [account]@gmail

as per the instructions, I get:

A Xapian exception occurred opening database: Couldn't stat '/home/[me]/.mail/[account].gmail/.notmuch/xapian'

So, what? gmailieer expects notmuch to already have been run? How is that possible, when the maildir is not yet downloaded?

(I have no solution for this. I am stuck.)

4. If I try to run

gmi init [account]@gmail

a second time, now I get a different error message:

lieer.local.RepositoryException: '.gmailieer.json' exists: this repository seems to already be set up!

(and if I remove this, I get a similar message for the 'mail' subdir previously created.)

5. Also, the README.md needs to make it clear whether it is able to work with a previously-existing maildir or not. And whether offlineimap or mbsync may be assumed. In another issue, it appears to be no for both of these. If that is the case, this should be made explicit in the README.md. (Although that is too bad, because it means having do re-download all my emails.)

But if a previously-existing maildir may not be used, then how is it that it is assumed that notmuch is already installed and its database initialized?

---

In the end, the README.md installation instructions simply do not work for me. I am dead in the water. I have spent all morning on it and tried to add some possible solutions, but I think really the best solution is that you run an actual test installation yourself from a virgin system and document it, step by step, in the README.md, so that the instructions actually work. I would do this for you (with a fork) but I myself cannot figure out what you wanted.

Hope this helps.

gmi init breaks with error: "ImportError: No module named 'oauth2client'"

I have next to zero knowledge of these kind of "python problems". I tried to do eveything by the book, however: selected py36 trough macports' sudo port select --set python python36, same with pip, installed oauth2 with pip --upgrade ...

Everything seemed to have gone fine. Then, when I tried gmi init my.account@gmail :

Traceback (most recent call last):
  File "/Users/m2habert/gitRepo/gmailieer/gmi", line 4, in <module>
    from lieer import Gmailieer
  File "/Users/m2habert/gitRepo/gmailieer/lieer/__init__.py", line 1, in <module>
    from .gmailieer import *
  File "/Users/m2habert/gitRepo/gmailieer/lieer/gmailieer.py", line 8, in <module>
    from    oauth2client import tools
ImportError: No module named 'oauth2client'

I've tried googling the error message, but couldn't make heads or tails of anything (it seems all related complaints are posted by developers programming their own APIs, nothing like a software user with installation problems)

¿Can anybody point me to the right direction? Thanks
Using MacOS 10.11.6

Error while syncing cleaned up inbox

I removed some 4000 emails on Gmail side (i.e. emptying trash) and when I do a sync I occasionally get the following:

Traceback (most recent call last):
  File "/nix/store/q1khwyfbip46yyddm0li2fsn0c3y9mx4-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 190, in sync
    self.pull (args, True)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 315, in pull
    self.partial_pull ()
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 457, in partial_pull
    self.local.remove (m['id'], db)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 298, in remove
    nmsg  = db.find_message_by_filename (fname)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/notmuch/database.py", line 560, in find_message_by_filename
    raise NotmuchError(status)
notmuch.errors.XapianError

Initial pull fails with KeyError

Output:

pull: full synchronization (no previous synchronization state)
fetching messages: 353225it [18:57, 310.48it/s]                                 
receiving content:  73%|████████▋   | 257019/353225 [6:17:36<2:56:26,  9.09it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 8, in <module>
    g.main ()
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 527, in full_pull
    updated = self.get_content (message_gids)
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 592, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/usr/lib/python3.6/site-packages/lieer/remote.py", line 106, in func_wrap
    return func (self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/lieer/remote.py", line 277, in get_messages
    batch.execute (http = self.http)
  File "/usr/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/googleapiclient/http.py", line 1464, in execute
    self._callback(request_id, response, exception)
  File "/usr/lib/python3.6/site-packages/lieer/remote.py", line 257, in _cb
    cb (resp)
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 590, in _got_msg
    self.local.store (m, db)
  File "/usr/lib/python3.6/site-packages/lieer/local.py", line 343, in store
    self.update_tags (m, p, db)
  File "/usr/lib/python3.6/site-packages/lieer/local.py", line 351, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/usr/lib/python3.6/site-packages/lieer/local.py", line 351, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_9'

Can't initialize

I get this error when trying to run init; creating the file results in gmi saying that '.gmailieer.json' exists: this repository seems to already be set up.

username@computername:~/.mail> gmi init [email protected]
initializing repository in: /home/username/.mail..
Traceback (most recent call last):
File "/usr/bin/gmi", line 4, in
import('pkg_resources').run_script('gmailieer==0.3', 'gmi')
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 1524, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 136, in main
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 140, in initialize
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/local.py", line 199, in initialize_repository
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/local.py", line 81, in write
File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/username/.mail/.gmailieer.json'

Fetching all the metadata every time?

Each time I run "gmi sync" it takes several minutes to fetch metadata for some 70,000 items. Is this intended behavior or is it supposed to be faster once the initial sync has been completed?

Sync drafts

According to the readme, gmailieer cannot add email. This is somewhat problematic for drafts. I could set up mbsync just for drafts, but I think that would create duplication since once a draft is pushed, gmi sync would add it to its maildir as well.

I'm not familiar with gmail API, but can drafts be somehow supported? If not, is it possible to black list some tags?

Is this a replacement of abunchoftags?

?

lieer.remote.GenericException: no 'history' record returned, even though several pages were indicated.

Note sure what happened. I was removing the inbox tag on about 30 messages or so, and then syncing.

pull: partial synchronization.. (hid: 12008538)
fetching changes: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/home/jak/Downloads/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 190, in sync
    self.pull (args, True)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 315, in pull
    self.partial_pull ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 324, in partial_pull
    for hist in self.remote.get_history_since (self.local.state.last_historyId):
  File "/home/jak/Downloads/gmailieer/lieer/remote.py", line 166, in get_history_since
    raise Remote.GenericException ("no 'history' record returned, even though several pages were indicated.")
lieer.remote.GenericException: no 'history' record returned, even though several pages were indicated.
fetching changes: 100it [00:00, 607.63it/s]

http.client.RemoteDisconnected not handled

I got this while running gmi pull

log.txt

AttributeError while pulling mail

$ gmi sync                                                              ~/Mail/personal.gmail
receiving metadata: 0it [00:00, ?it/s]
resolving changes: 0it [00:00, ?it/s]
push: nothing to push
remote historyId: 12267967
pull: full synchronization (no previous synchronization state)
fetching messages: 139732it [11:53, 134.01it/s]                                               
receiving content:   0%|                               | 1/139732 [00:03<155:04:25,  4.00s/it]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.2', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 190, in sync
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 527, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 595, in get_content
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 100, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 271, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 251, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 593, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 294, in store
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 335, in update_tags
AttributeError: 'Database' object has no attribute 'index_file'

Unhandled exception: cannot reduce request any further

During the initial clone I got this error:

log.txt

synchronization state not saved while first pulling

I have a lot of email, and syncing for the first time is a problem, due to Gmail throttling.

I try to cancel the first run, to get reset at the Gmail end and avoid the throttling, but I'm back in the first situation again, and I will end up with duplicate files on disk:

(venv) 20:13 CHARLIE~/Maildir/[email protected][[email protected]] % PYTHONPATH=~/src/notmuch/bindings/python/  ~/src/gmailieer/gmi pull    (chp02-rsvp|✚23…)
pull: full synchronization (no previous synchronization state)
fetching messages: 557143it [19:02, 487.47it/s]
receiving content :   0%|                                                                                             | 129/557143 [00:17<13:22:16, 11.57it/s]
reducing batch request size to: 100
reducing batch request size to: 50
receiving content :  30%|██████████████████████████▎                                                              | 164746/557143 [4:08:49<9:26:39, 11.54it/s]
reducing batch request size to: 25

^Creceiving content :  34%|██████████████████████████████▎                                                          | 190047/557143 [5:23:08<9:19:35, 10.93it/
s]Traceback (most recent call last):
  File "/Users/charlie/src/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 272, in pull
    self.full_pull ()
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 488, in full_pull
    updated = self.get_content (message_ids)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 556, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 82, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 204, in get_messages
    batch.execute (http = self.http)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/googleapiclient/http.py", line 1417, in execute
    self._execute(http, self._order, self._requests)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/googleapiclient/http.py", line 1349, in _execute
    headers=headers)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/httplib2/__init__.py", line 1322, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/httplib2/__init__.py", line 1072, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/httplib2/__init__.py", line 1054, in _conn_request
    content = response.read()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 455, in read
    return self._readall_chunked()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 561, in _readall_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 607, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
KeyboardInterrupt

Then I try again, without the throttle:

(venv) 9:26 CHARLIE~/Maildir/[email protected][[email protected]] % PYTHONPATH=~/src/notmuch/bindings/python/  ~/src/gmailieer/gmi pull     (chp02-rsvp|✚23…)
pull: full synchronization (no previous synchronization state)
fetching messages:  24%|███████████████████████▊                                                                          | 800/3296 [00:02<00:05, 430.98it/s]

Can we sync state during the initial pull, and then resume the pull?

Database object has no attribute get_revision

pull: full synchronization (no previous synchronization state)
fetching messages: 7479it [00:16, 322.74it/s]                                                                                                                                                  
receiving content: everything up-to-date.
receiving metadata:  59%|████████████████████████████████████████████████████████████████████████████▍                                                     | 4401/7479 [00:47<00:34, 88.41it/s]
reducing batch request size to: 100
receiving metadata: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7479/7479 [01:22<00:00, 90.88it/s]
Traceback (most recent call last):
  File "/home/aureooms/.bin/gmi", line 8, in <module>
    g.main ()
  File "/home/aureooms/.opt/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/aureooms/.opt/gmailieer/lieer/gmailieer.py", line 272, in pull
    self.full_pull ()
  File "/home/aureooms/.opt/gmailieer/lieer/gmailieer.py", line 499, in full_pull
    (rev, uuid) = db.get_revision ()
AttributeError: 'Database' object has no attribute 'get_revision'

Assertion Error status=count(messages)- can't really pinpoint the source

I can pull message, but not push.
Using the current HEAD, notmuch in git package version in Arch

During push an Assertion Error kills the script

Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.1', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1507, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/lieer/gmailieer.py", line 186, in sync
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/lieer/gmailieer.py", line 213, in push
  File "/usr/lib/python3.6/site-packages/notmuch/query.py", line 204, in count_messages
    raise NotmuchError(status)
  File "/usr/lib/python3.6/site-packages/notmuch/errors.py", line 129, in __new__
    subclass = cls.get_exc_subclass(status)  # which class to use?
  File "/usr/lib/python3.6/site-packages/notmuch/errors.py", line 115, in get_exc_subclass
    assert 0 < status <= len(subclasses)
AssertionError

status seems like the count of my messages: ~22k, and len(subclasses): 14,

The Function in count_messages in "notmuch/query.py" tries to count messages and should give a status back, But it seems, the status is the message count.
It seems the python script doesn't match with the notmuch library.

def count_messages(self):
        '''
        This function performs a search and returns Xapian's best
        guess as to the number of matching messages.

        :returns: the estimated number of messages matching this query
        :rtype:   int
        '''
        self._assert_query_is_initialized()
        count = c_uint(0)
        status = Query._count_messages(self._query, byref(count))
        if status != 0:
            raise NotmuchError(status)
        return count.value

If see the problem in the packaging of notmuch as well, you can close this issue.

Implement gmi send

Implement gmi send which sends messages using Users.messages.send, adds the messages instantly to the local repository and applies specified tags. Should be a drop-in for sendmail and support common functionality (e.g. the subset implemented by msmtp).

If drafts are supported this should use Drafts.send, but as far as I can see this will add a significant layer of complexity and we could skip that for now.

.gmailieer.json destroyed when running out of space

When the disk is full, gmailier overwrites its file with an empty one and then throws an exception that the disk is full. It should probably write to a temporary file and rename it instead.

MAX_CONNECTION_ERRORS is not defined

fetching messages: 436942it [20:23, 228.73it/s]
receiving content:  28%|███████████████████████████████████████████████████████▏                                                                                                                                           | 123800/436942 [6:43:02<18:42:11,  4.65it/s]connection failed, re-trying: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/hvhy4l4zl4m8rh2q5pdd9b8pa2r5bz2v-python3.6-google-api-python-client-1.5.1/lib/python3.6/site-packages/googleapiclient/http.py", line 1409, in execute
    self._refresh_and_apply_credentials(request, http)
  File "/nix/store/hvhy4l4zl4m8rh2q5pdd9b8pa2r5bz2v-python3.6-google-api-python-client-1.5.1/lib/python3.6/site-packages/googleapiclient/http.py", line 1123, in _refresh_and_apply_credentials
    creds.refresh(http)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 597, in refresh
    self._refresh(http.request)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 780, in _refresh
    self._do_refresh_request(http_request)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 799, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 562, in new_request
    redirections, connection_type)
  File "/nix/store/xcyj7i0gghrh4r2v4v91aa2ia370z0wc-python3.6-httplib2-0.9.2/lib/python3.6/site-packages/httplib2/__init__.py", line 1314, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/nix/store/xcyj7i0gghrh4r2v4v91aa2ia370z0wc-python3.6-httplib2-0.9.2/lib/python3.6/site-packages/httplib2/__init__.py", line 1064, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/nix/store/xcyj7i0gghrh4r2v4v91aa2ia370z0wc-python3.6-httplib2-0.9.2/lib/python3.6/site-packages/httplib2/__init__.py", line 988, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 986, in send
    self.sock.sendall(data)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/ssl.py", line 972, in sendall
    v = self.send(byte_view[count:])
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/ssl.py", line 941, in send
    return self._sslobj.write(data)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/ssl.py", line 642, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 314, in pull
    self.full_pull ()
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 537, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 602, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/remote.py", line 314, in get_messages
    if conn_errors > MAX_CONNECTION_ERRORS:
NameError: name 'MAX_CONNECTION_ERRORS' is not defined

On NixOS with version 0.5.

README.md instructions for setup.py need to be fleshed out

In the README.md, it is suggested to symlink gmi, "or use setup.py".

When I tried running python setup.py and just got this:

$ python3 setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
$ _

so I ran:

$ python3 setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ extensions (compile/link to build directory)
  build_clib        build C/C++ libraries used by Python extensions
  build_scripts     "build" scripts (copy and fixup #! line)
  clean             clean up temporary files from 'build' command
  install           install everything from build directory
  install_lib       install all Python modules (extensions and pure Python)
  install_headers   install C/C++ header files
  install_scripts   install scripts (Python or otherwise)
  install_data      install data files
  sdist             create a source distribution (tarball, zip file, etc.)
  register          register the distribution with the Python package index
  bdist             create a built (binary) distribution
  bdist_dumb        create a "dumb" built distribution
  bdist_rpm         create an RPM distribution
  bdist_wininst     create an executable installer for MS Windows
  check             perform some checks on the package
  upload            upload binary package to PyPI

Extra commands:
  easy_install      Find/get/install Python packages
  bdist_egg         create an "egg" distribution
  alias             define a shortcut to invoke one or more commands
  install_egg_info  Install an .egg-info directory for the package
  rotate            delete older distributions, keeping N newest files
  develop           install package in 'development mode'
  upload_docs       Upload documentation to PyPI
  saveopts          save supplied options to setup.cfg or other config file
  test              run unit tests after in-place build
  setopt            set an option in setup.cfg or another config file
  egg_info          create a distribution's .egg-info directory

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help
$ _

..which I didn't know what to do with.

Ultimately, I just gave up and symlinked gmi, as suggested. Just simpler.

KeyError: '15bac9a5ba0b826b' in update_tags: self.mids[mid]

Traceback (most recent call last):
  File "/home/jak/bin/gmi", line 8, in <module>
    g.main ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 190, in sync
    self.pull (args, True)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 315, in pull
    self.partial_pull ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 463, in partial_pull
    r = self.local.update_tags (m, None, db)
  File "/home/jak/Downloads/gmailieer/lieer/local.py", line 303, in update_tags
    fname = self.mids[mid]
KeyError: '15bac9a5ba0b826b'```

Problems using notmuch database on a path containing symbolic links

For some reasons particular to my setup, my notmuch database is configured using a path containing a symbolic link. Notmuch does not complain about that and works like a charm.

However, gmailieer does not like symbolic links and always tries to use the real path to the database.

https://github.com/gauteh/gmailieer/blob/f610557c2964e1e3c628e04276b45d20c8ba7066/lieer/local.py#L145

Even if the paths are logically the same, the real path string is evidently different from the path (that contains the symbolic link) that I configured notmuch with. Thus I always, incorrectly ,get the following message.

local mail repository not in notmuch db

For the time being I created a script that, using sed, alters notmuch configuration and after gmailieer is done goes back to the original configuration. It is not ideal, but works for me.

I was wondering, could the code above be altered to try and match the symbolic link path with the absolute one?

AttributeError exception

I hit this while testing #4

log.txt

ship oauth2 client id in some safe way

So that users don't need an API key. Should be configurable, since there is an API limit on requests which may easily be transcended when doing initial pulls.

Replied and Passed maildir flags are not pulled

KeyError: 'history'

During a sync I got this exception:

log.txt

Initial `push` (after `pull`) hits API limits

Hello,

I successfully moved two of three accounts to gmailieer. The problem is with the third and biggest account (roughly 200.000 messages):

I can successfully init and pull the messages, but a following push or sync will get throttled more and more and finally stop as gmailieer can't reduce the batch size any further.

Is there something I can do about this?

Updating tags on non-existent file

After performing a sync of my pretty large mailbox, when updating tags, I consistently see an exception about failing to update tags on a non-existent file. I think gmailieer may have somehow got the file name wrong, but I'm not sure... Here's the trace:
...
resolving changes: 100%
receiving content: 100%
updating tags (0Δ): 0%| | 0/13602 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/local/bin/gmi", line 8, in
g.main ()
File "/b/git/gmailieer/lieer/gmailieer.py", line 145, in main
args.func (args)
File "/b/git/gmailieer/lieer/gmailieer.py", line 215, in sync
self.pull (args, True)
File "/b/git/gmailieer/lieer/gmailieer.py", line 341, in pull
self.partial_pull ()
File "/b/git/gmailieer/lieer/gmailieer.py", line 492, in partial_pull
r = self.local.update_tags (m, None, db)
File "/b/git/gmailieer/lieer/local.py", line 401, in update_tags
raise Local.RepositoryException ("tried to update tags on non-existant file: %s" % fname)
lieer.local.RepositoryException: tried to update tags on non-existant file: /path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2,

The file "/path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2," does not exist in the mail store, but file "/path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2,S" does.

Could it be possible that gmailieer has had a problem parsing this file name?

notmuch search for "/path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2,S" correctly identifies the email and shows the tags applied to it. The message only has one mime part which is text/html (no text/plain part).

Also, is there a particular reason why this exception hangs gmailieer? I think it would be better if it exited.
-- Matt

<HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/julian.klode%40gmail.com/messages/15c69db4486d0718/modify?alt=json returned "Can only add on e of: INBOX, SPAM, TRASH">

If I locally tag a mail as both +inbox and +trash (which makes no sense, really), the sync fails. gmailieer should instead just drop the inbox tag and add the trash tag I guess.

"Backend Error" not handled

I noticed I had a "new" tag in every message. A leftover from my old notmuch config. I removed it from all messages, and ran "gmi sync". It started checking every message flags against the remote, and eventually failed with the attached log. Maybe it should retry?

log.txt

customize ignoring tags

I'm using "snoozing" mails quite often. Atm i'm removing the inbox tag and adding two new tags: snooze, snoozeuntil-. If matches the current date the email will get readded to inbox :)

gmi syncs both tags towards gmail and clutters the taglist.

Ignoring tags is a thing already. It would be nice to customize it :)

Missing GMail labels

Similarly to #46 I've:

pull: full synchronization (no previous synchronization state)                                                                                    [3/8741]
fetching messages: 123866it [04:10, 414.05it/s]
receiving content:   0%|                                                                             | 3/57916 [00:00<9:22:58,  1.71it/s]Traceback (most r
ecent call last):
  File "/nix/store/5zb6zlyxg2xjz7p3kqr9hahl3vdbzm63-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 530, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 595, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/googleapiclient/http.py", line 1437, in execute
    self._callback(request_id, response, exception)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 262, in _cb
    cb (resp)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 593, in _got_msg
    self.local.store (m, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 346, in store
    self.update_tags (m, p, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_3'

I did not create a label while it was running and I do not have Label_3 as label.

"sent" label is not synced back

Apparently, the "sent" label (pseudo label) is only synced from GMail to the notmuch database, not vice versa:

If I send an e-mail using GMail's web interface (or IMAP) it gets saved by GMail and appears under "sent messages"; when synced by gmailieer, it gets the notmuch tag "sent". So far so good.

If I tag a message as sent using "notmuch tag +sent" it gets the tag, and gmailieer seems to update something on sync but the message does not appear under "sent messages" in GMail (only "all mail").

I know that "sent" is not quite a label for GMail but something else - but maybe it could be synced both ways nonetheless?

[Background: I sometimes send using none-GMail-SMTP servers and want to mbsync (via IMAP) a copy of the sent message to my GMail account; this works fine, but I can't tag the copy as sent (i.e. sync the sent label) using gmailieer.]

Use wd instead of wd/mail

Currently, gmi puts the maildir store into a subdirectory mail of the current working directory. This is suboptimal when you want to have your gmi-managed maildir and other maildirs (synced via mbsync or offlineimap, say) under the same maildir root:

mailroot/imapaccount/cur
mailrrot/imapaccount/...
mailroot/gmailaccount/mail/cur
mailroot/gmailaccount/mail/...

This ends up as folders imapaccount and gmailaccount at the root level, but all that gmailaccount has is an extra unnecessary subfolder mail that contains all mails (when imapaccount contains its imap folders).

gmi's json files need to be ignored by notmuch anyways. Therefore, I suggest to put the maildir directly under the workdir (omit /mail), maybe based on some config setting. This could help maildir based MUAs (though notmuch based is preferred, I know).

Weird corner cases with neomutt moving unread emails from cur/ to new/

So, (neo)mutt will apparently move unread emails from cur/ to new/ when marking them as trash (for example, any other label should do the trick as well), gmailieer then just says: Not in the repo.

After initial pull, too many messages have inbox tag

This is my post-new hook (the only hook I have):

#!/bin/sh

# immediately archive all messages from "me"
notmuch tag -new -- tag:new and from:email@domain

# finally, remove new
notmuch tag -new -- tag:new

and my .notmuch_config:

[database]
...
[user]
...
[new]
tags=new
ignore=*.json;
[search]
exclude_tags=deleted;spam;
[maildir]
synchronize_flags=true
[crypto]
gpg_path=gpg

After the first full pull (gmi pull), I see about >80k messages tagged with inbox while gmail shows me only 1700? I guess I miss something, but I don't understand the discrepacy between the numbers?

gmi pull keeps crashing

$ gmi pull                                                                                                                                                              /data/Mail/personal
pull: full synchronization (no previous synchronization state)
fetching messages: 139863it [12:33, 153.90it/s]                                                                                                                receiving content: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [02:58<00:00,  3.28s/it]receiving metadata:   3%|██▌                                                                                             | 3683/139806 [01:14<38:10, 59.42it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.2', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 531, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 562, in get_meta
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 100, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 271, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 251, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 560, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 326, in update_tags
lieer.local.RepositoryException: tried to update tags on non-existant file: /data/Mail/personal/mail/cur/12e10631fc4211b6:2,

This is right after the initial pull (which also crashed with similar stack trace).

"cannot reduce request any further"

I set up alot to use the maildir syncronized by gmailieer as send_box. When I send a mail (and store and tag it "sent") and then syncronize I get the following error.

[~/mail/account.patricktotzke.gmail.com] ~/projects/gmailieer/gmi sync                                                    
receiving metadata:   0%|                                                                                               | 0/9 [00:00<?, ?it/s]reducing batch request size to: 25
reducing batch request size to: 12
reducing batch request size to: 6
Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/usr/lib/python3/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 1451, in execute
    self._callback(request_id, response, exception)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 258, in _cb
    raise Remote.BatchException(excep)
lieer.remote.BatchException: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/patricktotzke%40gmail.com/messages/1511291035.M158427P31771Q1.piu?format=minimal&alt=json returned "Invalid id value">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 193, in sync
    self.push (args, True)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 238, in push
    self.remote.get_messages (gids, _got_msg, 'minimal')
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 305, in get_messages
    raise Remote.BatchException ("cannot reduce request any further")
lieer.remote.BatchException: cannot reduce request any further
[1]    431 abort      ~/projects/gmailieer/gmi sync

AttributeError on notmuch database object

Traceback (most recent call last):
  File "/usr/local/bin/gmi", line 8, in <module>
    g.main ()
  File "/home/maxking/Documents/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/maxking/Documents/gmailieer/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/home/maxking/Documents/gmailieer/lieer/gmailieer.py", line 538, in full_pull
    (rev, uuid) = db.get_revision ()
AttributeError: 'Database' object has no attribute 'get_revision'

Not sure what this means but I was trying to create a fresh maildir using gmailieer. I have python3.5.2 on Linux Mint 18.1.

Label name exists or conflicts

Traceback (most recent call last):
  File "/home/maxking/Documents/gmailieer/lieer/remote.py", line 569, in __create_label__
    lr = self.service.users ().labels ().create (userId = self.account, body = label).execute ()
  File "/usr/local/lib/python3.5/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 409 when requesting https://www.googleapis.com/gmail/v1/users/raj.abhilash1%40gmail.com/labels?alt=json returned "Label name exists or conflicts">

If a local label exists, it tries to create a create same label remotely and fails

Sync SPAM and TRASH

At the moment the value of remote.not_sync is fixed to
not_sync = set (['CHAT', 'SPAM', 'TRASH'])
so that SPAM and TRASH are not downloaded.

In order to be able to undelete or remove from spam, I would need to sync these as well. Is it possible to make this a configurable option?

Exclude voicemail label in push

Hi. I hit an error trying to push the voicemail tag / label to gmail. I added it to the currently excluded labels in lieer/local.py and it resolved the problem. Here's a back trace:
push: creating label: voicemail.. Traceback (most recent call last): File "/usr/local/bin/gmi", line 8, in <module> g.main () File "/b/git/gmailieer/lieer/gmailieer.py", line 143, in main args.func (args) File "/b/git/gmailieer/lieer/gmailieer.py", line 193, in sync self.push (args, True) File "/b/git/gmailieer/lieer/gmailieer.py", line 246, in push actions.append (self.remote.update (rm, nm, self.local.state.last_historyId, self.force)) File "/b/git/gmailieer/lieer/remote.py", line 110, in func_wrap return func (self, *args, **kwargs) File "/b/git/gmailieer/lieer/remote.py", line 518, in update return self.__push_tags__ (gid, add, rem) File "/b/git/gmailieer/lieer/remote.py", line 110, in func_wrap return func (self, *args, **kwargs) File "/b/git/gmailieer/lieer/remote.py", line 534, in __push_tags__ (lid, ll) = self.__create_label__ (a) File "/b/git/gmailieer/lieer/remote.py", line 110, in func_wrap return func (self, *args, **kwargs) File "/b/git/gmailieer/lieer/remote.py", line 651, in __create_label__ lr = self.service.users ().labels ().create (userId = self.account, body = label).execute () File "/home/xxx/.local/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper return wrapped(*args, **kwargs) File "/home/xxx/.local/lib/python3.5/site-packages/googleapiclient/http.py", line 841, in execute raise HttpError(resp, content, uri=self.uri) googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/xxx/labels?alt=json returned "Invalid label name">
Can you commit this to master please?
Thanks :-)
-- Matt

Invalid label name ?

I see this during gmi sync

Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 193, in sync
    self.push (args, True)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 245, in push
    actions.append (self.remote.update (rm, nm, self.local.state.last_historyId, self.force))
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 503, in update
    return self.__push_tags__ (gid, add, rem)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 519, in __push_tags__
    (lid, ll) = self.__create_label__ (a)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 630, in __create_label__
    lr = self.service.users ().labels ().create (userId = self.account, body = label).execute ()
  File "/usr/lib/python3/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 838, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/octo%40gmail.com/labels?alt=json returned "Invalid label name">

segmentation fault

I got the following backtrace during a sync

log.txt

Multiple gmail accounts?

Is there a good way of setting up multiple accounts in a single repository?

muted tag is invalid to push

pushing, 0 changed:   0%|                                                                                                                                                               | 0/162 [00:00<?, ?it/s]
push: creating label: muted..
Traceback (most recent call last):
  File "/Users/charlie/src/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 186, in sync
    self.push (args, True)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 224, in push
    r = self.remote.update (m, self.local.state.last_historyId, self.force)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 83, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 376, in update
    self.__push_tags__ (mid, add, rem)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 83, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 396, in __push_tags__
    (lid, ll) = self.__create_label__ (a)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 83, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 433, in __create_label__
    lr = self.service.users ().labels ().create (userId = self.account, body = label).execute ()
  File "/Users/charlie/src/gmailieer/venv/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.6/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/charlie%40eatyourpets.com/labels?alt=json returned "Invalid label name">
pushing, 0 changed:   1%|▉                                                                                                                                                      | 1/162 [00:00<00:30,  5.24it/s]

Apparently, muted isn't available in the API!

I can add it to ignore_labels in local.py but we should note in the README that syncing muting isn't supported.

Speed up initial download

My personal email goes back over 10 years, and I'd rather not have all of that downloaded on my laptop (even setting the max messages per label in Gmail doesn't help with that, because I have gained way too many labels over the years). offlineimap allows you to sync mails from certain 'folders'. Would be good if gmailieer allows that.

Ability to control logging output

Hi. I've used offlineimap for my personal email for a few years now and have recently switched to gmailieer for my gmail account. It's great! Thank you for all the hard work. I also run offlineimap on a system I created at work to harvest email and I would like to switch this to gmailieer too. The periodic mail sync on that machine is driven by cron. Offlineimap is configured on that machine to log to syslog. Since cron treats anything written to stdout as something it needs to notify me about, using gmailieer would incur me receiving email from this machine every mail sync cycle unless I do some redirect tweaking/hacking :-)
Offlineimap offers the ability to use a few different schemes for logging, eg syslog, tty and offers a quiet option too. I really like the gmailieer feedback using tqdm but would it be possible to introduce some sort of logging redirection, maybe just sending stdout to syslog? In that case, maybe the progress bar feedback wouldn't be as important.
Cheers,
-- Matt

Unexpected behavior: inbox emptied

Unexpected behavior: inbox emptied

I've just switched to gmailieer, and followed the instructions on the webpage,
and did an initial download, and there are several things that seem odd to me,
(I've tried finding a IRC-channel or FAQ to ask these questions).

1. After running gmi sync from the folder it has emptied my inbox. Prior
   to the sync, I had +2000 conversation threads in the web-interface of gmail,
   after the sync I have 7. The rest seems to reside in .mail/gmail/mail/cur
   (about 20k of files).

2. Most importantly for me: Is it possible to push the messages back up to
   web-gmail some how? And why/how can this happen?

3. When sending a mail, and using mbsync and notmuch, I'd have a fcc of the
   mail to my "sent" folder. Should I do something similar for gmailieer, or it
   should not be fcc-ed anywhere?

4. Looking at the email in notmuch (emacs), not all messages that I've
   trashed, using web-interface of gmail, are marked with the trash tag,
   only some are.

Note, I ran gmi sync several times, to properly propagate tag changes I had
done to the downloaded mail in notmuch.el, and on one occasion, gmailieer
produced thousands of rows like the following:

update: gid does not match any file name of message, probably a draft, skipping: 12e247f736244708
update: gid does not match any file name of message, probably a draft, skipping: 12e2479a1d3a6ec0
update: gid does not match any file name of message, probably a draft, skipping: 12e2475f9eba8394

EDIT: Using my browser to check gmail, I see that the mail that was
trashed before is back, so I assume, the trash tag issue is because they've
ended back in my inbox for some reason.

use a batch method when pushing

just like pulling. This is useful if you decide to run notmuch tag --batch and it touches 80k files.