oliver006/elasticsearch-gmail

Index your Gmail Inbox with Elasticsearch

elasticsearch-gmail's Issues

create_index() adds a mapping but doesn't create an index

Hi,

as said in title, your function 'create_index' seems not create the 'gmail' index. It misses lines like this (need some improvements) :

index_name = "gmail"
es_url = "http://localhost:9200/%s" % (index_name)
request = HTTPRequest(es_url, method="POST", request_timeout=10)
response = yield http_client.fetch(request)
if not response.code in [200, 201]:
print "\nfailed to create index%s" % item['message-id']

info on how to run from cli

I tried moving through this and it appears to be failing on the import... I am running a vagrant-vm and get everything installed just fine.
I don't know how to invoke the script properly...
I've tried so many ways. This seems like it would give results... though it does nothing much.
python index_emails.py test.mbox
Any help or tips are appreciated! This has been a fun project so far. Stumbling at the end. Thanks!

can we index email body field?

Hey, thanks for a great tool! I got it to successfully import my large mbox file into ES with no problems. All of your queries on your readme worked fine.

It seem, though, that you aren't grabbing the email body. Is that right?
https://github.com/oliver006/elasticsearch-gmail/blob/master/src/index_emails.py#L89
Could you please add it to the code so that we are indexing email body's into ES?

It look like from here: http://www.qmail.org/man/man5/mbox.html that the body is between the _From line and a blank line.

Between the From_ line and the blank line is a message in
RFC 822 format, as described in qmail-header(5), subject to
>From quoting as described below.

Thanks again!

Filter queries for time?

This would probably be a good example.

AttributeError: module 'mailbox' has no attribute 'UnixMailbox'

When running the code out of the box (all dependencies are imported and work alright) I get:

raceback (most recent call last):
  File "index_emails.py", line 220, in <module>
    IOLoop.instance().run_sync(load_from_file)
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/ioloop.py", line 458, in run_sync
    return future_cell[0].result()
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/ioloop.py", line 436, in run
    result = func()
  File "index_emails.py", line 169, in load_from_file
    mbox = mailbox.UnixMailbox(open(tornado.options.options.infile, 'rb'), email.message_from_file)
AttributeError: module 'mailbox' has no attribute 'UnixMailbox'
Exception ignored in: <bound method HTTPClient.__del__ of <tornado.httpclient.HTTPClient object at 0x10dfaf518>>
Traceback (most recent call last):
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/httpclient.py", line 82, in __del__
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/httpclient.py", line 87, in close
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/simple_httpclient.py", line 118, in close
  File "/anaconda3/envs/elasticsearch_gmail/lib/python3.6/site-packages/tornado/httpclient.py", line 203, in close
RuntimeError: inconsistent AsyncHTTPClient cache

Python 3.5.6

subject encoding

There is a problem when the subject of an email contains a special character such as "°".
In this case, the subject is

'subject': <email.header.Header object at 0x7f799db1d6d0>

Traceback :

Traceback (most recent call last):
File "src/index_emails.py", line 231, in
IOLoop.instance().run_sync(load_from_file)
File "/usr/lib/python3/dist-packages/tornado/ioloop.py", line 576, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/tornado/ioloop.py", line 547, in run
result = func()
File "src/index_emails.py", line 196, in load_from_file
upload_batch(upload_data)
File "src/index_emails.py", line 93, in upload_batch
upload_data_txt += json.dumps(item) + "\n"
File "/usr/lib/python3.8/json/init.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type Header is not JSON serializable

Possible resolution in def convert_msg_to_json:

    if "subject" in result:
        result['subject'] = str(result['subject'])

tornado.httpcient.HTTPError: 406 Not acceptable

Hi. I have a problem. I get this error:

python index_emails.py --infile=email.mbox [I 180429 15:01:57 index_emails:170] Starting import from file email.mbox Traceback (most recent call last): File "index_emails.py", line 222, in <module> IOLoop.instance().run_sync(load_from_file) File "/Library/Python/2.7/site-packages/tornado/ioloop.py", line 458, in run_sync return future_cell[0].result() File "/Library/Python/2.7/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "/Library/Python/2.7/site-packages/tornado/ioloop.py", line 436, in run result = func() File "index_emails.py", line 188, in load_from_file upload_batch(upload_data) File "index_emails.py", line 92, in upload_batch response = http_client.fetch(request) File "/Library/Python/2.7/site-packages/tornado/httpclient.py", line 102, in fetch self._async_client.fetch, request, **kwargs)) File "/Library/Python/2.7/site-packages/tornado/ioloop.py", line 458, in run_sync return future_cell[0].result() File "/Library/Python/2.7/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "<string>", line 3, in raise_exc_info tornado.httpclient.HTTPError: HTTP 406: Not Acceptable

What . can i do?

[UPDATE]
I tried to see what i wrong... and.. when it want to create index it gets:
[I 180429 15:08:35 index_emails:81] HTTP 400: Bad Request

How to run queries

I am not able to run the queries shown in the readme.

$ curl -H 'Content-Type: application/json'  'http://localhost:9200/gmail/email/_search?

pretty&search_type=count' -d '{"aggs": { "emails":{"terms" : { "field" : "to",  "size": 10 }} } }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "No search type for [count]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "No search type for [count]"
  },
  "status" : 400
}

$ curl -H 'Content-Type: application/json' 'http://localhost:9200/gmail/email/_search?pretty' -d

```json
'{"filter": { "range" : { "date_ts" : {"gte": "now-6M" } } } }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Unknown key for a START_OBJECT in [filter].",
        "line" : 1,
        "col" : 12
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Unknown key for a START_OBJECT in [filter].",
    "line" : 1,
    "col" : 12
  },
  "status" : 400
}

I can see some output for:

curl -H 'Content-Type: application/json' 'http://localhost:9200/_search' -d'

{
    "query" : {
        "match_all" : {}
    }
}'

Appreciate any help.

How to run the example? (I get ImportError when running the script)

Can you please add a step-by-step explanation how to run the example? (I have no experience with Python coding, just want to run the example).

Python is installed. I also executed your PIP commands (was already included in Anaconda).
I have downloaded your source code (one Python file) in the same folder as the emails.mbox file from Google. I tried:

python index_emails.py
Traceback (most recent call last):
File "index_emails.py", line 12, in
from DelegatingEmailParser import DelegatingEmailParser
ImportError: No module named DelegatingEmailParser

Link to download Gmail data not working

Hello,

I would like to go through this interesting tutorial. However, the link to download the Gmail data does not work.

Can I also download it via the official Google download feature (https://support.google.com/accounts/answer/3024190) or is this a different data set and structure?

Kai