mbovo/imapindexer

A cli tool to index IMAP mailboxes into Elasticsearch / Zincsearch

Infrastructure Engineer / SCM / SRE

• Building and Kicking infrastructures since 2007
• In love with automation and self-healing systems

imapindexer's Issues

Incremental sync

This to track the implementation of incremental sync
We should be able to save current status and avoid re-download every messages in a mailbox if already scanned (and if not forced by user)

This would require a way to save current status (maybe into elasticsearch itself)

gmail error: "parse \"0.0.0.0:4080/api/_bulkv2\": first path segment in URL cannot contain colon"

Hey

thanks for this... I am playing around to index my gmail, but it seems to not like something..
BTW is there a arg to dump the imap data on the way through before its sent to zinc, so i can check...

I get:

ERR failed to index document error="parse \"0.0.0.0:4080/api/_bulkv2\": first path segment in URL cannot contain colon"

ima --loglevel debug --imap.address imap.gmail.com:993 --imap.mailbox INBOX --imap.username [email protected] --imap.password XXX --indexer.batch 5 --indexer.buffer 1000 --zinc.address 0.0.0.0:4080 --zinc.username [email protected] --zinc.password xxx
3:23PM INF Starting imapindexer
3:23PM DBG Debug mode enabled
3:23PM INF Indexer settings batch=5 buffer=1000 workers=1
3:23PM DBG Waiting for messages to index
3:23PM INF Mailboxes found mailboxes=1
3:23PM DBG Starting worker i=0 mailbox=INBOX mboxes=1
3:23PM INF Parsing Mailbox mailbox=INBOX
3:23PM DBG Got Message mailbox=INBOX subject="Fwd: Hey Gerard Here. please add me :)" uid=13164
3:23PM DBG Got Message mailbox=INBOX subject="RE: digging" uid=14280
3:23PM DBG Enqueue Subject="Fwd: Hey Gerard Here. please add me :)" batch=1 mailbox=INBOX uid=13164
3:23PM DBG Enqueue Subject="RE: digging" batch=2 mailbox=INBOX uid=14280
3:23PM DBG Got Message mailbox=INBOX subject="RE: digging" uid=14305
3:23PM DBG Got Message mailbox=INBOX subject="Delivery Status Notification (Failure)" uid=14325
3:23PM DBG Enqueue Subject="RE: digging" batch=3 mailbox=INBOX uid=14305
3:23PM DBG Enqueue Subject="Delivery Status Notification (Failure)" batch=4 mailbox=INBOX uid=14325
3:23PM DBG Got Message mailbox=INBOX subject="Delivery Status Notification (Failure)" uid=14326
3:23PM DBG Got Message mailbox=INBOX subject="Delivery Status Notification (Failure)" uid=14328
3:23PM ERR failed to index document error="parse \"0.0.0.0:4080/api/_bulkv2\": first path segment in URL cannot contain colon"
3:23PM ERR failed to index document error="parse \"0.0.0.0:4080/api/_bulkv2\": first path segment in URL cannot contain colon"
3:23PM DBG Got Message mailbox=INBOX subject=Investment uid=14637
3:23PM ERR failed to index document error="parse \"0.0.0.0:4080/api/_bulkv2\": first path segment in URL cannot contain colon"
3:23PM DBG Got Message mailbox=INBOX subject=Investment uid=14638
3:23PM DBG Got Message mailbox=INBOX subject="RE: Investment" uid=14646

my zinc is setup working for bulk api:

curl http://localhost:4080/api/_bulk -i -u XXX  --data-binary "@olympics.ndjson"
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Zinc: 0.4.7
Date: Sun, 20 Aug 2023 12:59:24 GMT
Content-Length: 53

{"message":"bulk data inserted","record_count":36935}%