schluggi/pymap-copy

Copy and transfer IMAP mailboxes

pymap-copy

In our company we often have to copy mailboxes from one to another server. For this we used IMAPCopy as so far. Due to compatibility issues, first of all the missing SSL/TLS and STARTTLS support I wrote my own python-based version. I hope you like it!

Features

• Copy folders and subfolders
• Copy mails even with flags (seen, answered, ...)
• Connect via SSL/TLS (by default), STARTTLS or without encryption
• Support incremental copy (copies only new mails/folders)
• User specific redirections (with wildcard support)
• Auto subscribe new folders (by default)
• Auto find the special IMAP folders Drafts, Trash, etc. (by default)
• Quota checking (by default)
• Over all progress bar
• Buffer utilization for maximum performance
• Optimized for large mailboxes
• Workaround for Microsoft Exchange Server's IMAP bug
• Statistics
• Simple usage

📦 Requirements

• Python >= 3.6

🧑‍🔧 Installation

pip (recommended)

1. Install pymap-copy: python3 -m pip install pymap-copy
2. Start the program:pymap-copy.py --help

Upgrade: python3 -m pip install --upgrade pymap-copy

whl / release

1. Download the latest release
2. Install the wheel-file: python3 -m pip install pymap_copy-X.X-py3-none-any.whl
3. Start the program:pymap-copy.py --help

Upgrade: Simply install a newer release

git

1. Clone this repo
2. Install the requirements by running python3 -m pip install -r requirements.txt
3. Start the program:./pymap-copy.py --help

Upgrade: git pull

Usage

By running the following command the whole structure (folders & mails) from user1 will be copied to the mailbox of user2.

pymap-copy.py \
--source-user=user1 \
--source-server=server1.example.org \
--source-pass=2345678 \
--destination-user=user2 \
--destination-server=server2.example.info \
--destination-pass=abcdef

If you just want to look what would happen append -d/--dry-run.

Incorrect login

If your password contains special characters (like !, $, #, ...), you have to quote them with a backslash (\) in front. This is a common mistake (#8).

Redirections and destination root

Redirections

You want to merge INBOX.Send Items with the INBOX.Send folder? You can do this with -r/--redirect. The syntax of this argument is simple source:destination. For this example you can use -r "INBOX.Send Items:INBOX.Send" to put all mails from the source folder INBOX.Send Items to destination folder INBOX.Send. Please make sure you use quotation marks if one of the folders includes special characters. In addition, the folder names must be case-sensitive with the correct seperator. Do a dry run first (-d/--dry-run), to check that everything will redirect correctly.

Destination root

In some cases it is necessary to copy all mails from source into an import folder on destination. In this case you can use --destination-root to define the import folder: --destination-root INBOX.Import.

Special case: The source has another root than the destination.

Current folder: INBOX (144 mails, 49.0 MB) -> INBOX (non existing)
Current folder: INBOX.Folder1 (4 mails, 7.2 MB) -> INBOX.Folder1 (non existing)
Current folder: Trash.Folder1 (22 mails, 1.1 MB) -> Trash.Folder1 (non existing)

This often does not work. Most mail providers do not allow folders parallel to INBOX.

If you want to merge all folders into INBOX you can use --destination-root INBOX --destination-root-merge. The result should be as shown:

Current folder: INBOX (144 mails, 49.0 MB) -> INBOX (non existing)
Current folder: INBOX.Folder1 (4 mails, 7.2 MB) -> INBOX.Folder1 (non existing)
Current folder: Trash.Folder1 (22 mails, 1.1 MB) -> INBOX.Trash.Folder1 (non existing)

Without --destination-root-merge INBOX would be prepended to all folders:

Current folder: INBOX (144 mails, 49.0 MB) -> INBOX.INBOX (non existing)
Current folder: INBOX.Folder1 (4 mails, 7.2 MB) -> INBOX.INBOX.Folder1 (non existing)
Current folder: Trash.Folder1 (22 mails, 1.1 MB) -> INBOX.Trash.Folder1 (non existing)

ℹ️ As always: Do a dry run (-d/--dry-run) to ensure that everything is going well.

Performance optimization

You could change the buffer size with -b/--buffer-size to increase the download speed from the source. If you know the source mailbox contains a lot of small mails use a higher size. In the case of lager mails use a lower size to counter timeouts. If you communicate via a bad internet connections you also should use a lower sized buffer.

Preventing timeouts

To prevent timeouts, both servers (the source and destination) will automatically be set into the IMAP idle mode. Most servers can hold this idle mode for 30 minutes. The idle mode restarts every 28 minutes (1680 seconds) so there should be no timeout. If a timeout occurs nevertheless you can change the restart interval by using --idle-interval followed by the desired number of seconds.

Use of source-folder argument

As a further optimization you can target specific folders you want to copy to the destination (versus the default of everything). Use -f / --source-folder to only copy that folder(s). The flag can be specified multiple times to indicate multiple folders to copy. The argument does support wildcard by using * and the end.

Copy only INBOX:

--source-folder INBOX 

Copy INBOX.Archives and all subfolders:

--source-folder INBOX.Archives.* --source-folder INBOX.Archives

You can also use this argument but please notice, that all folders started with INBOX.Archives (like INBOX.Archives123 and INBOX.ArchivesNew) will be copied too (if they exists).

--source-folder INBOX.Archives*

Microsoft Exchange Server IMAP bug

If your destination is an Microsoft Exchange Server (EX) you'll probably get a bad command exception while copying some mails. This happens because the EX analyses (and in some cases modifies) new mails. This is a bug in this lookup process (since EX version 5 😒). To prevent an exception you can use the argument --max-line-length 4096. This will skip all mails with lines more than 4096 characters.

You got broken pipe? This is also an Exchange bug feature. There is a limit of failures (by default three) in a single connection. Once you reach the limit, the server will disconnect you and pymap-copy will show an error for each further mail. Mostly these error occur because the size of the mail is larger than the maximum allowed size. The best way is to increase the limit (you need admin access to the server) by following these instructions. You can also exclude these mails from copy by using the --max-mail-size argument.

Encryption & Ports

By default, pymap-copy will use port 993 with ssl/tls. You can change this behavior by using --source-encryption/--destination-encryption and --source-port/--destination-port. If no port is specified, it will choose the default port based on the given encryption.

Possible encryption are tls, ssl (the same as tls), none and starttls.

Default Ports

Encryption	Port
tls	993
ssl	993
starttls	143
none	143

Credits

Created and maintained by Lukas Schulte-Tickmann / Schluggi.

pymap-copy's Issues

"logging" is missing

It seems that there is a missing definition of logging:

Getting source quota        : Traceback (most recent call last):
  File "/usr/local/Caskroom/miniforge/base/bin/pymap-copy.py", line 273, in <module>
    logging.info(f'Getting source quota...')
NameError: name 'logging' is not defined

Reconnect to server

By merging #13 i've removed some reconnects that are necessary for gmail.
I'll implement add them in a different way later.

Edit: I never had this issue. Maybe someone can post there experience. After i implement #15 the idle process should be prevent any timeout.

improve exception handling

Cleanup ssl arguments

Currently there are these arguments:

--source-no-ssl
--source-use-starttls

--destination-no-ssl
--destination-use-starttls

And this would be much cleaner and more simple to switch ports automatically.

--source-encryption=[ssl/tls/starttls/none]
--destination-encryption=[ssl/tls/starttls/none]

Add an option for logging

This idea comes from #13.

Limit mailsize

Hello Mr. Schluggi,
I would like to appreciate a "--max-mail-size" argument.

It would be very appreciated if you could set the max size in MB.

Thank you very much.

over all progress bar

Please add something like an overall progress bar. Thanks!

Improving idle

With #13 @doransmestad has implemented imap idle. Awesome improvement btw!

This works fine but at this moment the idle restarts every 10000 Mails.
In fact the idle only timeouts after 30min (based on this thread). I could improve this by starting a new thread witch restart the idle every 28 minutes.

Encoding error

Hi,

the server at mx.ccn.net gives the following error:

Getting source folders : Traceback (most recent call last):
File "/home/max/pymap-copy/pymap-copy/utils.py", line 57, in decode_mime
words.append(word.decode(encoding or 'utf8'))
LookupError: unknown encoding: dos-862

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./pymap-copy.py", line 220, in
subject = decode_mime(data[b'ENVELOPE'].subject)
File "/home/max/pymap-copy/pymap-copy/utils.py", line 62, in decode_mime
words.append(fallback(word))
File "/home/max/pymap-copy/pymap-copy/utils.py", line 43, in fallback
return mime_bytes.decode(charset)
TypeError: decode() argument 'encoding' must be str, not None

TypeError: '>' not supported between instances of 'int' and 'NoneType'

Hi. Encountered such an issue.
size or args.max_mail_size seems to be None in my case.

[  0.0%] Progressing... (buffer 1/1) (mail 1/26) (14.2 MB) (2016-01-18 16:09:01): Fw:Traceback (most recent call last):
  File "pymap-copy.py", line 558, in <module>
    elif size > args.max_mail_size:
TypeError: '>' not supported between instances of 'int' and 'NoneType'

get quota in percent

Error regarding quota

Hi,

I get the following error:

Getting source quota : Traceback (most recent call last):
File "./pymap-copy.py", line 161, in
source_quota = source.get_quota()[0]
IndexError: list index out of range

I commented the line(s) out but wanted to let you know anyway.

Thx

add option to list all destination folders

In some cases you don`t know the destination folder structure. It would be nice to have an option to list all of them.

pymap-copy incorrectly copies folders with names containing a dot "."

On the source box, I had a folder with a dot "." in the name (eg "part1.part2"). When copying, pymap-copy created a nesting like "part1/part2" (i.e. part2 became a separate folder inside part1).

No delimiter defiend when no mails

If one of the mailboxes has no mails, pymap-copy can't determine the delimiter and the tool will crash.

This will be fixed with #17.

Password with special character quote notice

If the --source-pass has a special character like ! the applications throws an error and removes the --source-pass from the command in the history.

If you quote the password with " the same error occurs.

Please add a note that if the password has a special character you need to quote with '.

Add support for multiple namespaces

At the moment only one namespace is supported.

There should be support for multiple namespaces.
As shown here: https://tools.ietf.org/html/rfc2342.html

add possibility to define source and destination folder?

Can you add something like --from and --to to specify a source and destination folder?

add possibility to link folders

add option to disable colors

--no-colors would be nice if you want to run the script on windows/in powershell.

add support for self signed certificates

add support for self signed certificates

[README] Simple usage .py missing

Simple usage command is ./pymap-copy \ which results in an error:

bash: ./pymap-copy: file or directory not found

Please update the command to ./pymap-copy.py \

Check necessity for two source/mailbox arguments

There are two arguments to define the source root:

--source-root
--source-mailbox

Is this really necessary?