isbg/isbg

THIS PROJECT HAS BEEN MIGRATED TO GITLAB.COM: https://github.com/isbg/isbg/issues/131

IMAP Spam Begone

isbg is a script and a python 3 module that makes it easy to scan an IMAP inbox for spam using SpamAssassin and get your spam moved to another folder.

Unlike the normal mode of deployments for SpamAssassin, isbg does not need to be involved in mail delivery, and can run on completely different machines to where your mailbox actually is. So this is the perfect tool to take good care of your ISP mailbox without having to leave it.

You can read the full documentation in Read the docs.

Table of Contents

• Support
• Features
• Installation
  • Dependencies
  • Install via pip (easiest method)
  • Install from source
  • Uninstall from source
  • Building Windows Installer
• Usage
  • SpamAssassin
    • Configure your spamassassin
  • CLI Options
  • How does it work?
  • Multiple accounts
  • Saving your password
  • SSL
  • Read and Seen flags
  • Gmail Integration
  • Ignored emails
  • Partial runs
• Contact and about
• License

Support

Please note, the current release of isbg only supports python 3. Python 2 is compatible with older releases.

Features

• Works with all common IMAP servers
• Works on Linux, MacOS X and Windows (even smartphones!)
• Can do IMAP over SSL
• Can remember your password
• Will work painlessly against multiple IMAP accounts and servers
• Is not involved in the mail delivery process, and so can run on any machine that can contact your IMAP server
• Highly configurable
• Sensible defaults so you don't have to do any configuring :-)
• Possibility to skip spam detection to stick only to the teach feature
• Don't fail when meeting horrible and bad formed mail
• Lock file to prevent multiple instance to run at the same time (for cron jobs)

Installation

isbg install a python package module and also a script to use it isbg, it also install another script to unwrap messages: isbg_sa_unwrap.

Dependencies

isbg is written in the Python language. Python is installed by default on most Linux systems. You can can find out more about Python at Python home page.

Make sure you have SpamAssassin installed. All the necessary information can be found on the SpamAssassin wiki. SpamAssassin should be on your $PATH (it installs in /usr/bin/ by default)

To run, isbg also depends on some python modules.

• docopt for command line options.
• cchardet or chardet for encoding detection.
• xdg to found the .cache directory. xdg is not required, if it's not installed, isbg will try to found .cache.

Install via pip (easiest method)

Install as user:

$ pip3 install isbg --user

Install as root:

$ sudo pip3 install isbg

To list the files installed:

$ pip3 show isbg --files

And to uninstall it:

$ pip3 uninstall isbg

Install from source

Download a copy of the isbg repository onto your machine:

$ cd ~
$ git clone https://github.com/isbg/isbg.git

Install program dependencies:

$ cd /isbg
$ python3 setup.py install -r installed_files.txt

Files will be installed under /usr/local/. installed_files.txt contains a list of all the dependencies installed.

Uninstall from source

Uninstall program dependencies:

$ tr '\n' '\0' < installed_files.txt | xargs -0 rm -vf --

Building Windows Installer

Run the following to build a windows installer:

$ python3 setup.py bdist_wininst

Usage

SpamAssassin

If you have never used SpamAssassin before, you'll probably be quite nervous about it being too good and taking out legitimate email, or not taking out enough spam. It has an easily adjustable threshold to change how aggressive it is. Run the following command to create your preferences file:

$ spamassassin  </dev/null >/dev/null
Created user preferences file: /home/rogerb/.spamassassin/user_prefs

You can then edit $HOME/.spamassassin/user_prefs and change the thresholds.

You can also edit the system-wide settings in /etc/spamassassin/locals.cf.

If you want to use the --learnspambox or --learnhambox, you'll have to configure your spamassassin.

Configure your spamassassin

If you want to use --learnspambox or --learnhambox features, you have to add this configuration:

Allow Tell

You have to start spamd with the --allow-tell option.

On Debian systems (Debian and Ubuntu), you have to edit /etc/default/spamassassin and replace:

OPTIONS="-D --create-prefs --max-children 5 --helper-home-dir"

by:

OPTIONS="-D --allow-tell --create-prefs --max-children 5 \
--helper-home-dir"

Don't forget to restart your spamd server after that (sudo service spamassassin restart on Debian).

Setup a DNS cacher

By default, SpamAssassin will perform many DNS lookups for NetworkTests to significantly improve scoring of messages primarily by DNSBlocklists like Spamhaus, SORBS, etc. If you don't use a DNS cacher and you run isbg often, chances are you will hit the maximum number of free accepted queries on certain hosts and will get blocked.

To improve performance, scoring and make sure DNSBlocklists don't start blocking you, it is a good idea to set up a DNS cacher.

Using spamc to get better performances

By default, SpamAssassin is spooled up and down each and every time isbg has an email to scan.

You can improve isbg's performance by running SpamAssassin as a daemon and by telling isbg to use that instead of running a regular process.

On most Linux distributions, you can run SpamAssassin as a daemon this way:

$ systemclt enable spamassassin
$ systemctl start spamassassin

You can then run isbg with the --spamc option to make use of the daemon.

CLI Options

The default behavior of isbg is to not make any changes your Inbox unless you specify specific command line options. Consequently you can experiment without worry at the beginning.

Your first step is to create a new folder to receive suspected spam. I use one named 'spam'.

Run isbg with the --help option to see what options are available or check its manual page with $ man isbg [1].

You can also unwrap SpamAssassin bundled emails with isbg_sa_unwrap, you can use the --help option to check the available options or $ man isbg_sa_unwrap to check its manual page [2].

[1]	The rendered man page can be read in the "Manual page for isbg" section of this documentation.

[2]	The rendered man page can be read in the "Manual page for isbg_sa_unwrap" section of this documentation.

How does it work?

IMAP assigns each message in a folder a unique id. isbg scans the folder for messages it hasn't seen before, and for each one, downloads the message and feeds it to SpamAssassin. If SpamAssassin says the message is spam, then the SpamAssassin report is uploaded into your spam folder. Unless you specify the --noreport option, in which case the message is copied from your Inbox to the Spam folder (the copy happens on the IMAP server itself so this option is good if you are on a low bandwidth connection).

Multiple accounts

By default isbg saves the list of seen IMAP message unique IDs in a file in your home directory. It is named .isbg-trackXXXX where XXXX is a 16 byte identifier based on the IMAP host, username and port number. Consequently you can just run isbg against different servers/accounts and it will automatically keep the tracked UIDs separate. You can override the filename with --trackfile.

To run isbg for multiple accounts one after another, it is possible to use bash scripts like the ones in the examples directory. Since these scripts contain passwords and are thus sensitive data, make sure the file permissions are very restrictive.

Saving your password

If you don't want isbg to prompt you for your password each time, you can specify the --savepw option. This will save the password in a file in your home directory. The file is named $HOME/.cache/isbg/.isbg-XXXX where XXXX is a 16 byte identifier based on the IMAP host, username and port number (the same as for the multiple accounts description above). You can override the filename with --passwdfilename.

The password is obfuscated, so anyone just looking at the contents won't be able to see what it is. However, if they study the code to isbg then they will be able to figure out how to de-obfuscate it, and recover the original password. (isbg needs the original password each time it is run as well).

Consequently you should regard this as providing minimal protection if someone can read the file.

SSL

isbg can do IMAP over SSL if your version of Python has been compiled with SSL support. Since Python 2.6, SSL comes built in with Python.

However you should be aware that the SSL support does NOT check the certificate name nor validate the issuer. If an attacker can intercept the connection and modify all the packets flowing by, then they will be able to pose as the IMAP server. Other than that, the connection will have the usual security features of SSL.

Read and Seen flags

There are two flags IMAP uses to mark messages, Recent and Seen. Recent is sent to the first IMAP client that connects after a new message is received. Other clients or subsequent connections won't see that flag. The Seen flag is used to mark a message as read. IMAP clients explicitly set Seen when a message is being read.

Pine and some other mailers use the Recent flag to mark new mail. Unfortunately this means that if isbg or any other IMAP client has even looked at the Inbox, the messages won't be shown as new. It really should be using Seen.

The IMAP specification does not permit clients to change the Recent flag.

Gmail Integration

Gmail has a few unique ways that they interact with a mail client. isbg must be considered to be a client due to interacting with the Gmail servers over IMAP, and thus, should conform to these special requirements for proper integration.

There are two types of deletion on a Gmail server.

• Type 1: Move a message to '[Gmail]/Trash' folder.

  This "removes all labels" from the message. It will no longer appear in any folders and there will be a single copy located in the trash folder. Gmail will "empty the trash" after the received email message is 30 days old.

  You can also do a "Normal IMAP delete" on the message in the trash folder to cause it to be removed permanently without waiting 30 days.

• Type 2: Normal IMAP delete flag applied to a message.

  This will "remove a single label" from a message. It will no longer appear in the folder it was removed from but will remain in other folders and also in the "All Mail" folder.

  Enable Gmail integration mode by passing --gmail in conjunction with --delete on the command line when invoking isbg. These are the features which are tweaked:

  • The --delete command line switch will be modified so that it will result in a Type 1 delete.
  • The --deletehigherthan command line switch will be modified so that it will results in a Type 1 delete.
  • If --learnspambox is used along with the --learnthendestroy option, then a Type 1 delete occurs leaving only a copy of the spam in the Trash.
  • If --learnhambox is used along with the --learnthendestroy option, then a Type 2 delete occurs, only removing the single label.

Reference information was taken from gmail IMAP usage.

Ignored emails

By default, isbg ignores emails that are bigger than 120,000 bytes since spam are not often that big. If you ever get emails with score of 0 on 5 (0.0/5.0), it is likely that SpamAssassin is skipping it due to size.

Default maximum size can be changed with the use of the --maxsize option.

Partial runs

By default, isbg scans 50 emails for operation: spam learn, ham learn and spam detection. If you want to change the default, you can use the --partialrun option specifying the number. isbg tries to read first the new messages and tracks the before seen to not reprocess them.

This is useful when your inbox has a lot of emails, since deletion and mail tracking are only performed at the end of the run and full scans can take too long.

If you want that isbg does track all the emails you can disable the partialrun with --partialrun=0.

Contact and about

Please join our isbg mailing list if you use isbg or contribute to it! The mailing list will be used to announce project news and to discuss the further development of isbg.

You can also hang out with us on IRC, at #isbg on Freenode.

See the CONTRIBUTORS file in the git repository for more information on who wrote and maintains this software.

License

This program is licensed under the GNU General Public License version 3.

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

isbg's Issues

Replace CONTRIBUTORS file by a simple list

The current CONTRIBUTORS file has a weird hierarchy thing and IMHO it should be replace by a simple list of people who worked on the project.

I need to be able to do shorter runs

Hi,

First, thanks a lot for your wonderful tool! I have banged my head against a wall for so many hours until I found your tool.

I currently have over 6000 mails in my inbox that I need to clean up and doing a full run will take many hours; this is something that I can't simply do in one shot. The problem is that the trackfile and the deletion/expunge in the server does not happen until the very end of the operation! If I cancel the operation all reports are in the Spam folder, but the originals remain and the next run starts from the beginning. This means I will never be able to really clean my stuff.

I am going to implement a new parameter (--partialrun num) that will break the main loop once a certain number of emails have been processed. I think this should do the trick while being simple to develop and easy to understand. It might not be the ideal solution (deleting and writing to trackfile after each message is parsed) but that would be too costly for me and probably for you.

My question is whether you would be interested to see such feature into the master branch. Just let me know, and I will do the commits.

Thanks!

getmessage / assertok logic error

While working on #63 I stumbled over this weird logic:

# Retrieve the entire message
def getmessage(uid, append_to=None):
    res = imap.uid("FETCH", uid, "(RFC822)")
    assertok(res, 'uid fetch', uid, '(RFC822)')
    if res[0] != "OK":
        assertok(res, 'uid fetch', uid, '(RFC822)')
        try:
            body = res[1][0][1]
        except:
            if opts["--verbose"] is True:
                print("Confused - rfc822 fetch gave " + repr(res))
                print("""The message was probably deleted
                      while we were running""")
            if append_to:
                append_to.append(uid)
    else:
        body = res[1][0][1]
return body

#...

# This function checks that the return code is OK
# It also prints out what happened (which would end
# up /dev/null'ed in non-verbose mode)
def assertok(res, *args):
    if opts["--verbose"] is True:
        print(repr(args), "=", res)
    if res[0] != "OK":
        errorexit("\n%s returned %s - aborting\n"
% (repr(args), res), exitcodeimap)

Shouldn't the part of getmessage after if res[0] != "OK": never be reachable because assertok would errorexit?

What's the intent of that code?

0/x spams learnt

Running

isbg.py --imaphost xxxxx --imapuser xxxxx --imapinbox INBOX --spaminbox Junk --learnspambox learn_spam --verbose --teachonly

Yields:

(...)
('7', 'Message was already un/learned\n')
0/7 spams learnt

Even though some of the messages contained in the learn_spam folder were brand new and never before seen by the script... Why is that so?

Please remove link in github description

The link to http://redmine.ookook.fr/projects/isbg at the top of https://github.com/ook/isbg (description) has been broken for quite a time now. Please remove it.

Update README

So far the README hasn't been touched. It needs to be done.

Implement a progress indicator

When the default X messages loop will be implemented, it would be nice to have a progress indicator.

You'd have to scan the box for the number of unread messages, divide by X and then after each loop display a little message.

STDIN/OUT issue when run via cron in OS X

Hello @ook!

First, thanks for writing isbg. As a mutt user with a remote mail server that does really lousy spam filtering, this is like a dream come true. I've run into a bit of a snag running it under OS X:

X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=mstyne>
X-Cron-Env: <USER=mstyne>
X-Cron-Env: <HOME=/Users/mstyne>
Date: Tue,  7 Oct 2014 10:33:01 -0400 (EDT)

Traceback (most recent call last):
  File "isbg.py", line 537, in <module>
    p = Popen(satest, stdin=PIPE, stdout=PIPE, close_fds=True)
  File "/usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Running it from the shell works just fine. Am I missing something about the nature of cron (or maybe just a Python idiosyncrasy) that's keeping the script from accessing STDIN/STDOUT when invoked this way?

Thanks again,
Mike

Missing dependency: docopt

When I run a command like
python isbg.py --imapuser me --savepw --spaminbox Junk
I get the error "Missing dependency: docopt" and "Option processing failed - name 'docopt' is not defined"

I suppose that is because my new and shining Debian server does not have python-docopt installed. Unfortunately, there is no such package yet for Debian stable. Does that mean I'll have to stick to isbg 0.99?

ISBG not closing after a scan

I have found that ISBG is not closing after completing a scan, but still responds to ^c.
Most recently this was doing learnanddelete, all the spam box was deleted, and it seems that the script hung.

It is running on raspbian (latest), on a Pi 2.

Any thoughts please

Error on opening pipe to spamassassin executable

Hi,

I see the following error;

Traceback (most recent call last):
File "/home/john/Software/isbg/isbg.py", line 552, in
p = Popen(satest, stdin=PIPE, stdout=PIPE, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 394, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I am running on arch linux. I am checking the mail of 4 email addresses. I am running a cron job every 10 minutes. The first of the email addresses often fails with the above error (although not always).
The other email addresses are always succesful.
I have tried using spamc as well as spamd with the same result.

The command line in the script look like this....

/usr/bin/python2 ~/Software/isbg/isbg.py --imaphost mail1.server.co.uk --imapuser mail1 --delete --expunge >> ~/Logs/spamlogs.log 2>&1

Any suggestions would be helpful

Detected Spam not deleted

I run isbg.py against a testing inbox. There are 3 mail, 2 are spam using a test string.
SA is detecting the 2 spam mails correctly, but isbg.py does nothing.

That's my command:

isbg.py --imaphost $host --imapuser $user --imappasswd $pass --delete --flag --noreport --spamc

Please see my logs:

Feb 19 13:49:23 rapi-print spamd[4403]: spamd: still running as root: user not specified with -u, not found, or set to root, falling back to nobody
Feb 19 13:49:24 rapi-print spamd[4403]: spamd: processing message <[email protected]> for root:65534
Feb 19 13:49:29 rapi-print spamd[4403]: pyzor: check failed: internal error, python traceback seen in response
Feb 19 13:49:29 rapi-print spamd[4403]: spamd: identified spam (1000.6/5.0) for root:65534 in 5.9 seconds, 4014 bytes.
Feb 19 13:49:29 rapi-print spamd[4403]: spamd: result: Y 1000 - GTUBE,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RDNS_NONE,TO_NO_BRKTS_NORDNS,T_DKIM_INVALID,T_URIBL_SEM,T_URIBL_SEM_FRESH,T_URIBL_SEM_FRESH_10,T_URIBL_SEM_FRESH_15,T_URIBL_SEM_RED scantime=5.9,size=4014,user=root,uid=65534,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=32990,mid=<[email protected]>,autolearn=no
Feb 19 13:49:30 rapi-print spamd[4404]: spamd: connection from localhost [127.0.0.1] at port 32995
Feb 19 13:49:30 rapi-print spamd[4404]: spamd: setuid to root succeeded
Feb 19 13:49:30 rapi-print spamd[4404]: spamd: still running as root: user not specified with -u, not found, or set to root, falling back to nobody
Feb 19 13:49:30 rapi-print spamd[4404]: spamd: processing message <[email protected]> for root:65534
Feb 19 13:49:31 rapi-print spamd[4371]: prefork: child states: IB
Feb 19 13:49:36 rapi-print spamd[4404]: pyzor: check failed: internal error, python traceback seen in response
Feb 19 13:49:36 rapi-print spamd[4404]: spamd: clean message (0.6/5.0) for root:65534 in 6.8 seconds, 3720 bytes.
Feb 19 13:49:36 rapi-print spamd[4404]: spamd: result: . 0 - HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RDNS_NONE,TO_NO_BRKTS_NORDNS,T_DKIM_INVALID,T_URIBL_SEM,T_URIBL_SEM_FRESH,T_URIBL_SEM_FRESH_10,T_URIBL_SEM_FRESH_15,T_URIBL_SEM_RED scantime=6.8,size=3720,user=root,uid=65534,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=32995,mid=<[email protected]>,autolearn=no
Feb 19 13:49:37 rapi-print spamd[4403]: spamd: connection from localhost [127.0.0.1] at port 33000
Feb 19 13:49:37 rapi-print spamd[4403]: spamd: setuid to root succeeded
Feb 19 13:49:37 rapi-print spamd[4403]: spamd: still running as root: user not specified with -u, not found, or set to root, falling back to nobody
Feb 19 13:49:37 rapi-print spamd[4403]: spamd: processing message <[email protected]> for root:65534
Feb 19 13:49:38 rapi-print spamd[4371]: prefork: child states: BI
Feb 19 13:49:44 rapi-print spamd[4403]: pyzor: check failed: internal error, python traceback seen in response
Feb 19 13:49:44 rapi-print spamd[4403]: spamd: identified spam (1000.6/5.0) for root:65534 in 7.2 seconds, 6635 bytes.
Feb 19 13:49:44 rapi-print spamd[4403]: spamd: result: Y 1000 - GTUBE,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RDNS_NONE,T_DKIM_INVALID,T_URIBL_SEM,T_URIBL_SEM_FRESH,T_URIBL_SEM_FRESH_10,T_URIBL_SEM_FRESH_15,T_URIBL_SEM_RED scantime=7.2,size=6635,user=root,uid=65534,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=33000,mid=<[email protected]>,autolearn=no
Feb 19 13:49:44 rapi-print pi: spamprotect - Results on checking for [email protected]: 0 spams found in 3 messages#0120/0 was automatically deleted
Feb 19 13:49:45 rapi-print spamd[4371]: prefork: child states: II

Change the license

The license we are using at the moment is more a lack of license than anything. This is problematic for packaging isbg into Linux distros. For example, Debian won't accept it because of the unclear license.

With a 2.0 version, I feel that I'll have the moral right to change the license Roger Binns chose, since a lot of the code will have changed by then.

I'm very fond of the GPLv3. I wonder if this should be licensed under the AGPLv3 though since one of the use cases is people offering isbg as a service.

I'd like isbg users input on this. What license do you prefer? Do you mind the (A)GPLv3?

Bulk issues / comments from first time use

Hello, I encountered several issues:

• problem with --savepw ... I would have expected that I can leave that option, and that if it finds a PW file that it uses it (instead of asking for a PW again)
• problem with --passwdfilename ... that option name is confusing: I was expecting to give it a filename that it reads the PW from.... instead it did what I expected --savepw to do
• the username@[email protected] notation is VERY confusing to read, change it to [email protected] (server: mailserver.maildomain.com) or something similar
• by default it did NOT remove the spam from the inbox (as per the docs). It left it there. THis should be more clearly documented: either default is "leave the spam and copy it" (and delete only if --delete is specified)
• a progress indicator (Processing nnn/MMM mails...) would be helpful to show when working with larger inboxes
• result info is incorrect: 12 spams found in 294 messages /// 0/12 was automatically deleted ... this is with --delete --expunge, and the messages did get deleted fro the inbox and showed up in the spambox

I think that's it for now, thanks for a great tool otherwise!
Cheers
Nils

Change the way track files work

At the moment, ISBG writes into a track file (.isbg-trackxxxxxxxx) to keep a log of the message it has read.

This is not nice since:

• it clutters you working directory
• is not efficient for large scale uses
• you can't read the md5 hashsum and know what trackfile belongs to what user account easily

I think it would be best to use something like sqlite to replace the track file behavior. It's very efficient for such light use, easily exportable and gives us some way to organise data.

Please make a release

Your last release was in 2010. Many packagers go by releases only, so all your recent changes are never seen by users.

Thanks!

Last pull broke isbg

Hi,

Since today, I get :
/usr/local/bin/isbg.py --verbose --imaphost localhost --imapuser machin --imapinbox INBOX --spaminbox spam --delete
IMAP password for machin@:
Traceback (most recent call last):
File "/usr/local/bin/isbg.py", line 5, in
pkg_resources.run_script('isbg==0.99', 'isbg.py')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1441, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/isbg-0.99-py2.7.egg/EGG-INFO/scripts/isbg.py", line 400, in

File "/usr/lib/python2.7/imaplib.py", line 172, in init
self.open(host, port)
File "/usr/lib/python2.7/imaplib.py", line 238, in open
self.sock = socket.create_connection((host, port))
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

and

isbg.py --imaphost localhost --savepw 14-06-26 / 18:08
IMAP password for @localhost:
Traceback (most recent call last):
File "/usr/local/bin/isbg.py", line 5, in
pkg_resources.run_script('isbg==0.99', 'isbg.py')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1441, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/isbg-0.99-py2.7.egg/EGG-INFO/scripts/isbg.py", line 403, in

File "/usr/lib/python2.7/imaplib.py", line 519, in login
raise self.error(dat[-1])
imaplib.error: [AUTHENTICATIONFAILED] Authentication failed.

Though it should work (password is good and it was working till today)

Thanks

Raphaël

Create an "isbg" github organisation and transfer this repository to it

Since you're still active on that project, it's time to give you the keys :)

[feature request] Re-write subject with detected spam level when placing the message into 'filtered' folder

Integrate RSpamD as spam filtering system

Would it be possible to use RSpamD as alternative to spamassasin?

Bug in error handling the lock file

Im now using branch 2.0
Saw this error....
Traceback (most recent call last):
File "isbg.py", line 906, in
isbg_run()
File "isbg.py", line 903, in isbg_run
isbg.do_isbg()
File "isbg.py", line 803, in do_isbg
errorexit(self.exitcodelocked)
TypeError: errorexit() takes exactly 2 arguments (1 given)

Looks like a simple ommision of a string in the call to errorexit()

Command line parameters do not work

Other than --help, none of the other parameters work for me. They just display the same output as --help. For instance:

./isbg.py --imaphost localhost

displays the usage and the options. This is with python 2.7.5 and python-docopt 0.6.1-2.

Problems with Wiki page

I am unable to reach the wiki page.

Use spamassassin instead of spamc/spamd combination

Have you considered running isbg with spamassassin (binary) as a user, without the need of spamd --allow-tell? It would simplify the installation and let the user have full control over the whole thing in case of shared/protected environments.

--partialrun strange behaviour?

When using the command --partialrun (lets say 100), is it meant to check the first 100 emails, or the last. I noticed when testing this on my work email, which has around 4000 messages stored, it only reads the first 100 oldest messages (with uid 1,2,3 ect).
Also when running with the --partialrun multiple times, it ignores the messages already seen and starts going further on the message list (from 1 to 100, 101 to 200). I did get around this by modifying the uid check so, that it first gets the uids, then limits the amount of uids we want to handle and then checks if we have already gone through them.

SSL should be the default

Hi,

One can never be too paranoid... but having your password travel the internet in clear text is way too much. I think SSL should be the default protocol and not the option, in order to prevent mistakes (like I did).

May I ask: could you please turn the --ssl parameter into a --nossl parameter that does the opposite? This way the feature of not-using-ssl is there, but is only optional.

If you don't have the time I will be very happy to push it myself... but of course the code will not be backwards-compatible.

Thanks!

Daemon mode

right now I run isbg as cronjob. Everytime it logs in and scans the INBOX for new mails.

It would be nice if isbg would provide a daemon mode where it only connects once to the IMAP server and then waits for new incoming mails to be scanned.

https://en.wikipedia.org/wiki/IMAP_IDLE

Create a mailing list for developpment and questions

https://mail.python.org looks like they have some nice mailman lists available for python projects.

Spam not deleted

I'm using isbg with these options:

isbg --delete --expunge --imaphost localhost --imappasswd mypass --imapuser myuser@mydomain --imapinbox INBOX --noreport --partialrun 50 --spamc --spaminbox Junk

and I get this output:

10 spams found in 50 messages
0/10 was automatically deleted

Why the 10 spams were not deleted? I thought --delete --expunge would delete them.

modify the script to learn from forwarded mails

Hi,

i've created a spam inbox where everybody can forward their spams to. so some people send an email with multiple attachments there. the real spam is inside the attachments.
currently im using some awkward combination of fetchmail and some homebrew shell scripts with mu, to extract the attachments and then learn them.

could that be integrated with isbg? would be really cool.

Occasional error report from spamassassin call

I see the following error from time to time...
It doesn't halt the scanning of the imap directory, but I thought you might want to know about it.

('uid fetch', u'4413', '(BODY.PEEK[])') = ['OK', [['11 (UID 4413 BODY[] {3539}', "'Return-Path: [email protected]\r\nDelivered-To: [email protected]\r\nReceived: fro..."], ')']]
Error communicating with ['spamassassin', '--exit-code']!
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/isbg/isbg.py", line 567, in spamassassin
score = p.communicate(body)[0].decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2982: ordinal not in range(128)
('uid fetch', u'4414', '(BODY.PEEK[])') = ['OK', [['12 (UID 4414 BODY[] {24451}', "'Return-Path: [email protected]\r\nDelivered-To: [email protected]..."], ')']]
[4414] score: -6.7/5.0

spam score "0/0"

I'm using spamassassin and passing the --spamc option. When isbg finds an email message that its going to skip, it appears the score is returned as "0/0" from spamd. There is a check for "0/0\n" which causes isbg to exit with an error code.

Has spamassassin changed sometime in the past, or why is the 0/0 check even in the code? If I comment that out things run fine for me. Any insight as to why this is happening?

Can't run ISGB on multiple accounts in parallel when using `--imapproxyuser`

I reverted cbbc5e9 because:

• If you did not use --imapproxyuser, it crashed (can't concatenate str and None)
• we already set track files based on the IMAP host, username and port number.

I know this means you can't use the --imapproxyuser parameters if you want to run on multiple accounts in parallel.

IMHO, the way to go on this is to take imapproxyuser in account when creating the 16 byte identifier based on the IMAP host, username and port number.

Learnham should remove spamassassin report from mailcontent

This report is based on isbg V1.0.

During my first run with isbg some mails were falsely identified as spam.
After putting them into my learnhambox folder, they were correctly learned as ham and put back into my movehamto mailbox, however their e-mail content still contained the spamassassin spam-report.

I would have expected that the spam-report is removed during the process of identifying a mail as ham.

May I ask you to implement the feature that the spamassassin report is removed, when the mail is identified as ham.

I believe that the necessary changes should go about here: https://github.com/isbg/isbg/blob/1.00/isbg.py#L497
And spamassassin provides the --remove-markup command line option for this functionality.

The savepw and noninteractive options are not working in branch 2.0

Using the noninteractive with a saved password using the version 1.0 I get this error:

Password file is /home/carlesmu/.isbg-6f417d524d9da20263eedd2815c1425a
Error reading pw!
Traceback (most recent call last):
  File "isbg/isbg.py", line 816, in do_isbg
    self.imappasswd = self.getpw(dehexof(open(self.passwdfilename, "rb").read().decode()), self.passwordhash)
  File "isbg/isbg.py", line 321, in getpw
    c = ord(data[i]) ^ hash[i]
TypeError: unsupported operand type(s) for ^: 'int' and 'str'

Also, when I try to save the password I get this error:

Traceback (most recent call last):
  File "isbg/isbg.py", line 906, in <module>
    isbg_run()
  File "isbg/isbg.py", line 903, in isbg_run
    isbg.do_isbg()
  File "isbg/isbg.py", line 838, in do_isbg
    f.write(hexof(self.setpw(self.imappasswd, self.passwordhash)).encode())
  File "isbg/isbg.py", line 333, in setpw
    res = [chr(x) for x in res]
TypeError: an integer is required

Unclear option: --delete

I have trouble understanding the meaning behind the --delete option. Does setting it mean that:

• the message will be copied to spam folder AND deleted from inbox, or
• deleted from the server so there's no trace of it?

Make sure ISBG verifies server SSL certificates

Pretty sure ISBG does not verify SSL certs when connecting to a server. This is bad.

More on how to fix this here: https://blog.sucuri.net/2016/03/beware-unverified-tls-certificates-php-python.html

Python 3

Python 2 is dying slowly. All the code should (eventually) become python3 code.

imaplib.abort: socket error: EOF

I run isbg every 4 minutes on my RaspberryPI to check e-mail on my dreamhost account. Most of the time it works without any problem, but for some mails which spamassassin marks as spam isbg crashes with the below exception. Next time isbg will crash again and again, until I manually remove that specific spam-mail from my inbox:

Traceback (most recent call last):
  File "/opt/isbg/isbg.py", line 591, in <module>
    res = imap.append(spaminbox, None, None, body)
  File "/usr/lib/python2.7/imaplib.py", line 324, in append
    return self._simple_command(name, mailbox, flags, date_time)
  File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python2.7/imaplib.py", line 867, in _command
    while self._get_response():
  File "/usr/lib/python2.7/imaplib.py", line 916, in _get_response
    resp = self._get_line()
  File "/usr/lib/python2.7/imaplib.py", line 1011, in _get_line
    raise self.abort('socket error: EOF')
imaplib.abort: socket error: EOF

After reading this stackoverflow page, it seems I was able to hack around this problem by changing line 591 from this:

res = imap.append(spaminbox, None, None, body)

into this (obviously not a pretty hack for general use):

try:    
  res = imap.append(spaminbox, None, None, body)                                                                                
except imap.abort:
  imap = imaplib.IMAP4_SSL(imaphost, imapport)
  imap.login(imapuser, imappasswd)
  imap.select(imapinbox, 1)
  res = imap.append(spaminbox, None, None, body)

--learnthendestroy should not delete hams

Yep, title says it all. I decided to add this lovely option to my cron to delete accumulated spam, and guess what. No more emails...

Drop Windows support

I do not use Windows and thus can't check if it works on it. I also do not wish to jump through hoops to make Windows support functional.

With v.1.01 (PR #44), there are very good chances the Windows support will be dropped.

Some messages that aren't spam are placed into 'filtered' folder

Every once in a while I see the message that went through isbg, wasn't labeled as spam (level is lower than threshold), but still placed into filtered folder.

Stop execution (crtl+c)

Stopping execution with CTRL+C doesn't work and leaves the script in a dirty state (e.g. locks). It would be very useful to have the signal work to block a long running execution.

Replace the password obfuscation by clear text

I really dislike security through obscurity. The obfuscation scheme in isbg is bad and should be removed.

It'd be great if isbg could check a password file for imaphost,imapuser couples when it is run. If there is a password there, use it. If not, ask for one.

That way, the --savepw option can be removed and we can nuke that part of the code to make it simpler.

The file should be implemented the same way issue #48 is.

Will always ask for password

Hi,

I saved my password with the --savepw option and the password file file is generated and found.

However, the line 149 in hexdigit() will fail:
locals()[p[0][2:]] = opts["--lockfilegrace"]

(I do not know what it does anyway).

This means that the password will always be prompted for, because p is not known and the try in line 339 will fail.

I deleted the line 149 and now it works for me.

I do not know Python, so perhaps someone can tell me what that did?

Greeting from Hamburg,
Christian

Not keeping track of read emails

Branch 2.0.0 doesn't seem to keep track of emails which have been read and processed.

Using verbose I saw;

('uid fetch', '4413', '(BODY.PEEK[])') = ['OK', [[b'11 (UID 4413 BODY[] {3539}', "b'Return-Path: [email protected]\r\nDelivered-To: [email protected]\r\nReceived: fr..."], b')']]
[4413] score: 2.7/5.0

Writing pastuids, 0 origpastuids, newpastuids: []
0 spams found in 10 messages
0/0 was automatically deleted

At the end of a limited run. I would have expected to see a list of newpastuids, but it is empty.
Redoing the command processes exactly the same emails.

Looking at the code on line 991 it is;
body = self.getmessage(u, newpastuids)

getmessage appends the uids to a list using a named parameter, but the result is never passed back to the calling frame.
I therefore just added;
newpastuids.append(u)
After the getmessage command.

This seems to result in the desired behaviour.

--flag does nothing for me

Hi to all,
this is my call

isbg.py --imaphost $host --imapuser $user --imappasswd $pass --delete --flag --noreport --spamc

Accidentialy the --flag does nothing for me. I was expecting to get the mail subject marked with _SPAM_

Thanks, best Regards
\st

Make 2.0 branch python3-only

We've agreed to stop supporting python2.x in isbg.

Parse options from a configuration file

There should be a file where you can put some configuration so that you don't always have to input all the parameters you want when running isbg.

I like how pius does it.

The file should be save in XDG_CACHE_DIR by default, but it'd be neat to have a config file in /etc/isbg too if we are to package this properly.

Some code about XDG_CACHE we can use here (i wrote a big part a that function :P)

Make a website to showcase isbg

Maybe a readthedoc thingy? I'm not too fond of that, but it seems to work well for other projects.

I could always build a django website too.