marlam/mpop

POP3 client

mpop's Issues

1.4.15 build fails on macOS

Hello, while attempting to package mpop 1.4.15 on macOS for the Homebrew package manager, we encountered this error:

mpopd.c:277:41: error: no member named 'st_mtim' in 'struct stat'
            mail.mtime.tv_sec = statbuf.st_mtim.tv_sec;
                                ~~~~~~~ ^
mpopd.c:278:42: error: no member named 'st_mtim' in 'struct stat'
            mail.mtime.tv_nsec = statbuf.st_mtim.tv_nsec;
                                 ~~~~~~~ ^
2 errors generated.

It looks like macOS is not fully POSIX compliant in this regard. There have been similar issues in the past involving other open-source software (that have since been resolved): zstd, optipng. Thanks!

IMAP Support?

Hi,

I was reading the mpop homepage, and noticed this section:

• IMAP support (or rather, the ability to retrieve mails from IMAP mailboxes).

  mpop does not currently support this, but it is certainly possible to add this feature if there is interest.

In discussions with other users, I think I can say that there exists an interest in adding IMAP support, either next to POP or as it's own separate program.

Unless something has changed, I hope this issue could serve as a foundation to start implementing this feature, discuss changes, etc. And if there's any way to help, I'd gladly do so.

feature request: transform email

It would be nice if mpop could, on the fly, transform (headers of) emails. Something like a transform prog option that runs prog with the email on stdin, and uses the stdout as the transformed email.

Rationale: I'm fetching emails from an Outlook server, and storing them on a private server (maildir). Gmail is then accessing my private server and fetching these. Problematically, Outlook adds "Received" headers, and Gmail complains that Outlook is not a permitted sender for whichever address the email originated from. I'd like to "sanitize" the email by flushing the "Received" headers from Outlook.

Can't fetch from POP3s

Hi, thanks for mpop, I'm trying to play a bit with it.

I need to fetch mail from a postfix configured with TLS on port 995. I copy-pasted the example configuration and I am trying to retrieve the fingerprint of the SSL key, as suggested but the connection seems to hang:

$ mpop -d --timeout 3 myaccount
using account myaccount from /home/$USER/.config/mpop/config
host = mail.domain.org
port = 995
source ip = (not set)
proxy host = (not set)
proxy port = 0
timeout = 3 seconds
pipelining = auto
received_header = on
auth = choose
user = ...
password = *
passwordeval = (not set)
ntlmdomain = (not set)
tls = on
tls_starttls = on
tls_trust_file = system
tls_crl_file = (not set)
tls_fingerprint = (not set)
tls_key_file = (not set)
tls_cert_file = (not set)
tls_certcheck = on
tls_min_dh_prime_bits = (not set)
tls_priorities = (not set)
delivery = mbox /home/$USER/Mail/inbox
uidls file = ...
only_new = on
keep = off
killsize = off
skipsize = off
filter = (not set)
mpop: network read error: the operation timed out
mpop: error during mail retrieval (account domain.org from /home/$USER/.config/mpop/config)

this is my config file

defaults
tls on
delivery mbox ~/Mail/inbox
account myaccount
host mail.domain.org
user [email protected]
password asdfghjk
port 995

Same error when I try to fetch the SSL fingerprint with
mpop --serverinfo --tls --tls-certcheck=off --host=mail.domain.org

I'm using

$ mpop --version
mpop version 1.4.3
Platform: x86_64-pc-linux-gnu
TLS/SSL library: GnuTLS
Authentication library: GNU SASL; user and apop: built-in
Supported authentication methods:
user plain scram-sha-1 external gssapi apop cram-md5 digest-md5 login ntlm 
IDN support: enabled
NLS: enabled, LOCALEDIR is /usr/share/locale
Keyring support: none
Configuration file name: /home/$USER/.config/mpop/config

I think I'm doing something wrong but don't understand what. Fetching mail with Thunderbird works just fine

thanks!

AUTH XOAUTH2 may require a line break.

I'm connecting to outlook.office365.com via POP3s, and it's rejecting my auth. I had used the OAuth2 script contributed to Mutt to generate my tokens, and the script itself has a "test" mode to see if the authentification is accepted over POP3s; it worked for them. Exploring further, they send the AUTH line in two steps:

https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_oauth2.py#L393

That is, first AUTH XOAUTH2\r\n then the base64 payload. As far as I see, both syntaxes are OK from the standard point of view, so it might be due to a limit in the length of a line sent. In any case, I've simply changed the following in mpop, and it works perfectly now:

--- src/pop3.c  2021-11-04 07:02:01.000000000 +0000
+++ src/pop3.c      2022-08-24 02:45:13.544564722 +0100
@@ -2351,7 +2351,9 @@
     b64_len = BASE64_LENGTH(oa_len) + 1;
     b64 = xmalloc(b64_len);
     base64_encode(oauth, oa_len, b64, b64_len);
-    e = pop3_send_cmd(session, errstr, "AUTH XOAUTH2 %s", b64);
+    e = pop3_send_cmd(session, errstr, "AUTH XOAUTH2");
+    if (e == POP3_EOK && (e = pop3_get_msg(session, 1, errstr)) == POP3_EOK)
+      e = pop3_send_cmd(session, errstr, "%s", b64);
     free(oauth);
     free(b64);
     if (e != POP3_EOK)

SCRAM-SHA-1(-PLUS) + SCRAM-SHA-256(-PLUS) + SCRAM-SHA-512(-PLUS) + SCRAM-SHA3-512(-PLUS) supports

CRAM-MD5 to Historic:

• https://tools.ietf.org/html/draft-ietf-sasl-crammd5-to-historic-00 // 20 November 2008
• https://tools.ietf.org/html/draft-zeilenga-luis140219-crammd5-to-historic-00 // June 29, 2017

RFC6331: Moving DIGEST-MD5 to Historic:

• https://tools.ietf.org/html/rfc6331 since July 2011

RFC 8600:
"When using the SASL SCRAM mechanism, the SCRAM-SHA-256-PLUS variant SHOULD be preferred over the SCRAM-SHA-256 variant, and SHA-256 variants [RFC7677] SHOULD be preferred over SHA-1 variants [RFC5802]".

SCRAM-SHA-1(-PLUS):

• RFC5802: Salted Challenge Response Authentication Mechanism (SCRAM) SASL and GSS-API Mechanisms: https://tools.ietf.org/html/rfc5802
• RFC6120: Extensible Messaging and Presence Protocol (XMPP): Core: https://tools.ietf.org/html/rfc6120

SCRAM-SHA-256(-PLUS):

• RFC7677: SCRAM-SHA-256 and SCRAM-SHA-256-PLUS Simple Authentication and Security Layer (SASL) Mechanisms: https://tools.ietf.org/html/rfc7677 - since 2015-11-02
• RFC8600: Using Extensible Messaging and Presence Protocol (XMPP) for Security Information Exchange: https://tools.ietf.org/html/rfc8600 - since 2019-06-21: https://mailarchive.ietf.org/arch/msg/ietf-announce/suJMmeMhuAOmGn_PJYgX5Vm8lNA

SCRAM-SHA-512(-PLUS):

• https://tools.ietf.org/html/draft-melnikov-scram-sha-512

SCRAM-SHA3-512(-PLUS):

• https://tools.ietf.org/html/draft-melnikov-scram-sha3-512

-PLUS variants:

• RFC5056: On the Use of Channel Bindings to Secure Channels: https://tools.ietf.org/html/rfc5056
• RFC5929: Channel Bindings for TLS: https://tools.ietf.org/html/rfc5929
• Channel-Binding Types: https://www.iana.org/assignments/channel-binding-types/channel-binding-types.xhtml
• RFC 9266: Channel Bindings for TLS 1.3: https://tools.ietf.org/html/rfc9266

LDAP:

• RFC5803: Lightweight Directory Access Protocol (LDAP) Schema for Storing Salted: Challenge Response Authentication Mechanism (SCRAM) Secrets: https://tools.ietf.org/html/rfc5803

HTTP:

• RFC7804: Salted Challenge Response HTTP Authentication Mechanism: https://tools.ietf.org/html/rfc7804

2FA:

• Extensions to Salted Challenge Response (SCRAM) for 2 factor authentication: https://tools.ietf.org/html/draft-melnikov-scram-2fa

IANA:

• Simple Authentication and Security Layer (SASL) Mechanisms: https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml

Note, after SCRAM-SHA-1(-PLUS):

• GNU SASL (gsasl) supports SCRAM-SHA-256(-PLUS) since 1.9.1: http://git.savannah.gnu.org/gitweb/?p=gsasl.git;a=blob;f=NEWS;hb=HEAD
• Dovecot supports SCRAM-SHA-256(-PLUS) since 2.3.10: https://doc.dovecot.org/configuration_manual/authentication/authentication_mechanisms/
• Cyrus SASL/IMAP supports SCRAM-SHA-256(-PLUS) and more since 2.1.27: https://www.cyrusimap.org/sasl/sasl/authentication_mechanisms.html
• ...

Linked to:

• scram-sasl/info#1
• marlam/msmtp#36

Microsoft broke my config

My microsoft email setup using mutt_oauth2.py and the xoauth auth method in mpop worked up until July 6th, when it broke. mpop: POP3 server does not support authentication method XOAUTH2

I have tried letting mpop automatically select an auth method but it rejects my auth token. I don't know how to configure it to make microsoft auth work.

Segfault after retrieving all messages

It's possible that this is a configuration issue, so I am including my configuration here.

Mpop seems to segfault after downloading all messages from the target, but before the DELE's take place. This results in messages being downloaded repeatedly. I have tried this on versions 1.4.13 and 1.2.6(latest from distro repository). Let me know what other information I need to provide.

Config:

account aol
        host pop.aol.com
        port 995
        timeout 5
        auth on
        user ***********
        password **********
        tls on
        tls_starttls off
        tls_certcheck off
        delivery maildir /home/mail/local/
account default : aol

Last few lines from an strace

write(5, "Received: from pop.aol.com (jpop"..., 3908) = 3908
write(1, "100\n", 4100
)                    = 4
fsync(5)                                = 0
close(5)                                = 0
link("tmp/1631233776.M510392P23924Q5.onyx", "new/1631233776.M510392P23924Q5.onyx") = 0
unlink("tmp/1631233776.M510392P23924Q5.onyx") = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

Crash consistency of maildir delivery?

Hello,

I am reading the source here https://github.com/marlam/mpop-mirror/blob/master/src/delivery.c#L389-L423 and am wondering if it is possible to lose e-mails after a crash (very inconvenient especially if the server doesn't keep messages).

As I understand it, mpop writes the received e-mail in a temp file, fsync()s it, link()s it to its final pathname then unlink()s the temp file. Once mpop is done, the link()/unlink() calls have yet to hit the disk and the filesystem can reorder them. So it's possible that, after a crash, unlink() was persisted to disk, but not link(), resulting in a lost e-mail.

To my understanding, POSIX does not guarantee ordering of such operations, and many filesystems don't either.

One fix would be to fsync() the newly linked file and its directory entry before unlinking, like below:

diff --git a/src/delivery.c b/src/delivery.c
index c4acb62..09d7a0b 100644
--- a/src/delivery.c
+++ b/src/delivery.c
@@ -390,6 +390,7 @@ int delivery_method_maildir_close(delivery_method_t *dm, char **errstr)
 {
     maildir_data_t *maildir_data;
     char *newfilename;
+    int newfd;
 
     maildir_data = dm->data;
     if (fsync(fileno(dm->pipe)) != 0)
@@ -414,6 +415,32 @@ int delivery_method_maildir_close(delivery_method_t *dm, char **errstr)
         free(newfilename);
         return DELIVERY_EIO;
     }
+    if ((newfd = open(newfilename, O_RDONLY)) < 0) {
+        *errstr = xasprintf(_("cannot open %s%c%s: %s"), maildir_data->maildir,
+                            PATH_SEP, newfilename, strerror(errno));
+        free(newfilename);
+        return DELIVERY_EIO;
+    }
+    if (fsync(newfd) != 0) {
+        *errstr = xasprintf(_("cannot sync %s%c%s: %s"), maildir_data->maildir,
+                            PATH_SEP, newfilename, strerror(errno));
+        free(newfilename);
+        return DELIVERY_EIO;
+    }
+    close(newfd);
+    if ((newfd = open(maildir_data->maildir, O_RDONLY)) < 0) {
+        *errstr = xasprintf(_("cannot open %s: %s"), maildir_data->maildir,
+                  strerror(errno));
+        free(newfilename);
+        return DELIVERY_EIO;
+    }
+    if (fsync(newfd) != 0) {
+        *errstr = xasprintf(_("cannot sync %s: %s"), maildir_data->maildir,
+                  strerror(errno));
+        free(newfilename);
+        return DELIVERY_EIO;
+    }
+    close(newfd);
     (void)unlink(maildir_data->filename);
     free(newfilename);
     free(maildir_data->filename);

feature: set domain to verify TLS manually

In some cases, it is desirable to verify a TLS certificate against a hostname that is not the FQDN of the certificates Common or Alt Name.

This could be an IP-address (IPs of mailservers rarely change, no point to check for a new one every few minutes via an unencrypted and unsigned protocol like DNS) - or a .onion address as Tor Onion Service.

It would be awesome to allow this in mpop (and msmtp). In order to use an IP or onion as $host combined with $tls_trust_file, I would propose to add something like a $tls_verify_override_host setting which is verified against the hostname in the certificate.

An example:

account [email protected]
   host 192.0.2.0
   tls_trust_file /etc/ca-certificates/extracted/cadir/DST_Root_CA_X3.pem
   tls_verify_override_host example.org

This isn't completely new, f.e. unbound does something like this for DNS-over-TLS:

forward-addr: 1.1.1.1#cloudflare-dns.com

Thanks, and keep up the good work!

Vim syntax: \s+ instead of space

When I do alignment using spaces syntax shows errors. Tab alignment is fine. Not fair. :)

Not sure about current solution. But simplest one just replace spaces with \s+.

diff --git a/mpop.vim b/mpop.vim
index 42658787..9ea86d45 100644
--- a/mpop.vim
+++ b/mpop.vim
@@ -27,15 +27,15 @@ syntax match mpopOption /\<\(tls\|tls_starttls\|tls_trust_file\|tls_crl_file\|tl
 syntax match mpopOption /\<\(only_new\|keep\|killsize\|skipsize\|filter\|received_header\)\>/
 
 " Options which accept only an on/off value.
-syn match mpopWrongOption /\<\(tls\|tls_starttls\|tls_certcheck\|only_new\|keep\|received_header\) \(on$\|off$\)\@!.*$/
+syn match mpopWrongOption /\<\(tls\|tls_starttls\|tls_certcheck\|only_new\|keep\|received_header\)\s+\(on$\|off$\)\@!.*$/
 " Options which accept only an on/off/auto value.
-syn match mpopWrongOption /\<\(pipelining\) \(on$\|off$\|auto$\)\@!.*$/
+syn match mpopWrongOption /\<\(pipelining\)\s+\(on$\|off$\|auto$\)\@!.*$/
 " Options which accept numeric values.
-syn match mpopWrongOption /\<\(port\|proxy_port\|killsize\|skipsize\) \(\d\+$\)\@!.*$/
+syn match mpopWrongOption /\<\(port\|proxy_port\|killsize\|skipsize\)\s+\(\d\+$\)\@!.*$/
 " Option timeout accepts off and numeric values.
-syn match mpopWrongOption /\<timeout \(off$\|\d\+$\)\@!.*$/
+syn match mpopWrongOption /\<timeout\s+\(off$\|\d\+$\)\@!.*$/
 " Option auth accepts on, off and the method.
-syn match mpopWrongOption /\<auth \(on$\|off$\|user$\|plain$\|cram-md5$\|digest-md5$\|scram-sha-1$\|scram-sha-256$\|gssapi$\|external$\|login$\|ntlm$\|oauthbearer\|xoauth2\)\@!.*$/
+syn match mpopWrongOption /\<auth\s+\(on$\|off$\|user$\|plain$\|cram-md5$\|digest-md5$\|scram-sha-1$\|scram-sha-256$\|gssapi$\|external$\|login$\|ntlm$\|oauthbearer\|xoauth2\)\@!.*$/
 
 " Marks all wrong option values as errors.
 syn match mpopWrongOptionValue /\S* \zs.*$/ contained containedin=mpopWrongOption
diff --git a/msmtp.vim b/msmtp.vim
index a1cd2fc6..110736c3 100644
--- a/msmtp.vim
+++ b/msmtp.vim
@@ -27,13 +27,13 @@ syntax match msmtpOption /\<\(tls\|tls_starttls\|tls_trust_file\|tls_crl_file\|t
 syntax match msmtpOption /\<\(from\|from_full_name\|allow_from_override\|dsn_notify\|dsn_return\|set_from_header\|set_date_header\|set_msgid_header\|remove_bcc_headers\|undisclosed_recipients\|logfile\|logfile_time_format\|syslog\|aliases\|auto_from\|maildomain\)\>/
 
 " Options which accept only an on/off value.
-syn match msmtpWrongOption /\<\(tls\|tls_starttls\|tls_certcheck\|allow_from_override\|remove_bcc_headers\|undisclosed_recipients\|auto_from\) \(on$\|off$\)\@!.*$/
+syn match msmtpWrongOption /\<\(tls\|tls_starttls\|tls_certcheck\|allow_from_override\|remove_bcc_headers\|undisclosed_recipients\|auto_from\)\s+\(on$\|off$\)\@!.*$/
 " Options which accept only an on/off/auto value.
-syn match msmtpWrongOption /\<\(set_from_header\) \(on$\|off$\|auto$\)\@!.*$/
+syn match msmtpWrongOption /\<\(set_from_header\)\s+\(on$\|off$\|auto$\)\@!.*$/
 " Options which accept only an off/auto value.
-syn match msmtpWrongOption /\<\(set_date_header\|set_msgid_header\) \(auto$\|off$\)\@!.*$/
+syn match msmtpWrongOption /\<\(set_date_header\|set_msgid_header\)\s+\(auto$\|off$\)\@!.*$/
 " Option port accepts numeric values.
-syn match msmtpWrongOption /\<\(port\|proxy_port\) \(\d\+$\)\@!.*$/
+syn match msmtpWrongOption /\<\(port\|proxy_port\)\s+\(\d\+$\)\@!.*$/
 " Option timeout accepts off and numeric values.
 syn match msmtpWrongOption /\<timeout \(off$\|\d\+$\)\@!.*$/
 " Option protocol accepts smtp and lmtp.

mistaking for included 822 header as a new message?

greetings. i received an odd message today, with very few Received-By lines, etc. here it is as it appears in my inbox:

small-minshall_hv.txt

it appears it should have been part of a larger message (from my nospam.mbox -- see below):

weird-minshall_hv.txt

(in my inbox, the latter message was truncated, i.e., did not included the embedded message.)

i run procmail. i have a line at the beginning of my procmail rc file

:0 c
$BULKMAIL/nospam.mbox

which contained the above "weird" text.

also in my procmail rc file, i do

LOGABSTRACT=all
LOGFILE=$BULKMAIL/procmail-log.txt

and, for what seems the right point in procmail-log.txt, i see:

 Subject: new nongnu elpa package candidate: cpupower
  Folder: rcvstore +emacs-devel                                            5512
 Subject: [PATCH] * elpa-packages (cpupower): New package
  Folder: rcvstore +inbox                                                  2671

which i take to mean procmail was invoked twice, once for the "envelope", once for the embedded message.

my .mpoprc is pretty generic. it includes

# Deliver mail via procmail:
delivery mda /usr/bin/procmail ~/.procmailrc-client

i'm running

bash apollo2 (master): {50532} mpop --version
mpop version 1.4.13
Platform: x86_64-pc-linux-gnu
TLS/SSL library: GnuTLS
Authentication library: built-in
Supported authentication methods:
user plain external apop cram-md5 login oauthbearer xoauth2
IDN support: disabled
NLS: enabled, LOCALEDIR is /usr/local/share/locale
Keyring support: Gnome
Configuration file name: /home/minshall/.mpoprc

Copyright (C) 2021 Martin Lambers and others.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

i'll upgrade to 1.4.15, but i don't see anything in the git log that looks like a candidate.

i notice that the above inclusion was embedded with >From (rather than > From with a space), but i notice several other messages in nospam.mbox that lack the space, and appear to have been received correctly.