Error while loading messages

[mzellho]

Error while loading messages

[zeugmatis]

I'm running Courier-imap as well and can confirm. Tested via telnet, and with IMAP_DISABLE_THREADSORT=0 it is fine, but changing it to IMAP_DISABLE_THREADSORT=1 throws the same error:

a OK [READ-ONLY] Ok
a UID SORT (DATE) US-ASCII UNDELETED
a NO This command is disabled by the system administrator.
a logout

...so it looks like the ISP has set IMAP_DISABLE_THREADSORT=1 (which is not the default). If so, they should also remove the keywords "SORT" and "THREAD" in IMAP_CAPABILITY setting as well, so that it does not advertise it anymore:

IMAP_CAPABILITY="IMAP4rev1 UIDPLUS CHILDREN NAMESPACE QUOTA IDLE"

...as far as app:mail, I guess upon error for the command it could fallback to same behavior as if SORT is not advertised. Looks like more of an imap server config issue though, since they disabled serverside THREAD and SORT but did not remove it from their IMAP_CAPABILITY setting as well.

[ChristophWurst]

IMAP error reported by server

Please try again after enabling IMAP logging. Hopefully the imap log will shed more light on what's failing there.

[mzellho]

Added imap log

[ChristophWurst]

C: 4 UID SORT (DATE) US-ASCII UNDELETED S: 4 NO This command is disabled by the system administrator.

Hm. That looks suspicious

[ChristophWurst]

S: * CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION

Well, the server seems to support SORT

[mzellho]

Hmm, any idea on how to proceed? I could get in touch with my provider, but they will most likely go "Well, other Mail-Clients work. Use ours, this one is supported". I have Roundcube running on the same server as well, works fine. Also, I could try to install horde and see if it works, but I'm not sure if I have all the necessary permissions for that...

[ChristophWurst]

I'm pretty sure

mail/lib/mailbox.php

Lines 93 to 147 in 3249331

	/**
	* @param integer $from
	* @param integer $count
	* @param string $filter
	*/
	private function getSearchIds($from, $count, $filter) {
	if ($filter instanceof Horde_Imap_Client_Search_Query) {
	$query = $filter;
	} else {
	$query = new Horde_Imap_Client_Search_Query();
	if ($filter) {
	$query->text($filter, false);
	}
	}
	if ($this->getSpecialRole() !== 'trash') {
	$query->flag(Horde_Imap_Client::FLAG_DELETED, false);
	}
	$result = $this->conn->search($this->mailBox, $query, ['sort' => [Horde_Imap_Client::SORT_DATE]]);
	$ids = array_reverse($result['match']->ids);
	if ($from >= 0 && $count >= 0) {
	$ids = array_slice($ids, $from, $count);
	}
	return new \Horde_Imap_Client_Ids($ids, false);
	}
	/**
	* @param integer $from
	* @param integer $count
	*/
	private function getFetchIds($from, $count) {
	$q = new Horde_Imap_Client_Fetch_Query();
	$q->uid();
	$q->imapDate();
	$result = $this->conn->fetch($this->mailBox, $q);
	$uidMap = [];
	foreach ($result as $r) {
	$uidMap[$r->getUid()] = $r->getImapDate()->getTimeStamp();
	}
	// sort by time
	uasort($uidMap, function($a, $b) {
	return $a < $b;
	});
	if ($from >= 0 && $count >= 0) {
	$uidMap = array_slice($uidMap, $from, $count, true);
	}
	return new \Horde_Imap_Client_Ids(array_keys($uidMap), false);
	}
	public function getMessages($from = 0, $count = 2, $filter = '') {
	if (!$this->conn->capability->query('SORT') && (is_null($filter) || $filter === '')) {
	$ids = $this->getFetchIds($from, $count);
	} else {
	$ids = $this->getSearchIds($from, $count, $filter);
	}

is the faulty code. But I don't know yet why it's failing there, sorry.

"UID SORT" seems to be the right command, see https://tools.ietf.org/html/rfc5256#section-4

Maybe we need to query the capability differently or more precisely…

[mzellho]

Well, sadly I don't have too much clue here, but maybe I can find some time in the evening to look into it. If you want me to try something, just let me know.

[ChristophWurst]

Well, sadly I don't have too much clue here, but maybe I can find some time in the evening to look into it. If you want me to try something, just let me know.

Sure, go ahead. Would be great if we could debug and fix this together :-)

[mzellho]

Christoph, did you find out anything in the meantime? I could be looking into it now - any (new) suggestions where to start?

[zeugmatis]

Random suggestion but maybe can you test directly with your account / ISP using telnet? Over CLI - then copy/paste here. Simple tutorial is here. (There are lots of them on the web) - maybe try to duplicate the issue by hand going along with what is in your imap logs.

[mzellho]

Sorry, my SSH does not allow me telnet, ps, lsof ... (maybe I should just change the provider), but in the meantime I found some interesting facts:

Like, reading my own logfile, that my provider is using Courier-IMAP, apparently a 2008 version. In Roundcube's Forum I learned that Courier used to disable SORT even if it advertises it (maybe it still does?) :

##NAME: IMAP_DISABLETHREADSORT:0
#
# Set IMAP_DISABLETHREADSORT to disable the THREAD and SORT commands -
# server side sorting and threading.
#
# Those capabilities will still be advertised, but the server will reject
# them.  Set this option if you want to disable all the extra load from
# server-side threading and sorting.  Not advertising those capabilities
# will simply result in the clients reading the entire folder, and sorting
# it on the client side.  That will still put some load on the server.
# advertising these capabilities, but rejecting the commands, will stop this
# silliness.
#
IMAP_DISABLETHREADSORT=0

I would not be surprised if this was the case for the version on my provider's server.

For the moment, I commented out lines 143 and 145-147 in mailbox.php in order to always call getFetchIds(). This is an "okay" workaround for me at the moment. I'll try to find out why my provider is running such an outdated version (there are more recent versions, up to 03-Aug-2016) and why they have IMAP_DISABLETHREADSORT set to 0.

I know it will probably be impossible to get along with fake-advertised capabilities, but do you think you could somehow handle such a "lie" in a future version your app? If I can help, please just let me know. I wonder, how Roundcube does it. Maybe I can get their imap-logs and see what they do.

BTW: now, that I can see and use the app: pretty cool, so far. Keep up the good work :-)

[ChristophWurst]

What? Seriously? 😠

I knew not all servers support SORT, so I've added the check at

mail/lib/mailbox.php

Line 143 in 3249331

if (!$this->conn->capability->query('SORT') && (is_null($filter) || $filter === '')) {

.

The problem is that we really want to use SORT as it gives a nice performance improvement when loading the list of messages (we need it sorted and paginated). Therefore I won't simply remove that one method. Hm… let me think about a solution

[ChristophWurst]

well, we could wrap the SORT call in a try-catch block and use the slow fetching as fallback. Sounds hacky, but could work…

[mzellho]

Yep, I know - couldn't belileve what I was reading, especially that "advertising these capabilities, but rejecting the commands, will stop this silliness". Well, maybe there were some good reasons back in '08, but honestly: why would the client have to know that much detail about the server?

Regarding the fix I was also thinking of a try/catch. At least it would make the app working with medieval mailservers and that is a benefit for users of such while not causing troubles for others.

In any case: Thanks for your support! I'll try to find out more about the mailserver and its version tomorrow. Would you kindly ping me if a new version is shipped?

[ChristophWurst]

Regarding the fix I was also thinking of a try/catch.. At least it would make the app working with medieval mailservers and that is a benefit for users of such while not causing troubles for others.

Could you please try that and if that works, submit a pull request? If I'd fix it, I couldn't test it :-)

Would you kindly ping me if a new version is shipped?

Sure. You can also star/watch the repo, then you'll be notified about releases AFAIK.

You are also welcome to join us on IRC in #nextcloud-mail on freenode, help.nextcloud.com 💃

[ChristophWurst]

And there's an important event next week – the Nextcloud conf https://conf.nextcloud.com come and join us in Berlin ;-)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and questions.