Bug: Content encoding is incorrect

[titanism]

Bug: Content encoding is incorrect

[titanism]

Might be an issue on my side; using TEXT column (UTF-8 default) with SQLite for storing converted mimeTree as JSON stringified.

[titanism]

Only seems to happen with a period followed by two spaces and then another character. E.g. "Foo. Yeah. Bar"

[titanism]

Seems like it's only on our side when fetching. Localized issue, not WildDuck related.

[titanism]

Re-opening for now, it appears to happen if you do this and transfer between multiple folders. For example, create a draft, send it, and then transfer it from Sent Mail to Inbox, and vice-versa, a few times. The message is stored the same, but it's rendering wrong when being FETCH'd it appears. Not sure exactly what the culprit could be.... 🤦

[titanism]

Seems like this is directly related http://forums.mozillazine.org/viewtopic.php?p=14792984#p14792984

[titanism]

@andris9 in conclusion - I feel like this is a WildDuck-specific and Thunderbird-specific issue, When I check the message, it is stored correctly, but the issue above in screenshots with unicode replacement character (?) being inserted only happens when you move a message between folder.

[titanism]

Another related issue https://bugzilla.mozilla.org/show_bug.cgi?id=1435903 (maybe related to smtp-server)

[titanism]

Okay after much research, this appears to be a smtp-server issue. I've tried setting hide8BITMIME: true but that hasn't worked yet. Just wanted to update you that this is not WildDuck specific.

[titanism]

Somewhat at a loss here as to how to fix this @andris9 - not sure if you have any ideas. If you have a Buffer stored with 8-bit encoding and then it gets passed to rebuild of WildDuck's indexer, it gets sent over FETCH with incorrect encoding (and subsequently gets rendered wrong in email clients, e.g. Thunderbird).

[titanism]

I'm wondering if it's because we're using compile vs compileStream.

    const stream = imapHandler.compileStream(data);
    session.writeStream.write(stream);

    /*
    const compiled = imapHandler.compiler(data);
    // `compiled` is a 'binary' string
    totalBytes += compiled.length;
    if (instance?.constructor?.name === 'IMAP') {
      session.writeStream.write({ compiled });
    } else {
      writeStream.push({ compiled });
    }
    */

Sorry for all the spam.

[titanism]

Do we need to store the message using iconv-lite or something instead of storing as a raw buffer? We're not doing anything out of the ordinary, just storing the stream inbound as a buffer (SMTP) and then using existing rebuild() function from WildDuck. 🤦 🤦

[titanism]

@andris9 this actually appears to be a core bug in WildDuck!!!!

    const stream = imapHandler.compileStream(data);
    // eslint-disable-next-line no-await-in-loop
    const compiled = await getStream.buffer(stream);
    totalBytes += compiled.length;
    session.writeStream.write({ compiled: compiled.toString('binary') });
    // session.writeStream.write({ compiled });

    /*
    const compiled = imapHandler.compiler(data);
    // `compiled` is a 'binary' string
    totalBytes += compiled.length;
    if (instance?.constructor?.name === 'IMAP') {
      session.writeStream.write({ compiled });
    } else {
      writeStream.push({ compiled });
    }
    */

This code works. Something in imapHandler.compiler isn't returning the right binary string. I ended up using compileStream instead of compiler and then called it with toString('binary'). I think it's by default converting to a UTF-8 string somewhere (which is Buffer.toString default). Not exactly sure where.

Another issue I found was this, which most likely needs to be changed as such:

wildduck/imap-core/lib/imap-composer.js

Line 63 in fd98244

this.push(Buffer.from(compiled + '\r\n', 'binary'));

-        this.push(Buffer.from(compiled + '\r\n', 'binary'));
+        this.push(compiled);
+        this.push('\r\n');

[titanism]

@andris9 can we please release v1.41.0 to npm? this was a core bug, thank you 🙏

[titanism]

@andris9 it appears that this is still an issue, using latest v1.41.0 and still having this same encoding issue

[andris9]

See titanism@a8fbe83#commitcomment-134660563