Address fails to parse email addresses with extended characters

[mikel]

Address fails to parse email addresses with extended characters

[nburwell]

Since we are still on Ruby 1.8.7 and could not use the proposed patch to the treetop grammar by @tessi, as a workaround, we used the stringex gem, which adds a to_ascii method to String.

> m = Mail::Address.new('"Stéphane Guillou" <[email protected]>'.to_ascii)
=> ...
> m.name
=> "Stephane Guillou"
> m.address
=> "[email protected]"

That at least gets around the parse error and will convert special characters to their closest ascii equivalent.

[beders]

Thanks for working on this. It is greatly appreciated.
I tested with 2.7.0.rc1 but it doesn't seem to be happy with umlauts:
Here's an example:

>> puts Gem.loaded_specs["mail"].version
2.7.0.rc1
nil
>> Mail::Address.new('über <[email protected]>')
Mail::Field::ParseError: Mail::AddressList can not parse |über <[email protected]>|
Reason was: Only able to parse up to ü
	from /Users/beders/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/mail-2.7.0.rc1/lib/mail/parsers/address_lists_parser.rb:15336:in `parse'
	from /Users/beders/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/mail-2.7.0.rc1/lib/mail/elements/address.rb:191:in `parse'
	from /Users/beders/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/mail-2.7.0.rc1/lib/mail/elements/address.rb:31:in `initialize'
	from (irb):9:in `new'
	from (irb):9

[kevn]

This is RFC 5335: http://tools.ietf.org/html/rfc5335 Not sure how many MTAs support it, but we do have them in our system.

[mikel]

Yeah, I think the questions is, do I just pass on the non us-ascii chars or do I have Mail try to encode the incoming text stream by guessing?

Auto encode is not trivial but I think it is the way to go.

Mikel

[mikel]

OK, so I have now released Mail 2.2.0. Can you try again and let me know? It should be fixed.

[blackwinter]

Same problem here. Not fixed by 2.2.0:

Mail::Address.new("josé[email protected]")
Mail::Field::ParseError: AddressListsParser can not parse |josé[email protected]|
Reason was: Expected one of !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, }, ~,
, (, ", ., :, <, @, , at line 1, column 4 (byte 4) after

    from /usr/lib/ruby/gems/1.8/gems/mail-2.2.0/lib/mail/elements/address_list.rb:29:in `initialize'
    from /usr/lib/ruby/gems/1.8/gems/mail-2.2.0/lib/mail/elements/address.rb:179:in `new'
    from /usr/lib/ruby/gems/1.8/gems/mail-2.2.0/lib/mail/elements/address.rb:179:in `parse'
    from /usr/lib/ruby/gems/1.8/gems/mail-2.2.0/lib/mail/elements/address.rb:32:in `initialize'
    from (irb):3:in `new'
    from (irb):3

Mail::VERSION::STRING
=> "2.2.0"

It used to work with TMail, though:

TMail::Address.parse("josé[email protected]").address
=> ""josé"[email protected]"

Unfortunately, we can't switch to Mail until non-ASCII addresses are accepted.

cheers
jens

[mikel]

This should work now. Could you please check against the latest mail gem?

[kevn]

mikel - it seems to not yet work (v2.2.4), is that the version you're referring to? Here's a gist that I used to test: http://gist.github.com/440075

I tried with ruby 1.8.7 and 1.9.2 with source encoding to see if that had anything to do with it, but I still get the same errors:

Ruby version: 1.9.2
Mail gem version: 2.2.4
Source encoding: ASCII-8BIT
String encoding: ASCII-8BIT
/Users/kevin/.rvm/gems/ruby-1.9.2-head/gems/mail-2.2.4/lib/mail/elements/address_list.rb:29:in `initialize': AddressListsParser can not parse |josé[email protected]| (Mail::Field::ParseError)
Reason was: Expected one of !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, }, ~, 
, (, ", ., :, <, @, , at line 1, column 4 (byte 4) after 
    from /Users/kevin/.rvm/gems/ruby-1.9.2-head/gems/mail-2.2.4/lib/mail/elements/address.rb:179:in `new'
    from /Users/kevin/.rvm/gems/ruby-1.9.2-head/gems/mail-2.2.4/lib/mail/elements/address.rb:179:in `parse'
    from /Users/kevin/.rvm/gems/ruby-1.9.2-head/gems/mail-2.2.4/lib/mail/elements/address.rb:32:in `initialize'
    from mail_i18n_address.rb:18:in `new'
    from mail_i18n_address.rb:18:in `'

FWIW, gmail doesn't support these addresses, so I have no idea how anyone uses addresses like these in the wild.

[mikel]

OK, we just released mail-2.2.5, please check again?

Mikel

[jjulian]

This is still an outstanding issue with mail-2.2.14

Mail::Address.new("josé[email protected]")
AddressListsParser can not parse |josé[email protected]| (Mail::Field::ParseError)
Reason was: Expected one of !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, }, ~, , (, ", ., :, <, @, , at line 1, column 4 (byte 4) after 
from /gems/mail-2.2.14/lib/mail/elements/address.rb:179:in `new'
from /gems/mail-2.2.14/lib/mail/elements/address.rb:179:in `parse'
from /gems/mail-2.2.14/lib/mail/elements/address.rb:32:in `initialize'

[jjulian]

An update - this still an issue with mail-2.3.0 on ree-1.8.7-2011.02

[kewinwang]

if my mail from with
from "tuanpin Support---团拼网[email protected]"
Mail::Message#deliver! failed with Mail::Field::ParseError: AddressListsParser can not parse |Tuanpin Support---团拼网[email protected]|
how to fix it ?

[antekpiechnik]

Hey, did anyone come up with a workaround for this issue ?

[alvarezm50]

Is this fixed?

[maccman]

Just came across this issue - any updates?

[abhishiv]

Just FYI

Even addresses like this fail

"Stéphane Guillou" <[email protected]>

This is RFC complaint I guess, but still fails.

[lunaru]

Any word on this? With TMail out of the Ruby ecosystem, Mail is the only viable email gem out there and this fails:

Mail::Address.new '"你好" <[email protected]>'

If this isn't something that Mail should be handling, is there a better way in Ruby to parse names and emails from the example email format?

[esdu]

Any update on this?

[tessi]

I'm stuck at the same error with Mail::Address.new ""Ää Öö" [email protected]" (german umlauts in the display name).

Is there any progress with this issue?

[tessi]

FYI: do not take the commits above too serious. It fixes the failure I observed in ruby 1.9.x, but I don't see a way to make it work in 1.8.7. I'm still trying to find a way around that.

I'm not sure if UTF-8 characters in the display_name are RFC complaint. Section 2.2 of RFC 2822 says that any header field (including the from header) "MUST be composed of printable US-ASCII characters". Anyways, there are UTF-8 characters in the field, so I think we should be able to parse them.

[jeremy]

Note that you should only be receiving emails with unencoded utf-8 in the headers if your SMTP server supports the SMTPUTF8 extension. If you disable that, the sender should pass along normal ASCII messages.

We'll definitely want to support email internationalization efforts, but that's a giant, separate effort. The standards have been in the works for years now. Check out RFC 6532: http://datatracker.ietf.org/doc/rfc6532/

[edisgreat]

This seems to be a problem with Rails 4, Ruby version 2.1.0:

> Mail::Address.new "\"Ää Öö\" [email protected]"
=> Mail::Field::ParseError: Mail::AddressListsParser can not parse |"Ää Öö" [email protected]|
Reason was: Expected one of 
, \, " at line 1, column 2 (byte 2) after "

[wmorgan]

If you're interested in a workaround, here's a mess of regexes that parses addresslists:

https://gist.github.com/wmorgan/cd5a17616a864fed6108

It ain't pretty (then again, neither is ragel generated code), but it handles utf-8 strings just fine.

[ckarbass]

I'm seeing this as well with Rails 4 Ruby 2.0.0, has this been reopened?

[rcaught]

Seeing this with Rails 4.1.4, Ruby 2.1.2p95 and mail (2.5.4)

[matisojka]

Seeing this as well, with Rails 3.2.19, Ruby 2.1.1 and mail 2.5.4

[gr8bit]

Mail::Field::ParseError: Mail::AddressList can not parse |test@jobblödel.de| Reason was: Only able to parse up to test@jobblö

Still persists with mail 2.6.1 and Rails 4.1.6/Ruby 2.0.0p481 it seems :/

[bf4]

@jeremy should we re-open this issue or lock the comments?

[gr8bit]

First of all, thanks for this gem (in both meanings)! If you close this issue, please link to a fix/workaround. Google actually ranks this issue very high in search results for this bug.

Cheers

[jeremy]

To address this, we'll need to implement RFC 6532 support: #39 (comment)

Please do investigate! Your implementation and pull requests are welcome ❤️

[jeremy]

Unlocked so this original issue can remain a magnet for questions, discussion, and yes, the inevitable +1s 😗

[lunaru]

Which version of mail will include this fix? I see that it's not in the latest 2.6 release yet. Is this something slated for 2.7.x?

[jeremy]

@lunaru 2.7.0! You can try with 2.7.0.rc1 or bundle against the master branch.

[lunaru]

@beders I had the same problem. It looks like it's not in the 2.7.0.rc1 branch, but rather only on master still

[beders]

@lunaru Thank you! That indeed works!

[thromera]

Bump, any fixes scheduled to head on the master branch? Thanks.