Allow Zero width no-break space in float parser

[jonashaag]

Allow Zero width no-break space in float parser

[stinodego]

Why do you think this should be convertible to a float? Looks like a bad input to me.

[jonashaag]

I found this in a random CSV file that I wanted to parse. I guess it's a relatively rare but legitimate ("non-")character that's inserted by Excel/etc for number formatting

[mcrumiller]

Was it the first two bytes? FEFF is the optional byte-order mark for BE Utf-16, and it's probably ending up in your CSV due to being exported with a specified encoding and byte order. I've had plenty of CSVs that get wonky due to this mark, and I usually open them and re-encode them as UTF-8 in Notepad++ or something similar.

I don't know if polars checks the byte order mark if it exists. Maybe it should, but if so that should be its own issue. But it doesn't seem to have caused many problems for people up until now.

[jonashaag]

No, it was between two separators like this

transaction_id;amount
abcdef;-\ufeff42

[mcrumiller]

According to https://www.unicode.org/faq/utf_bom.html#bom6, that shouldn't be there at all. I'd do a replace prior to trying to do anything with that string.

[jonashaag]

Hmm ok sounds like an actual bug in my data then.

The only annoying thing that remains a potential todo is to improve displaying of the character in the error message. Although not sure how to display it since it's a "zero width" character...

[stinodego]

Yeah this doesn't sound like something Polars should fix. Closing then!