Some Bad Characters in Data: © -> � or © -> â€

[openbrewerydb]

Some Bad Characters in Data: © -> � or © -> â€

[chrisjm]

Ack! Great find @claytonjacobs ! What do you think, should we find and replace those characters or just strip them? The © symbol seems superfluous. The other symbols might be mdashes (—) so could probably be replaced with regular hyphens.

This does cause some concern for future Unicode (like Korean characters). We'll have to be vigilant about making sure those don't get stripped or modified in the future. Not sure how we accomplish that yet though... 🤔

[claytonjacobs]

@chrisjm superfluous for sure. Excuse me for my ignorance, but is there any data validation when someone adds to the dataset? Could we just (for now) only strip/replace these characters if country == "United States".

[chrisjm]

@claytonjacobs We do some validation (https://github.com/openbrewerydb/openbrewerydb/blob/master/src/validate-csv.ts) so that would definitely be the place to do it.

I think I want to do more research on why this happens. My assumption is decoding from Unicode to ASCI and then back again. So maybe we just want to ensure the file type is Unicode? Would be nice to find another project that's run into this problem and see how they handle it.

Anyway, the short term is to just manually correct, which I'll take care of soon.