Possibility of ignoring certain control chars in txt reader

[mmalecot]

Possibility of ignoring certain control chars in txt reader

[mmalecot]

Hi,

Thanks for your feedback!

It's quite complex to differentiate between text and binary because there is no infallible solution.

The approach used by file-format is fairly restrictive: text is only considered to be anything that is not a control character (excluding whitespaces). I could exclude other control characters as requested, but the problem would only be half solved.

Perhaps we should go for a simpler, but also more permissive, heuristic. A solution used by many programs is to simply check for NULL bytes in the stream. With such an approach, it should be OK for your files. I'm going to analyse how other popular tools such as grep, strings and diff work to decide.

A small remark on your match code: I see the replacement character and not BEL and ESC, maybe you should use '\x07' and '\x1b' in your code instead.

[SomeRandoNamedDave]

You're right about this solution being more of a hack, changing my code would be the better option in this instance.

Using the ascii chars was a habit I picked up from writing portable shell scripts, some older / more limited shells (ash, used by busybox or dash for example) don't interpret \e \x1b etc.

this isn't an issue with rust and most other languages plus the shell scripts are identified by the shebang anyway

Thanks for the response!

[mmalecot]

For the moment I prefer to keep the current heuristic because it seems to be more relevant and reliable.

To be seen later.

Thanks for your time!