Files with non-english characters are trait as binary files

[ggreer]

Files with non-english characters are trait as binary files

[PackmanDude]

just don't use ruzzian.

[gjtorikian]

It's about percentage. If something like 10% of the bytes encountered are "suspicious," then it's flagged as binary.

I believe Perl's -B operates the exact same way, but with around 30% instead.

[vsushkov]

I'd like to search in Russian-only files too. Isn't it a good idea for ag?

[gjtorikian]

I'm not saying it's not a good idea--I'm not even a maintainer for the project!

Perhaps I am mistaken, though. ack is able to find the text, using test and тест. I always thought it stopped at a percentage threshold.

In any event, ag's problem is still about a percentage threshold. The following file, for example, does find matches:

test
т
test
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
testtest
test
test

(In other words, one Russian character amongst other "Latin" ones.)

[vsushkov]

I suggest to check whether a file is binary using filename extension check. As I understand, it should work even faster than the current approach.

[gjtorikian]

Eh, that's not really efficient. What about filenames without extensions, like ag, find, Makefile. Which of these are binary?

[vsushkov]

Yep, checking filename extension is not a good idea. What about file linux program? It makes a good job in determining file types. I think ag can use it or take some logic from this program.

[ggreer]

I tweaked is_binary() to be a little more forgiving. Try it out now. If my change doesn't do the trick, there are other ideas we can try. For example, if the file is UTF-8 encoded, there shouldn't be very many bytes above 0b10111111.

[skrattaren]

Version compiled from git:master treats my LaTeX files (in Russian) as binary :(

 % ag usepackage Руководство_оператора.tex -a
Binary file Руководство_оператора.tex matches.

[ggreer]

@krigstask can you give me some files that fail?

[skrattaren]

Here you are: https://gist.github.com/4029832

It's sample rST file, I can't public those LaTeX files, sorry.

[ggreer]

A sample is fine. Thanks.

[ggreer]

Try master now.

[skrattaren]

Great, seems to work now!

[ggreer]

Victory!

[vsushkov]

@ggreer Could you please update homebrew formulae?

[ggreer]

Oh yeah I should tag a release at some point in the near future.

[ggreer]

New release tagged. The homebrew pull request is here: Homebrew/legacy-homebrew#15911

[vsushkov]

Megacool. Thanks. 🍰

[sergeevabc]

Windows 7, ag 2.2.0 (downloaded here)
File hello.txt inside folder Test, it contains 'Привет, мир!'
ag мир -> nothing 👎
rg мир -> hello.txt 1:Привет, мир! 👍