Use tokenizer

[pycqa]

Use tokenizer

[timothycrosley]

Marking resolved as the specific test case noted as a possible failure has now been resolved, and I can't personally think of a situation where the parsing would now fail. Thanks again for your input!

I will look further into switching to using the Python tokenizer in the future - however my immediate reaction is admittedly a little weary as it seems like that could reduce the performance of parsing (as it would add an entire additional pass over the file contents) and so far it seems pretty straight forward and testable to parse out just strings manually. That said, it could be super fast and my custom parsing of strings could be further broken yet. I am often incorrect and there is no guarantee this is an exception :)

Thanks!

Timothy

[myint]

Well, I think I could come up with counter examples all day. :)

For example, this

import os

'''
\'''
import junk
'''

gets corrupted by isort into

import os

import junk

'''
\'''
'''

[timothycrosley]

This issue has been solved in release 2.4.1 Thanks!