oss-fuzz 69058: TokenError

[nedbat]

oss-fuzz 69058: TokenError

[devdanzin]

It's possible to raise an EOF TokenError by decoding the bytes to cp273, but it is caught and results in a NotPython exception. Also, it's about a multi-line statement, while the original error is about a multi-line string. It only works with Python 3.11 or below, 3.12 won't raise a TokenError. Seems to be a coincidence.

from coverage.parser import PythonParser
text = b"\xFF\x8D\xA7\xDC\n''\xA7".decode("cp273")
PythonParser(text).parse_source()

  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/parser.py", line 271, in parse_source
    self._raw_parse()
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/parser.py", line 154, in _raw_parse
    tokgen = generate_tokens(self.text)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/phystokens.py", line 179, in generate_tokens
    return list(tokenize.generate_tokens(readline))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/tokenize.py", line 525, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/tok.py", line 22, in <module>
    PythonParser(text).parse_source()
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/parser.py", line 277, in parse_source
    raise NotPython(
coverage.exceptions.NotPython: Couldn't parse '<code>' as Python source: 'EOF in multi-line statement' at line 2

Here's the code I used to try to find matching errors:

from encodings.aliases import aliases
from coverage.parser import PythonParser, NotPython

btext = b"\xFF\x8D\xA7\xDC\n''\xA7"

encs = sorted(set(aliases.values()))
for enc in encs:
    try:
        text = btext.decode(enc)
    except:
        continue
    parser = PythonParser(text)
    try:
        parser.parse_source()
    except NotPython as n:
        print(enc, n)

[nedbat]

@devdanzin said on Mastodon:

Maybe the minimized test case is a seed for the fuzzer, instead of direct input?

"A minimized testcase, which is a fuzzer input that can be used to reproduce the bug."

pigweed.dev/pw_fuzzer/guides/reproducing_oss_fuzz_bugs.html

[nedbat]

This is fixed in commit 364282e

[nedbat]

This is now released as part of coverage 7.5.2.