Radon struggles to collect statistics from large files

[Sam152]

Radon struggles to collect statistics from large files

[rubik]

Hi Sam, thanks for sharing the example. Indeed, it's quite surprising to see such a long run time for such a simple file.

The raw command is definitely the slowest, and that's because it does not use the ast module to parse the file, instead it uses tokenize. The latter is written in pure Python instead of C, so that's already a slowing factor. Moreover, when parsing the AST we can use efficient techniques like the visitor pattern, which are not available with the tokenize module.

However, the superlinear complexity is definitely in Radon's code. It performs some complicated operations to count logical lines, and I suspect that's where the slowest code is. I think your example highlights one of the inefficiencies particularly well.

The next step would be to profile the code. A flamegraph should already give some very useful hints. I'll try to investigate this when I've got time.

[Sam152]

Thanks for the info, that's really helpful!