can pysource-codegen be used to generate code with hypothesmith

[zac-hd]

can pysource-codegen be used to generate code with hypothesmith

[Zac-HD]

That's really nice work, thanks for sharing it!

I think pysource-codegen is a great complement to hypothesmith, which works somewhat differently. (I'll focus on from_node() here, since from_grammar() is older and pretty janky)

• hypothesmith uses the libcst concrete syntax tree to generate source code, by starting from a node type and recursively choosing valid subcomponents.
  • Hypothesis focusses on smaller test cases, with a semi-arbitrary limit of 8K of entropy because skipping larger inputs usually finds more bugs per minute (with more, diverse, smaller inputs). If you need 10kloc though, a new tool is probably the way to go.
  • Sometimes generating weird strings, comments, whitespace, etc. uncovers bugs too.
  • I ensure valid generation via a combination of declaring how to generate valid instances of each node type, and rejection sampling. This takes a while to implement but means there's no fixup stage, and keeps internal reduction working.
• Hypothesis' shrinking works by replaying generation, but with different choices. Here's the paper on how that works (pdf).
  • As a neat side effect, this allows us to use targeted PBT, coverage-guided fuzzing, SMT solving with crosshair, and all the other nice stuff built in to or integrated with Hypothesis.

Overall though, hypothesmith is an unfinished proof-of-concept and I'm delighted to see other useful tools being created! I think it's unlikely that we could directly merge our tools, but if you'd like to contribute to hypothesmith or try using libcst I'd be happy to help out 🙂

[15r10nk]

Thank you, your answer was very helpful.

I think both tools have their strength.
I intentional choose the python ast over libcst, because my goal is to test executing.
executing operates on the python bytecode which is mostly independent of white spaces. Logical errors are more important in this case. Having two variables naming the same thing leads to more special cases than to name everything differently (scoping of function, nonlocal/global variables).

As I told earlier, I am a big fan of hypothesis. Thank you for your great work. Maybe I will find a way to integrate pysource-codegen into the hypothesis world. I will keep it in my mind while I refactor my code.

[Zac-HD]

Sounds great!

Hypothesis has some internal logic to generate things like duplicated names (based on mutating existing test cases), but I think at a lower rate than pysource-codegen 🙂