pudb has no test suite

[inducer]

pudb has no test suite

[inducer]

I really don't know how to test a debugger programmatically, and I'm not sure I'm itching to find out. Just file your bugs, I'll take a look.

[bukzor]

Would you mind if i tried to submit a test suite? Do you mind a test-depends of py.test ?

[inducer]

I like py.test, and if you manage to cook up a few meaningful tests, I'd be happy to take them.

[Julian]

I'd +1 this certainly.

@bukzor if you're working on this, you can certainly ping me if you'd like some help and I have some free time, it'd be something I'd support too.

[bukzor]

I've been a bit shy about getting started since the core dev seems to think it's hard.
Could you suggest a good starting point?
Maybe testing for a bug in a recent ticket would be it.

[inducer]

Don't let me discourage you! :)

[asmeurer]

The question is, how do we test this? If someone can setup a framework to do it, or at least give some examples to show how it can be done, I'd be glad to help out with the effort. There are some good code coverage tools that can be used to make sure that everything is tested.

From what I see, there are two things that need to be tested. The first is the debugger, which will be tricky to test because it messes with the Python that it is running in (perhaps the only way is to run it in a subprocess). The second issue is the UI. Does urwid have a standard framework for unit tests?

[Julian]

Well. Truthfully I don't have much experience writing suites for already existing code. However, as for the last two points, I think the primary concern should first be getting a few unit tests. For that we don't need to worry about anything other than pudb itself.

urwid doesn't have anything as far as I know, but we shouldn't need much. urwid works (I think :), and has its own test suite. At most, if we're gonna write some end-to-end tests, or testing the UI in something outside a unit test, we'd be dealing with just simulating some keypresses and probably checking the render()ed canvas on some widgets, which isn't too hairy.

I can set something real basic up with a few sample (real) tests pretty soon if that would help us all.

[asmeurer]

Right, I know we don't have to test urwid itself. We also don't have to test bdb. But we should test things like, "pressing Control-p brings up the preferences window", or "changing the order of the stack frames in the preferences changes the order visually, but doesn't change the behavior of u or d" (these are examples of things that I've broken or almost broken myself when hacking on pudb).

I don't think we need to test render(). There ought to be a way to create the UI objects without rendering them, and to make assertions on those objects (and leave the correctness of render to urwid itself).

[bukzor]

I've dug into this a bit, and have had some success.

Script: https://gist.github.com/4504037
Resulting output (html): http://jsfiddle.net/GAAyY/embedded/result/

While having the output is nice, I plan to do more semantic assertions on the ui widget objects themselves.

This should allow me to fairly directly assert things like "pressing Control-p brings up the preferences window".

[inducer]

Cool, I like that testing strategy a lot.

[bukzor]

inducer:

This almost works, but the last line is totally fake.
Could you help me fill in the blank?

https://gist.github.com/4504742

[inducer]

I feel like you'll have to shove your testing into a callback that gets invoked from Debugger.event_loop(). In particular, dialog boxes like the preferences use a recursive event loop, and as soon as you exit from that, the dialog is closed. Perhaps you could include an attribute Debugger.event_loop_callbacks. Then exactly one of these would be consumed and called per trip through the event loop instead of getting events from urwid if that list is non-empty. From that point, you could spoon-feed the debugger keys and examine the state.

To check whether the preferences window is open, examine the toplevel argument (that's the top-level widget). It should consist of what Debugger.dialog() builds, i.e. an Overlay with a LineBox inside, with another Pile nested inside, the first entry of which should be a Text with the dialog title.

HTH,
Andreas

[asmeurer]

So maybe we should use mock, or some similar library.

[bukzor]

@inducer If it's not too much to ask, would you download and run the above script?

https://gist.github.com/bukzor/4504742

I don't believe the dialog is closed at the point of the assertion, since the printed screen looks like it has an open dialog. The missing piece of the puzzle is the introspection of toplevel/Overlay/LineBox/Pile/Text.

[asmeurer]

Aside from the callback thing, urwid doesn't really seem to have any good way to say "this object is the one you're looking for". All of the classes are defined completely by their contents, and where they live with respect to one another, both of which are implementation details that should not be tested.

It would be nice if you could attach a name to each class, so that you could just write something like assert dialog_box.name == "Preferences", that way you know for sure you're looking at the particular UI element you want and not another one of the same type (the dialog box is a bad example because it does have something like a name, the title, but you see how this applies to all the other UI elements). Using this everywhere would also make the code easier to understand.