vaguery/gptp2015

"Genetic programming as if you meant it: real and imaginary user experience"

MS Prep for Genetic Programming Theory & Practice 2015

In brief

"Genetic programming as if you meant it: real and imaginary user experience"

My title is a direct nod to Keith Braithwaite's exercise for software developers, "TDD as if you meant it". Like the original exercise, what I describe here for genetic programming may seem onerous and difficult to most experienced theoreticians or practitioners. I'll make the case that this approach suggests---and demonstrates---a broader systematic framework in which genetic programming is far more likely to provide value to its users than we find in a quarter-decade of exploration. Along the way, I'm forced to address deep philosophical problems in the program of genetic progamming research and practice more broadly: viz., what it is "for" and when it is "done".

gptp2015's Issues

We need better examples of (un)acceptable warrants

You provide two small examples of justifications in Section 3.1.8, but I think we could really benefit from a few more examples. Maybe those turn up in the big example that isn't there (yet), but if not it might be nice if they got added somewhere else along the way.

You need to clearly define what you mean by "warrants"

I don't think that usage is terribly common (maybe I'm wrong?), and since you use it lots you need to clearly lay out what you mean by the term. I would probably also go further and use that opportunity to say a little about why you think it matters, i.e., why you think it's important/valuable for us to be (more) thoughtful in our choices and decisions.

You also use the phrase "theoretical warrants" at least once (Sec 1.1), so you might want/need to clarify that usage as well. I presume you mean "warrant" to mean something like "justification", in which case "theoretical warrant" is something like justification based on some sort of theoretical consideration. To me, however, it reads more like "hypothetical warrant", which I'm pretty sure isn't what you want.

Clarify your use of "launched" in the context of rubrics

You talk about "launching" rubrics, but I don't think it's very clear what you actually mean by that. I think, for example, that you mean more than "evaluate a script on an input", but I'm not sure?

Do you want to "charge" for computation?

At the moment running a really big script or computing a really expensive rubric has the same "cost" to the system as doing something simple. I realize that complicates the whole setup, but do you want to have some sense of charging for computation?

I think the use of "channel" & "conduit" needs clarification

I'm guessing this idea of the code/system/artifact as a "conduit of facts" comes from Pickering, but I don't think it will be super clear to a lot of your target audience.

Say more about the idea of this being an approach to *truly* automatic programming

I really like the comment in Section two that the middle three steps in the 5-step TDD process "feel like they could be easily automated", and wonder if more can/should be said on that. What you're implying (but not ever really saying) is that if we get this right, it can be about much more than just "doing GP better". We can instead radically change the way software is developed, using the computational horsepower at our finger tips to help us find our code instead of just compile it. We then get to focus on steps 1 and 5 – defining what it is the system should do (via tests), and ensuring that it's beautiful (maintainable, re-usable, whatever).

I'm baffled by this sentence

In Section 5.3 (Against replication) you say:

Twenty years and more of skepticism even from early-adopting users shouldn't be expected.

I tried to figure out what you meant there and propose a fix, but I could never get enough of a handle on it, so I'm throwing this one back in your court 😀

Clarify constraints on operator construction

In the Operators section (3.1.2) you have a paragraph that says (among other things):

no operator can explicitly invoke a particular rubric (or vice versa)

I think needs to be expanded on/clarified. It's pretty clear that in your head this disallows some particular "kind of thing", but it's not obvious to me that that thing is. Maybe an example of what you don't want to allow would be sufficient/helpful?

Clarify that there's a cost with creating lots of sub-rubrics

When I read it I missed the crucial point (or at least its importance) that if I make a rubric that has a dozen sub-rubrics, then all those sub-rubrics get added to the list as independent rubrics. Because of the properties of lexicase this could really end up skewing the selection behavior.

Thinking about this further, I really have doubts about letting people create lots of sub-rubrics like this. It seems in direct conflict with the "one thing at a time" philosophy, and it seems to create an unnecessary trap for the unwary. I think I'd be inclined to say that if you want something like "the average of these 30 rubrics" you just have to suck it up and add the 30 rubrics one at a time first. Who knows – you might decide after the sixth that this just isn't doing what you intended, or maybe you don't need the other 24 like you thought you did?

Some bits I really loved :-)

This isn't a real issue at all and you presumably want to just close it with a "Won't fix", but after dropping a pile of suggestions in your lap, I wanted to highlight some bits/lines that I really loved:

• "Instead I'll argue … that these "surprises" and "disappointments" … should be the main focus of our theoretical and practical work. Indeed much of the value can be lost … if we approach these phenomena as something to be "cleaned up" rather than accommodated."
• "Any interesting project will resist our initial plans and expectations."
• "In response to these constant distractions, much time and attention goes towards unhelpful work guaranteeing that which cannot be guaranteed… < and on through the rest of that paragraph >"
• "Finally, if there is a choice between a familiar language and an odd one, then the odd one should be chosen, all other things being equal."
• "GP dances with us, while most other machine learning methods are exactly the "mere tools" they have been designed to be."
• "If humans creating real intelligences treated them anything like the way computer scientists insist we treat nascent artificial ones, murder chargers would be forthcoming."

The next to the last one ("GP dances with us…") is in every way spectacular, and ought to be on shirts and billboards and things. It says, with clarity and elegance and grace, something really important that I think I've always sort of "got" implicitly, but which I've never seen clearly or articulated. That line alone is worth the price of admission. (But there's tons more that's cool, as well!)