add support for alternate partitioners

[coyotemarin]

add support for alternate partitioners

[coyotemarin]

It would probably also make sense to have a sort_values option that basically tells mrjob to pick any partitioner that sorts the values passed to a given reducer.

[irskep]

Why is this closed? I don't see any new code for it...

[jretz]

Uh, I didn't mean to close it - I just read it and didn't think I clicked anything but I must have.

[coyotemarin]

MRJobs should also have a SORT_VALUES attribute; if this is True, we pick a partitioner that sorts the values passed to any given reducer (there's a different one you should use depending on the Hadoop version). Probably runners should have a matching sort_values option that can't be defaulted from mrjob.conf.

[coyotemarin]

So, after doing some hand-testing, it isn't so simple as just setting the partitioner class.

Essentially what you have to do is get Hadoop to believe the entire line is the key (so it will sort by it), and then partition in such a way that all lines containing the same actual key go to the same reducer.

Now that I know this, this section of the Hadoop Streaming docs makes more sense.

[coyotemarin]

Breaking out sorting by value into a separate ticket: #240.