Clarify connection between fairness optimization choices on left and data on right

[pair-code]

Clarify connection between fairness optimization choices on left and data on right

[jameswex]

Thanks @kevinrobinson , this is great. Our visual designer @mpushkarna has also come up with some mocks for better display of fairness optimization choices, but we have yet to implement them. Tagged her in this to bring this to her attention as well.

[kevinrobinson]

One other thought on this, to brainstorm a step forward. This visual from Hardt el al. (2016) keeps coming to mind as one of the main visuals I'm using to think about this across groups (versus comparing across confusion matrices or looking at differences in ROCs). It's a bit different from the idea above (see all the values that each strategy optimizes for at once), but maybe another thing to try:

I was thinking about if there was a way to include that into some part of this:

To sort of brainstorm, here's a terrible sketch of what I'm imagining as a way to maybe get at some ways to build intuition on how the choice of these optimization strategies impact different examples in the data set (the slice names don't match, but hopefully this gets the point across):

and then clicking on a point selects that slice and that strategy, and shows its confusion matrix and ROC in the side panel:

In the smiling classifier, there's not "score" so I was imagining that axis could instead either encode the continuous [0-1] "inference score." Or maybe it'd be better to build on how the what-if tool conceptualizes this now, and have that axis encode the "threshold" that each strategy would apply to each slice. That might be more domain-agnostic for binary classifiers but that's just a guess.

I realize this is more brainstorm-y than the original issue, and probably a separate panel, but since I was thinking about it I figured it was worth sharing a sketch at least of what I was thinking as I was using the performance & fairness panel, and talking about it with others trying to build intuition about how these kinds of technical conceptualizations of fairness work :)

[kevinrobinson]

@jameswex Awesome! 👍 @mpushkarna I'll be excited to check out what you're working on too, and happy to brainstorm more if that's helpful sometime too.

[mpushkarna]

+Mahima Pushkarna <[email protected]> -- Mahima Pushkarna Boston, MA 02115 *E:* [email protected] *W:* www.mahimapushkarna.com [Kindly consider the environment before printing this mail]

…

On Wed, May 15, 2019 at 8:51 AM Kevin Robinson ***@***.***> wrote: @jameswex <https://github.com/jameswex> Awesome! 👍 @mpushkarna <https://github.com/mpushkarna> I'll be excited to check out what you're working on too, and happy to brainstorm more if that's helpful sometime too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9?email_source=notifications&email_token=ACCFEKVYU5A5RZ6LP42UCY3PVQBMBA5CNFSM4HI2PZ4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVORV2A#issuecomment-492641000>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACCFEKVB2GWIRZVU6L4U573PVQBMBANCNFSM4HI2PZ4A> .

[kevinrobinson]

@mpushkarna also I just ran into this talk on fairness at IO on the twitters, and I'm wondering if this flashed screenshot is related to what you're working on? It looks like it's designed to allow comparing a single metric across more subgroup slices at once: