koaning/scikit-fairness

this repo might get accepted

Scikit-Fairness

This project was a cool idea, but in the end, we felt it better to merge with fairlearn.

scikit-fairness's Issues

[FEATURE] EncoderFilter

The InformationFilter works but it has a downside: it can only filter away linear information. So what if we do a dimensionality reduction technique that might filter away non-linearities?

Take this neural network:

By having a network set up this way we can get two gradient signals. One will tune the encoding, the other will filter away information that has predictive power for sensitive attributes.

Documentation: Sphinx or Mkdocs

Now is the time to decide on the direction of documentation. Part of me is in favor of using mkdocs instead of sphinx.

There's a few reasons.

1. Markdown is the lingua-franka now and .rst is a bottleneck when it comes to getting folks to contribute.
2. Mkdocs has nice customisation features and we can still make an amazing API doc. The downside is that the docstrings will need to be written in a new style. See example here (expand the source code button).

make informationfilter 100% compatible

ensure it passes all tests

probably best sphinx style

https://mistune.readthedocs.io/en/latest/
https://github.com/typlog/sphinx-typlog-theme

[FEATURE] Constrained Linear Regression

Could a constrained linear regression not work?

Add unit tests for equal opportunity / p_percent_score

The coverage report is low in the metrics and that is because we are not testing the warnings or the value errors. These are relevant tests to add.

Consider fairlearn

Hey, just wanted to add awareness of fairlearn, which seems to have as contributors Adrin Jalali, who is one of scikit-learn's core contributors. I didn't take the time to check it out thoroughly though.

add arrests dataset

add boston housing dataset

old issue: koaning/scikit-lego#300

add group metric

add fairness report

Hi, I am Nick. Work for WellcomeTrust during the day and volunteer with DataKind UK at night. In both places I am somehow involved with ethics and fairness. Will be happy to contribute in one way or another to this library.

At the Wellcome we have developed a relatively simple fairness report function similar to the classification report of sklearn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html that computes instead of precision, recall, f1, true positive rate, false positive rate etc per group of interest.

The API atm is fairness_report(y_true, y_pred, groups, group_names) with groups indicating membership on a group of interest which can also be a matrix if examples can belong to multiple groups in which case group names gives the name of the columns. An additional average parameter can be added to give more control on how these cases are handled which at the moment defaults to micro. Let me know if this is an addition you are interested in.

add fairness meta estimator

add demographic parity

[ ] code added
[ ] tests pass

add extra metrics

Metrics, be they objects or closures?

A discussion worth having.

To quote @MBrouns from here.

One thing I'm still in doubt about is whether we should refactor the metrics to be classes rather than closures. Pickle doesn't do well on closures, but. on the other hand you don't often have to pickle metrics. GridSearch is the only object I can think off where we might want to pickle an object that has metrics

add markdown support to docs

and also remove all scikit-lego stuff

Collaboration on Boston Housing case study?

hello! I found out about this work from your conversation in fairlearn/fairlearn#406, so thanks for bringing it up over there 👍

I see some work on https://scikit-fairness.netlify.app/fairness_boston_housing.html, which seems great in that it makes an attempt to include some wider sociotechnical scope. It just so happens I started working on something similar this week! :)

One of my own assumptions is that no existing tools embrace the sociotechnical aspect of fairness work, and a key challenge is showing what this kind of interdisciplinary work actually looks like. I'm hoping that working in a concrete scenario could help clarify where existing fairness tools encourage practioners to avoid the sociotechnical nature of the work, and instead find some promising new directions to explore for design and research.

In research-y terms, given existing research-practioner gaps in fairness tools (eg, Holsten et al 2019; Maidado et al 2020), what kinds of tools can we explore that would move us closer to embracing the sociotechnical nature of ML fairness work (eg, Selbst et al 2019; Seo Jo and Gebru 2019)?

I've started working on this just on my own, and I picked the "Boston housing" dataset because it's widely used for ML education, deals with an extraordinarily contested sociotechnical context, and I couldn't find any educational material online that even acknowledged that. This kind of blew my mind :)

If you're interested, I'd love to chat and see if there are ways to collaborate on this!

[FEATURE] Adverserial Networks

I'm not sure what backend we'd want to use for this, but there may be something to be said to also allow for an implementation like this example.

It sounds like a fun exercise to build this in jax, but I don't know if the maintenance is going to be a pain. @MBrouns got strong opinions on this?