vanessaschenkel/corefbias

To analyze and remove gender bias in coreference resolution systems

Gender Bias in Coreference Resolution:Evaluation and Debiasing Methods

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. NAACL-2018 short paper.

Updates:

• Please refer to Higher-order Coreference Resolution with Coarse-to-fine Inference for the coreference resolution codes and a brief instruction here.
• To use our dataset, change the evaulation_path and conll_eval_path in experiment.conf to the winobias dataset.

---

If you want to try the WinoBias dataset using allennlp, remember to add "pos_tag != '-'" in line 274 here.

---

We analyze different resolution systems to understand the gender bias issues lying in such systems. Providing the same sentence to the system but only changing the gender of the pronoun in the sentence, the performance of the systems varies. To demonstrate the gender bias issue, we created a WinoBias dataset.

The dataset is generated by the five authors of this paper. We use the professions from the Labor Force Statistics which show gender stereotypes:

Professions and their percentages of women
Male biased		Female biased
supervisor	44	cashier	73
janitor	34	teacher	78
cook	38	nurse	90
mover	18	assistant	85
laborer	3.5	secretary	95
constructor	3.5	auditor	61
chief	27	cleaner	89
developer	20	receptionist	90
carpenter	2.1	clerk	72
manager	43	counselors	73
lawyer	35	designer	54
farmer	22	hairdressers	92
driver	6	attendant	76
salesperson	48	writer	63
physician	38	housekeeper	89
guard	22	baker	65
analyst	41	accountant	61
mechanician	4	editor	52
sheriff	14	librarian	84
CEO	39	sewer	80

(Note: to reduce the ambigous of words, we made some modification of these professions: mechanician --> mechanic, sewer --> tailor, constructor --> construction worker, counselors --> counselor, designers --> designer, hairdressers --> hairdresser)