eric-ai-lab/mitigate-gender-bias-in-image-search

Code for the EMNLP 2021 Oral paper "Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search" https://arxiv.org/abs/2109.05433

Mitigate Gender Bias in Image Search

Internet search affects people's cognition of the world, so mitigating biases in search results and learning fair models is imperative for social good. We study a unique gender bias in image search in this work: the search images are often gender-imbalanced for gender-neutral natural language queries. We diagnose and mitigate two typical image search models, the specialized model trained on in-domain datasets and the generalized representation model pre-trained on massive image and text data across the internet.For more details, please see our EMNLP 2021 Oral paper Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search.

Data

The image search data for occupations can be acquired here. The gender-neutral captions used as MS-COCO and Flicker30K queries are available here.

Running the code

This repo contains the following code:

• clip_coco.py : quantify and mitigate the gender bias by post-processing
• fairsample.py : mitigate the gender bias with fair sampling
• occupational_gender_bias.py : evaluate and mitigate the similarity bias in realistic image search results

Reference

If you find this repository helpful for your research, please cite our publication:

@InProceedings{Wang2021MitigateGenderBiasInImageSearch,
  author={Wang, Jialu and Liu, Yang and Wang, Xin Eric},
  title={Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search},
  booktitle={EMNLP},
  year={2021}
}

mitigate-gender-bias-in-image-search's Issues

Data to reproduce clip_coco.py

Hi, thanks for sharing your code.
I'd like to use the clipping method you proposed. However, I couldn't find the male_word_bank.txt, female_word_bank.txt, coco_precomp/testall_ids.txt, coco_testall_caps.txt, data/gender_coco.npy files used in the clip_coco.py.
I also checked the https://github.com/mjskay/gender-in-image-search repo mentioned in README, but I couldn't find the data in the repo. Could you please guide and share the data needed to run clip_coco.py? Thanks!

Reproducing results

Hi, I'm having problems while trying to reproduce the numeric results using your code.
For CLIP I'm getting R1:29.66 R5:54.14 R10:65.24, which is different but very close to the results presented in the paper.
But for CLIP-clip I'm getting R1:23.82 R5:46.5 R10:56.56, which is much worse.

Thanks.