nitkannen/multimodal-cultural-fairness

A collection of papers focussing on cultural fairness, inclusivity and diversity in multimodal models

Multimodal Cultural Fairness

A collection of papers focussing on cultural fairness, inclusivity and diversity in multimodal models

Culture in Text-to-Vision models

• Challenges and Strategies in cross–cultural NLP: https://aclanthology.org/2022.acl-long.482.pdf

• Cultural and Linguistic Diversity Improves Visual Representations

• Inspecting the Geographical Representativeness of Images from Text-to-Image Models

• Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models

• Auditing and Mitigating Cultural Bias in LLMs

• Cultural Concept Adaptation on Multimodal Reasoning

• Social Biases through the Text-to-Image Generation Lens

• Variation of Gender Biases in Visual Recognition Models Before and After Finetuning

• AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia

• ITI-GEN: Inclusive Text-to-Image Generation [ICCY 2023] [Google, CMU]

• Beyond the Surface: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation [Google]

• Holistic Evaluation of Text-to-Image Models [T2I evaluation]

• Stanceosaurus: Classifying Stance Towards Multicultural Misinformation [EMNLP 2022]

• HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models

• CultureLLM: Incorporating Cultural Differences into Large Language Models

• Investigating Cultural Alignment of Large Language Models

• Large Language Models as Superpositions of Cultural Perspectives

• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models [Georgia Tech] (text)

• Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing

• CIC: A framework for Culturally-aware Image Captioning (very good paper)

• Visually Grounded Reasoning across Languages and Cultures (MILA)

• Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning

• ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding

• Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Dataset [Same authors as Image captioning] [data for t2I]

• SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation

• Easily accessible text-to-image generation amplifies demographic stereotypes at large scale

• Diversity is not a one-way street: Pilot study on ethical interventions for racial bias in text-to-image systems

• Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density

• DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity

• Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. NEURIPS 2023

• Inspecting the Geographical Representativeness of Images from Text-to-Image Models

• Culture-Gen: Revealing Global Cultural Perception in Language Models through Natural Language Prompting (Yejin's Group)

• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

• CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies (Diyi Yang)

• NORMAD: A Benchmark for Measuring the Cultural Adaptability of Large Language Models (CMU)

Culture Grounding

• A Computational Approach to Identifying Cultural Keywords Across Languages
• DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures
• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models
• Extracting Cultural Commonsense Knowledge at Scale (WWW 23)

Survey Paper

• On the Cultural Gap in Text-to-Image Generation: https://arxiv.org/pdf/2307.02971.pdf
• Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation

Diversification

• Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generation
• Diversify, Don’t Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images
• Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting [EMNLP 2023]
• Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images [Google Research]

Diversity Evaluation

• Rarity Score paper
• Evaluating the Evaluation of Diversity in Natural Language Generation
• The Vendi Score: A Diversity Evaluation Metric for Machine Learning
• An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric
• DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity
• Measuring Diversity in Co-creative Image Generation

Fairness

• Social Bias Probing: Fairness Benchmarking for Language Models
• Social Biases through the Text-to-Image Generation Lens
• FACET - New benchmark by Meta to evaluate fairness of vision models
• Finetuning Text-To-Image Diffusion Models for Fairness [ICLR 2024]
• Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness
• Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
• Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models [EMNLP 2023]
• Universal Prompt Optimizer for Safe Text-to-Image Generation
• Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models [CVPR 2023]
• The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test [UCLA]
• A Unified Framework and Dataset for Assessing Gender Bias in Vision-Language Models [Microsoft]
• Reliable Fidelity and Diversity Metrics for Generative Models
• RANDOM NETWORK DISTILLATION AS A DIVERSITY METRIC FOR BOTH IMAGE AND TEXT GENERATION

T2I models

• Stable Diffusion
• Imagen 2, 3, 3.5
• DallE-3 (with GPT-4)
• Miro
• Parti
• Muse