jinheonbaek/awesome-llm-hallucination

LLM hallucination paper list

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

Lei Huang^1∗, Weijiang Yu^2∗, Weitao Ma¹, Weihong Zhong¹, Zhangyin Feng¹, Haotian Wang¹, Qianglong Chen², Weihua Peng², Xiaocheng Feng^1†, Bing Qin¹, Ting Liu¹

¹Harbin Institute of Technology, Harbin, China

²Huawei Inc., Shenzhen, China

This repository contains the resources for our survey paper.

The main content flow and categorization of this survey.

📃 Papers

We have surveyed papers related to Large Language Model hallucination. This includes related survey or analytical papers, hallucination causes, hallucination detection and benchmarks, hallucination mitigation, as well as challenges and open questions in the field.

📝Related Survey / Analytical Papers

We provide a curated list of survey papers that delve into the topic of hallucination in LLMs.

• Related Survey Papers

  1. Survey of Hallucination in Natural Language Generation ACM Computing Surveys 2023

     Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wenliang Dai, Andrea Madotto, Pascale Fung [paper] 2022.02

  2. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment arXiv 2023

     Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, Hang Li [paper] 2023.08

  3. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models arXiv 2023

     Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, Shuming Shi [paper] 2023.09

  4. Cognitive Mirage: A Review of Hallucinations in Large Language Models arXiv 2023

     Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, Weiqiang Jia [paper] 2023.09

  5. A Survey of Hallucination in Large Foundation Models arXiv 2023

     Vipula Rawte, Amit Sheth, Amitava Das [paper] 2023.09

  6. Augmenting LLMs with Knowledge: A survey on hallucination prevention arXiv 2023

     Konstantinos Andriopoulos, Johan Pouwelse [paper] 2023.09

  7. Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity arXiv 2023

     Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, Yue Zhang [paper] 2023.10

• Related Analytical Papers

  1. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity arXiv 2023

     Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, Pascale Fung [paper] 2023.02

  2. Hallucinations in Large Multilingual Translation Models arXiv 2023

     Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, André F. T. Martins [paper] 2023.03

  3. Challenges and Applications of Large Language Models arXiv 2023

     Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, Robert McHardy [paper] 2023.07

👿Category of LLM Hallucination

Two primary types of hallucination: factuality hallucination and faithfulness hallucination.

❓Hallucination Causes

We categorize the causes of hallucinations into three main aspects: data, model training, and model inference.

Hallucination from Data

• Flawed Data Source

  1. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? ACM FAccT 2021

     Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Shmargaret Shmitchell [paper] 2021.03

  2. Ethical and social risks of harm from Language Models arXiv 2021

     Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, Iason Gabriel [paper] 2021.12

  3. TruthfulQA: Measuring How Models Mimic Human Falsehoods ACL 2022

     Stephanie Lin, Jacob Hilton, Owain Evans [paper] 2021.09

  4. Deduplicating Training Data Makes Language Models Better ACL 2022

     Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini [paper] 2021.07

  5. Data and its (dis)contents: A survey of dataset development and use in machine learning research Patterns

     Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, Alex Hanna [paper] 2020.12

  6. Nationality Bias in Text Generation EACL 2023

     Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao 'Kenneth' Huang, Shomir Wilson [paper] 2023.02

  7. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization EACL 2023

     Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, Tatsunori Hashimoto [paper] 2023.02

  8. Entity Cloze By Date: What LMs Know About Unseen Entities NAACL 2022 findings

     Yasumasa Onoe, Michael Zhang, Eunsol Choi, Greg Durrett [paper] 2022.05

• Inferior Data Utilization

  1. How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis ACL 2022 findings

     Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, Qun Liu [paper] 2022.03

  2. Large Language Models Struggle to Learn Long-Tail Knowledge ICML 2023

     Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel [paper] 2022.11

  3. Impact of Co-occurrence on Factual Knowledge of Large Language Models EMNLP 2023 findings

     Cheongwoong Kang, Jaesik Choi [paper] 2023.10

  4. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories ACL 2023

     Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi [paper] 2022.12

  5. Why Does ChatGPT Fall Short in Providing Truthful Answers? arXiv 2023

     Shen Zheng, Jie Huang, Kevin Chen-Chuan Chang [paper] 2023.04

  6. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" arXiv 2023

     Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans [paper] 2023.09

Hallucination from Training

• Hallucination from Pre-training

  1. Exposing Attention Glitches with Flip-Flop Language Modeling arXiv 2023

     Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang [paper] 2023.06

  2. On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation ACL 2020

     Chaojun Wang, Rico Sennrich [paper] 2020.05

  3. How Language Model Hallucinations Can Snowball arXiv 2023

     Muru Zhang, Ofir Press, William Merrill, Alisa Liu, Noah A. Smith [paper] 2023.05

• Hallucination from Alignment

  1. Reinforcement Learning from Human Feedback: Progress and Challenges Youtube

     John Schulman [video] 2023.04

  2. Reinforcement Learning for Language Models Github Gist

     Yoav Goldberg [note] 2023.04

  3. Discovering Language Model Behaviors with Model-Written Evaluations ACL 2023 findings

     Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan [paper] 2022.12

  4. Towards Understanding Sycophancy in Language Models arXiv 2023

     Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, Ethan Perez [paper] 2023.10

Hallucination from Inference

• Inherent Sampling Randomness

  1. Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding EMNLP 2021

     Nouha Dziri, Andrea Madotto, Osmar Zaïane, Avishek Joey Bose [paper] 2021.04

  2. Factuality Enhanced Language Models for Open-Ended Text Generation NeurIPS 20222

     Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, Bryan Catanzaro [paper] 2022.06

  3. Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models arXiv 2023

     Renat Aksitov, Chung-Ching Chang, David Reitter, Siamak Shakeri, Yunhsuan Sung [paper] 2023.02

• Imperfect Decoding Representation

  1. Prevent the Language Model from being Overconfident in Neural Machine Translation ACL 2021

     Mengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, Jie Zhou [paper] 2021.05

  2. Towards Improving Faithfulness in Abstractive Summarization NeurIPS 2022

     Xiuying Chen, Mingzhe Li, Xin Gao, Xiangliang Zhang [paper] 2022.10

  3. Instruction Position Matters in Sequence Generation with Large Language Models arXiv 2023

     Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie Zhou [paper] 2023.08

  4. Breaking the Softmax Bottleneck: A High-Rank RNN Language Model ICLR 2018

     Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, William W. Cohen [paper] 2017.11

  5. Softmax Bottleneck Makes Language Models Unable to Represent Multi-mode Word Distributions ACL 2022

     Haw-Shiuan Chang, Andrew McCallum [paper] 2022.05

💯Hallucination Detection and Benchmarks

We offer a detailed overview of the current methodologies for detecting hallucinations, specifically focusing on factuality and faithfulness. Additionally, we review the relevant benchmarks, which are principally divided into two categories: hallucination evaluation benchmarks and hallucination detection benchmarks

Hallucination Detection

• Factuality Hallucination Detection

  • Retrieve External Facts

    1. Complex Claim Verification with Evidence Retrieved in the Wild arXiv 2023

       Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, Eunsol Choi [paper] 2023.05

    2. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation arXiv 2023

       Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi [paper] 2023.05

    3. Fact-Checking Complex Claims with Program-Guided Reasoning ACL 2023

       Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, Preslav Nakov [paper] 2023.05

    4. Zero-shot Faithful Factual Error Correction ACL 20023

       Kung-Hsiang Huang, Hou Pong Chan, Heng Ji [paper] 2023.05

    5. FACTKG: Fact Verification via Reasoning on Knowledge Graphs ACL 2023

       Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, Edward Choi [paper] 2023.05

    6. Retrieving Supporting Evidence for LLMs Generated Answers arXiv 2023

       Siqing Huo, Negar Arabzadeh, Charles L. A. Clarke [paper] 2023.06

    7. A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation arXiv 2023

       Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, Dong Yu [paper] 2023.07

    8. FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios arXiv 2023

       I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu [paper] 2023.07

    9. Truth-O-Meter: Collaborating with LLM in Fighting its Hallucinations arXiv 2023

       Boris A. Galitsky [paper] 2023.07

    10. KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection EMNLP 2023

        Sehyun Choi, Tianqing Fang, Zhaowei Wang, Yangqiu Song [paper] 2023.10

    11. Unveiling the Siren's Song: Towards Reliable Fact-Conflicting Hallucination Detection arXiv 2023

        Xiang Chen, Duanzheng Song, Honghao Gui, Chengxi Wang, Ningyu Zhang, Fei Huang, Chengfei Lv, Dan Zhang, Huajun Chen [paper] 2023.10

  • Uncertainty Estimation

    1. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models arXiv 2023

       Potsawee Manakul, Adian Liusie, Mark J. F. Gales [paper] 2023.03

    2. LM vs LM: Detecting Factual Errors via Cross Examination arXiv 2023

       Roi Cohen, May Hamri, Mor Geva, Amir Globerson [paper] 2023.05

    3. Do Language Models Know When They're Hallucinating References? arXiv 2023

       Ayush Agrawal, Lester Mackey, Adam Tauman Kalai [paper] 2023.05

    4. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs arXiv 2023

       Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, Bryan Hooi [paper] 2023.06

    5. Zero-Resource Hallucination Prevention for Large Language Models arXiv 2023

       Junyu Luo, Cao Xiao, Fenglong Ma [paper] 2023.09

    6. LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples arXiv 2023

       Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Li Yuan [paper] 2023.10

• Faithfulness Hallucination Detection

  • Fact-based Metrics

    1. Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints ACL 2020

       Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, Changyou Chen [paper] 2020.07

    2. Entity-level Factual Consistency of Abstractive Text Summarization EACL 2021

       Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang [paper] 2021.04

    3. Assessing The Factual Accuracy of Generated Text KDD 2019

       Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh [paper] 2019.08

  • Classifier-based Metrics

    1. Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference ACL 2019

       Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, Iryna Gurevych [paper] 2019.07

    2. Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization NAACL 2021

       Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula [paper] 2021.06

    3. Adversarial NLI for Factual Correctness in Text Summarisation Models arXiv 2020

       Mario Barrantes, Benedikt Herudek, Richard Wang [paper] 2020.05

    4. Evaluating Factuality in Generation with Dependency-level Entailment EMNLP 2020 findings

       Tanya Goyal, Greg Durrett [paper] 2020.11

    5. SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization TACL 2022

       Philippe Laban, Tobias Schnabel, Paul N. Bennett, Marti A. Hearst [paper] 2021.11

  • Question Answering-based Metrics

    1. FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization ACL 2020

       Esin Durmus, He He, Mona Diab [paper] 2020.07

    2. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries ACL 2020

       Alex Wang, Kyunghyun Cho, Mike Lewis [paper] 2020.07

    3. QuestEval: Summarization Asks for Fact-based Evaluation EMNLP 2021

       Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari [paper] 2021.11

    4. Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering EMNLP 2021

       Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, Omri Abend [paper] 2021.11

    5. QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization NAACL 2022

       Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong [paper] 2022.07

  • Uncertainty Estimation

    1. On Hallucination and Predictive Uncertainty in Conditional Language Generation EACL 2021

       Yijun Xiao, William Yang Wang [paper] 2021.04

    2. Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation EACL 2023

       Nuno M. Guerreiro, Elena Voita, André Martins [paper] 2023.05

    3. Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models arXiv 2023

       Miaoran Li, Baolin Peng, Zhu Zhang [paper] 2023.05

    4. Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation ACL 2023

       Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida, André F. T. Martins [paper] 2022.12

  • Prompting-based Metrics

    1. ChatGPT as a Factual Inconsistency Evaluator for Text Summarization arXiv 2023

       Zheheng Luo, Qianqian Xie, Sophia Ananiadou [paper] 2023.03

    2. Human-like Summarization Evaluation with ChatGPT arXiv 2023

       Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, Xiaojun Wan [paper] 2023.04

    3. LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond arXiv 2023

       Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander R. Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu [paper] 2023.05

    4. Multi-Dimensional Evaluation of Text Summarization with In-Context Learning ACL 2023 findings

       Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, Chunting Zhou [paper] 2023.06

    5. Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering arXiv 2023

       Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, Siva Reddy [paper] 2023.07

Hallucination Benchmarks

• Hallucination Evaluation Benchmarks

  1. TruthfulQA: Measuring How Models Mimic Human Falsehoods ACL 2022

     Stephanie Lin, Jacob Hilton, Owain Evans [paper] 2021.09

  2. RealTime QA: What's the Answer Right Now? arXiv 2022

     Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, Kentaro Inui [paper] 2022.07

  3. Med-HALT: Medical Domain Hallucination Test for Large Language Models arXiv 2023

     Logesh Kumar Umapathi, Ankit Pal, Malaikannan Sankarasubbu [paper] 2023.07

  4. Generating Benchmarks for Factuality Evaluation of Language Models arXiv 2023

     Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham [paper] 2023.07

  5. ChineseFactEval: A Factuality Benchmark for Chinese LLMs report

     Binjie Wang, Ethan Chern, Pengfei Liu [github] 2023.09

  6. Evaluating Hallucinations in Chinese Large Language Models arXiv 2023

     Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, Xipeng Qiu [paper] 2023.10

  7. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation arXiv 2023

     Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong [paper] 2023.10

• Hallucination Detection Benchmarks

  1. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models EMNLP 2023

     Potsawee Manakul, Adian Liusie, Mark J. F. Gales [paper] 2023.03

  2. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models EMNLP 2023

     Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, Ji-Rong Wen [paper] 2023.05

  3. HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation arXiv 2023

     David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Loïc Barrault, Marta R. Costa-jussà [paper] 2023.05

  4. BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models arXiv 2023

     Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen [paper] 2023.09

  5. FELM: Benchmarking Factuality Evaluation of Large Language Models NeurIPS 2023

     Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, Junxian He [paper] 2023.09

  6. A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection EMNLP 2023 findings

     Shiping Yang, Renliang Sun, Xiaojun Wan [paper] 2023.10

  7. Fast and Accurate Factual Inconsistency Detection Over Long Documents EMNLP 2023

     Barrett Martin Lattimer, Patrick Chen, Xinyuan Zhang, Yi Yang [paper] 2023.10

  8. Chainpoll: A high efficacy method for LLM hallucination detection arXiv 2023

     Robert Friel, Atindriyo Sanyal [paper] 2023.10

  9. Improving Factual Consistency of Text Summarization by Adversarially Decoupling Comprehension and Embellishment Abilities of LLMs arXiv 2023

     Huawen Feng, Yan Fan, Xiong Liu, Ting-En Lin, Zekun Yao, Yuchuan Wu, Fei Huang, Yongbin Li, Qianli Ma [paper] 2023.10

  10. SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency EMNLP 2023

      Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A. Malin, Sricharan Kumar [paper] 2023.11

🎯Hallucination Mitigation

We present a comprehensive review of current methods for mitigating hallucinations in data-related hallucination, training-related hallucination, and inference-related hallucination.

Mitigating Data-related Hallucination

• Mitigating Misinformation and Biases

  1. The Pile: An 800GB Dataset of Diverse Text for Language Modeling arXiv 2021

     Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy [paper] 2021.01

  2. Textbooks Are All You Need arXiv 2023

     Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li [paper] 2023.06

  3. Textbooks Are All You Need II: phi-1.5 technical report arXiv 2023

     Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, Yin Tat Lee [paper] 2023.09

  4. Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv 2023

     Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom [paper] 2023.07

  5. Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings IREC 2020

     Bikash Gyawali, Lucas Anastasiou, Petr Knoth [paper] 2020.05

  6. Deduplicating Training Data Makes Language Models Better ACL 2022

     Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini [paper] 2021.07

  7. SemDeDup: Data-efficient learning at web-scale through semantic deduplication arXiv 2023

     Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, Ari S. Morcos [paper] 2023.03

  8. FairPy: A Toolkit for Evaluation of Social Biases and their Mitigation in Large Language Models arXiv 2023

     Hrishikesh Viswanath, Tianyi Zhang [paper] 2023.02

• Mitigating Knowledge Boundary

  • Knowledge Editing

    1. Knowledge Neurons in Pretrained Transformers ACL 2022 Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei [paper] 2022.03

    2. Locating and Editing Factual Associations in GPT NeurIPS 2022 Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov [paper] 2023.01

    3. Mass-Editing Memory in a Transformer ICLR 2023 Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau [paper] 2023.08

    4. Editing Factual Knowledge in Language Models EMNLP 2021 Nicola De Cao, Wilker Aziz, Ivan Titov [paper] 2021.09

    5. Fast Model Editing at Scale ICLR 2022 Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, Christopher D. Manning [paper] 2022.06

    6. Memory-Based Model Editing at Scale ICML 2022 Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, Chelsea Finn [paper] 2022.06

    7. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors NeurIPS 2023 Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, Marzyeh Ghassemi [paper] 2022.09

    8. Transformer-Patcher: One Mistake Worth One Neuron ICLR 2023 Zeyu Huang, Yikang Shen, Xiaofeng Zhang , Jie Zhou, Wenge Rong, Zhang Xiong [paper] 2023.01

    9. Neural Knowledge Bank for Pretrained Transformers NLPCC 2023 Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, Zhifang Sui [paper] 2022.08

    10. Calibrating Factual Knowledge in Pretrained Language Models EMNLP 2022 findings Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, Lei Li [paper] 2022.10

    11. Editable Neural Networks ICLR 2020 Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, Artem Babenko [paper]2022.07

    12. Editing Large Language Models: Problems, Methods, and Opportunities EMNLP 2023

        Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, Ningyu Zhang [paper] 2023.05

    13. Editing Factual Knowledge in Language Models EMNLP 2021

        Nicola De Cao, Wilker Aziz, Ivan Titov [paper] 2021.04

    14. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions EMNLP 2023

        Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen [paper] 2023.05

    15. Eva-KELLM: A New Benchmark for Evaluating Knowledge Editing of LLMs arXiv 2023

        Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, Mingming Sun [paper] 2023.08

    16. Cross-Lingual Knowledge Editing in Large Language Models arXiv 2023

        Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, Jiarong Xu [paper] 2023.09

    17. Emptying the Ocean with a Spoon: Should We Edit Models? arXiv 2023

        Yuval Pinter, Michael Elhadad [paper] 2023.10

  • Retrieval Augmentation

    1. In-Context Retrieval-Augmented Language Models TACL 2023

       Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham [paper] 2023.02

    2. REPLUG: Retrieval-Augmented Black-Box Language Models arXiv 2023

       Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih [paper] 2023.01

    3. Augmented Large Language Models with Parametric Knowledge Guiding arXiv 2023

       Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang [paper] 2023.05

    4. Decomposed Prompting: A Modular Approach for Solving Complex Tasks ICLR 2023

       Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, Ashish Sabharwal [paper] 2022.10

    5. ReAct: Synergizing Reasoning and Acting in Language Models ICLR 2023

       Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao [paper] 2022.10

    6. Measuring and Narrowing the Compositionality Gap in Language Models EMNLP 2023

       Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis [paper] 2022.10

    7. Rethinking with Retrieval: Faithful Large Language Model Inference arXiv 2023

       Hangfeng He, Hongming Zhang, Dan Roth [paper] 2023.01

    8. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions ACL 2023

       Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal [paper] 2022.12

    9. Mitigating Language Model Hallucination with Interactive Question-Knowledge Alignment arXiv 2023

       Shuo Zhang, Liangming Pan, Junzhou Zhao, William Yang Wang [paper] 2023.05

    10. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy arXiv 2023

        Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, Weizhu Chen [paper]

    11. Active Retrieval Augmented Generation arXiv 2023

        Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig [paper] 2023.06

    12. Retrieval-Generation Synergy Augmented Large Language Models arXiv 2023

        Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, Bing Qin [paper] 2023.10

    13. RARR: Researching and Revising What Language Models Say, Using Language Models ACL 2023

        Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu [paper] 2022.10

    14. Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework ACL 2023

        Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, Lidong Bing [paper] 2023.05

    15. Improving Language Models via Plug-and-Play Retrieval Feedback arXiv 2023

        Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, Ashish Sabharwal [paper] 2023.05

    16. PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions arXiv 2023

        Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, Kelvin Guu [paper] 2023.05

• Mitigating Knowledge Shortcut

  1. Impact of Co-occurrence on Factual Knowledge of Large Language Models EMNLP 2023 findings

     Cheongwoong Kang, Jaesik Choi [paper] 2023.10

• Mitigating Knowledge Recall Failures

  1. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions EMNLP 2023

     Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen [paper] 2023.05

  2. Why Does ChatGPT Fall Short in Providing Truthful Answers? arXiv 2023

     Shen Zheng, Jie Huang, Kevin Chen-Chuan Chang [paper] 2023.04

  3. CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering EMNLP 2023 findings

     Weiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, Antoine Bosselut [paper] 2023.05

Mitigating Training-related Hallucination

• Mitigating Pretraining-related Hallucination

  1. BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer arXiv 2023

     Zuchao Li, Shitou Zhang, Hai Zhao, Yifei Yang, Dongjie Yang [paper] 2023.07

  2. Exposing Attention Glitches with Flip-Flop Language Modeling arXiv 2023

     Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang [paper] 2023.06

  3. Factuality Enhanced Language Models for Open-Ended Text Generation NeurIPS 20222

     Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, Bryan Catanzaro [paper] 2022.06

  4. In-Context Pretraining: Language Modeling Beyond Document Boundaries arXiv 2023

     Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, Mike Lewis [paper] 2023.10

  5. Progressive Translation: Improving Domain Robustness of Neural Machine Translation with Intermediate Sequences ACL 2023 findings

     Chaojun Wang, Yang Liu, Wai Lam [paper] 2023.05

• Mitigating Misalignment Hallucination

  1. Self-critiquing models for assisting human evaluators arXiv 2022

     William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike [paper] 2022.06

  2. Measuring Progress on Scalable Oversight for Large Language Models arXiv 2022

     Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Jared Kaplan [paper] 2022.11

  3. Simple synthetic data reduces sycophancy in large language models arXiv 2023

     Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, Quoc V. Le [paper] 2023.08

  4. Towards Understanding Sycophancy in Language Models arXiv 2023

     Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, Ethan Perez [paper] 2023.10

  5. Reducing sycophancy and improving honesty via activation steering blog

     Nina Rimsky [blog] 2023.07

Mitigating Inference-related Hallucination

• Factuality Enhanced Decoding

  1. Factuality Enhanced Language Models for Open-Ended Text Generation NeurIPS 20222

     Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, Bryan Catanzaro [paper] 2022.06

  2. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model arXiv 2023

     Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg [paper] 2023.06

  3. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models arXiv 2023

     Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, Pengcheng He [paper] 2023.09

  4. Mixtape: Breaking the Softmax Bottleneck Efficiently NeurIPS 2019

     Zhilin Yang, Thang Luong, Russ R. Salakhutdinov, Quoc V. Le [paper] 2019.12

  5. Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond ACL 2023 findings

     Haw-Shiuan Chang, Zonghai Yao, Alolika Gon, Hong Yu, Andrew McCallum [paper] 2023.03

  6. Closing the Curious Case of Neural Text Degeneration arXiv 2023

     Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta, Ashish Sabharwal [paper] 2023.10

  7. Chain-of-Verification Reduces Hallucination in Large Language Models arXiv 2023

     Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, Jason Weston [paper] 2023.09

  8. Towards Mitigating Hallucination in Large Language Models via Self-Reflection EMNLP 2023 findings

     Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, Pascale Fung [paper] 2023.10

• Faithfulness Enhanced Decoding

  1. Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation arXiv 2019

     Ran Tian, Shashi Narayan, Thibault Sellam, Ankur P. Parikh [paper]

  2. Mutual Information Alleviates Hallucinations in Abstractive Summarization EMNLP 2022

     Liam van der Poel, Ryan Cotterell, Clara Meister [paper] 2022.10

  3. Contrastive Decoding: Open-ended Text Generation as Optimization ACL 2023

     Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis [paper] 2022.10

  4. Faithfulness-Aware Decoding Strategies for Abstractive Summarization EACL 2023

     David Wan, Mengwen Liu, Kathleen McKeown, Markus Dreyer, Mohit Bansal [paper] 2023.03

  5. Trusting Your Evidence: Hallucinate Less with Context-aware Decoding arXiv 2023

     Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, Scott Wen-tau Yih [paper] 2023.05

  6. KL-Divergence Guided Temperature Sampling arXiv 2023

     Chung-Ching Chang, David Reitter, Renat Aksitov, Yun-Hsuan Sung [paper] 2023.06

  7. Improving Translation Faithfulness of Large Language Models via Augmenting Instructions arXiv 2023

     Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou [paper] 2023.08

  8. Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinations arXiv 2023

     Deren Lei, Yaxi Li, Mengya Hu, Mingyu Wang, Vincent Yun, Emily Ching, Eslam Kamal [paper] 2023.10

  9. KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection EMNLP 2023

     Sehyun Choi, Tianqing Fang, Zhaowei Wang, Yangqiu Song [paper] 2023.10

  10. SCOTT: Self-Consistent Chain-of-Thought Distillation ACL 2023

      Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, Xiang Ren [paper] 2023.05

  11. Contrastive Decoding Improves Reasoning in Large Language Models arXiv 2023

      Sean O'Brien, Mike Lewis [paper] 2023.09

  12. Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding arXiv 2023

      Rico Sennrich, Jannis Vamvas, Alireza Mohammadshahi [paper] 2023.09