MindScope: Exploring Cognitive Biases in Large Language Models Through Multi-Agent Systems

Detecting cognitive biases in large language models (LLMs) is a fascinating task that aims to probe the existing cognitive biases within these models.Current methods for detecting cognitive biases in language models generally suffer from incomplete detection capabilities and a restricted range of detectable bias types.To address this issue, we introduced the’MindScope’ dataset, which distinctively integrates static and dynamic elements.The static component comprises 5,170 open-ended questions spanning 72 cognitive bias categories.The dynamic component leverages a rule-based, multi-agent communication framework to facilitate the generation of multi-round dialogues.This framework is flexible and readily adaptable for various psychological experiments involving LLMs.In addition, we introduce a multi-agent detection method applicable to a wide range of detection tasks, which integrates Retrieval-Augmented Generation (RAG), competitive debate, and a reinforcement learning-based decision module.Demonstrating substantial effectiveness, this method has shown to improve detection accuracy by as much as 35.10% compared to GPT-4.Codes and appendix are available at https://github.com/2279072142/MindScope.