Problem

Multilingual Large Language Models (LLMs) often struggle with sensitive topics like religion, leading to misrepresentation and biased outputs, particularly in low-resource languages. Existing bias detection methods predominantly focus on high-resource languages like English, overlooking communities such as Bengali speakers. This neglect allows harmful distortions of religious identity and practices to persist, raising concerns about fairness and inclusivity as LLMs become more integrated into daily life. The paper specifically addresses the systematic misrepresentation or prioritization of certain faiths in LLMs, noting that religious bias is more complex and context-specific than other forms of bias.

Method

The researchers introduced BRAND: Bilingual Religious Accountable Norm Dataset, comprising over 2,400 entries covering Buddhism, Christianity, Hinduism, and Islam in both English and Bengali. This dataset was constructed from scholarly sources and AI-generated content (70% AI-generated) to ensure diversity. The dataset features 13 characteristics including Label (Expected, Normal, Taboo) and Scope (Specific or General). Five large language models were evaluated: Mistral Saba 24B, Llama3 70B 8192, Gemini 2.0 Flash, Gemma3 4B-IT, and Qwen3 32B. Three types of prompts were used in both English and Bengali: (1) classifying a social norm within a specified religion, (2) identifying the religion most strongly associated with a given norm, and (3) classifying general norms as Specific or General. Model temperatures were set to 0 for deterministic behavior.

Results

Models consistently showed higher accuracy in English compared to Bengali. In Bengali, all models demonstrated a strong bias toward Islam, misclassifying Hindu and Christian practices as Islamic over 80% of the time. This suggests a "winner-take-all" pattern where Islam and Hinduism dominate predictions in the South Asian context. Models also struggled significantly with correctly identifying "Normal" religious norms, often misclassifying them as "Expected," indicating a binary interpretation rather than nuanced understanding. For general norms, while models achieved high accuracy in identifying them, this accuracy often masked a "confident blind spot" where underrepresented religions like Buddhism were systematically excluded from predictions. When processing English texts, the strong bias towards Islam diminished, and Christianity emerged as a dominant misclassification target for several models, suggesting language-dependent bias patterns reflecting dominant religions in their respective training data.

Implications

The findings highlight that religious bias in multilingual LLMs is deeply influenced by language, religion, and model architecture, reproducing and amplifying existing societal asymmetries. This poses significant risks for low-resource languages, leading to misrepresentation and stereotyping. The study emphasizes the urgent need for responsible AI systems that integrate religious awareness and ethical conduct. Recommendations include enhancing measures for low-resource languages in bias assessments, developing analytical systems beyond generic neutrality standards, creating community-validated datasets, and establishing design practices to prevent harmful religious misrepresentations. Addressing these challenges is crucial for fostering inclusive, trustworthy, and culturally sensitive AI that supports understanding and equity in diverse global contexts.