Cross-Lingual RAG for Question Answering with Confidence-Aware Hallucination Detection

Abstract

Access to the government welfare information still remains unequal in linguistically diverse countries such as India, where a significant portion of the population speaks primarily in regional languages and they are substantially underserved by existing NLP infrastructure.  Kannada, spoken by over 40 million people and recognised as a classical language, is one such case and yet no document-grounded question answering system currently exists for Indian government schemes in Kannada. This paper presents SchemeQA-Kannada, a retrieval-augmented system built around government schemes or programs. We constructed a manually verified dataset of 315 QA pairs, including 63 unanswerable questions designed to evaluate a system’s ability to abstain rather than provide hallucinated responses to user. We developed a section-boosted MuRIL retrieval pipeline over a FAISS index, incorporating a tuned section-awareness score and a three-stage scheme detection mechanism combining exact alias matching, fuzzy string similarity, and MuRIL semantic similarity. We introduce a cross-lingual hallucination detection mechanism that translates Kannada answers to English before NLI entailment verification via mDeBERTa-v3-base-xnli, and a Source-Aware Confidence Score (SACS) that weights entailment score alongside source credibility tier. Evaluation was performed on a held-out test set of 48 pairs which shows that our system reduces hallucination rate to 16.1% against baselines ranging from 31.6% to 100%, while achieving a BERT Score of 0.7468, which is considerably above most comparators. Ablation analysis identifies that cross-lingual NLI verification as the most impactful component, which is responsible for the largest reduction in hallucination relative to naive retrieval. Existing systems for Indian government scheme advisory do not address hallucination detection or source credibility in low-resource Indic languages, this work targets that gap directly for Kannada language.