TOHA flags unsupported AI answers from attention maps

In the setting TOHA targets, an AI assistant receives a question and a bundle of company documents before it answers. The sentence can sound convincing and still have no support in that context. Researchers at Skoltech and Sberbank’s Center for Practical Artificial Intelligence have proposed TOHA (TOpology-based HAllucination detector), a method that flags these potential hallucinations from the model’s own attention maps.

The method is designed for retrieval-augmented generation, or RAG — systems that supply a language model with information from corporate knowledge bases, documents or other external sources. Existing detectors can demand large amounts of annotated data or extra computation, such as generating several answers and comparing them. TOHA takes a different route: it turns the model’s attention matrices into graphs and compares the structures linked to the original context and the generated response.

That comparison produces MTop-Div, a topological measure of divergence. For certain attention heads, the researchers found that higher values were consistently associated with responses not supported by the supplied context. A small set of annotated examples is used to configure the system and select the informative heads, with no separately trained classifier required.

The team evaluated TOHA on question-answering and text-summarization tasks. Across several benchmarks, the authors report results comparable to or better than existing methods while requiring relatively little annotated data and computational resources. They also say the approach reached results comparable to SelfCheckGPT, a more computationally intensive method based on generating multiple responses.

So, concretely, the method could give banks, insurers, law firms and other companies a less expensive way to screen answers produced from their internal documents. The implementation is already part of SIRIN, an open-source library developed by Sberbank’s Center for Practical Artificial Intelligence. Its current promise is targeted detection of contextual inconsistency: the researchers describe a signal of potential hallucination, not a guarantee that every response is correct.