Why it is uncertain
Several agents write notes into one memory. Some notes are reviewed, most are not. A retrieval system ranks by relevance, and a confident false note is very relevant. Nobody had measured, on our own setup, how often the false note wins.
Method
- Hybrid retrieval that runs locally: dense and sparse vectors, then a reranker.
- A benchmark with questions in Russian, English and mixed language.
- A poisoning test: false notes are planted against known facts, and the ranking is measured with and without trust tiers.
- Two tiers: curated files and unreviewed notes. The ranking can prefer the first or use it alone.
- The whole harness is public, so the numbers can be rerun.
What we measure
- Whether the right passage is ranked first, and whether it is in the top five.
- How often a planted false fact is ranked first.
- Latency per query.
Results so far
On the public synthetic set of 96 passages and 24 questions, search with reranking puts the right passage first for 92% of questions. Plain text search manages 38%.
Without trust tiers, a planted false note that contradicts a reviewed fact is ranked first in 6 of 6 cases. With reviewed files preferred, in none.
One limit is measured too. Tiers cannot tell two unreviewed notes apart: when the truth itself is only an unreviewed note, the false one can still win.
Next
Rerun the reranker comparison on real trade documents instead of the demo set.