CVIT, International Institute of Information Technology, Hyderabad, India
Read the paper Get the code Cite
ਪੰਜਾਬ ਵਿਧਾਨ ਸਭਾ
Punjabi, Gurmukhi scriptपंजाब विधान सभा
Hindi, Devanagari scriptPunjab Vidhan Sabha
English, Latin scriptDecades of Indian parliamentary records survive only as degraded, multi-column scans that mix Hindi, Punjabi and English. On these pages, OCR failure is the main obstacle to finding information.
We present an end-to-end framework that pairs vision-based layout segmentation and multi-script OCR with an agentic retrieval-augmented generation (RAG) pipeline. An iterative loop checks the retrieved evidence, rewrites the query when it falls short, and searches again before answering. On the 200,000-page Punjab Vidhan Sabha archive, the system outperforms standard text-only baselines.
Questions are embedded in their own language, so a Punjabi question can find an English page. Answers come back in the language of the question.
A fine-tuned YOLO layout model splits each scan into regions and a CRNN recognizer reads them.
BAAI/bge-m3 embeds the text and metadata into a ChromaDB vector store.
The top-5 chunks come back by cosine similarity. An LLM judges whether they are enough.
If not, the query is rephrased around terms that OCR errors may have hidden, then retrieval runs again.
A generation LLM answers in the user’s language, grounded in the validated pages.
On 1,000 stratified pages and 200 gold questions, indexed against the full archive, the optimized pipeline leads on every retrieval metric.
| Model / pipeline | MRR@1 | Hit@3 | Hit@5 | Prec@3 |
|---|---|---|---|---|
| LaBSE | 0.320 | 0.510 | 0.590 | 0.170 |
| NetraEmbed (dim 2560) | 0.605 | 0.843 | 0.912 | 0.281 |
| Multilingual-E5-Large | 0.620 | 0.855 | 0.915 | 0.285 |
| Standard BGE-M3 | 0.775 | 0.900 | 0.940 | 0.300 |
| Ours (optimized pipeline) | 0.810 | 0.925 | 0.955 | 0.325 |
Selected baselines shown. The paper compares more embedding models.
What each part of the system contributes, measured on the same 200-query benchmark and the 20 hand-transcribed pages.
L is the maximum number of retrieve–rewrite rounds. L = 1 is single-pass retrieval with no query rewriting. Answer F1 nearly doubles, so the loop mostly recovers context that one pass misses entirely.
| Max loops (L) | MRR@1 | Precision@3 | Answer F1 |
|---|---|---|---|
| 1 (no rewriting) | 0.89 | 0.33 | 0.18 |
| 2 | 0.93 | 0.35 | 0.26 |
| 3 | 0.97 | 0.37 | 0.35 |
Values read from the paper’s Fig. 6 and rounded. The text states only the L = 1 and L = 3 values.
With the same CRNN recognizer, the fine-tuned YOLO layout model lifts word accuracy from 60.35% to 80.03% over CRAFT. Larger OCR systems score higher, but our pipeline stays fully local.
| Layout model | OCR model | CRR (%) | WRR (%) |
|---|---|---|---|
| Gemini-3.1 | Gemini-3.1 | 99.7 | 89.6 |
| Surya2 | Surya2 | 99.6 | 88.9 |
| Sarvam | Sarvam | 99.5 | 89.4 |
| YOLO (fine-tuned) | CRNN (ours) | 95.67 | 80.03 |
| CRAFT | CRNN (ours) | 89.00 | 60.35 |
Swapping in stronger OCR changes retrieval only slightly: MRR@1 stays between 0.810 and 0.825 across all four systems.
| OCR model | MRR@1 | Hit@3 | Hit@5 | Prec@3 |
|---|---|---|---|---|
| Gemini-3.1 | 0.825 | 0.920 | 0.960 | 0.3533 |
| Sarvam | 0.820 | 0.935 | 0.955 | 0.325 |
| Surya2 | 0.815 | 0.925 | 0.950 | 0.305 |
| Ours | 0.810 | 0.925 | 0.955 | 0.325 |
Average perplexity of three instruction-tuned backbones on the multi-script archive documents. Lower is better.
| Language model | Average perplexity |
|---|---|
| Mistral-7B | 7.81 |
| Phi-3 | 4.99 |
| Qwen2.5 | 4.46 |
@misc{deshpande2026readingocrnoiserobust,
title={Reading Beyond OCR Noise: Robust Cross-Lingual RAG for Legislative Archives},
author={Radha Krishna Deshpande and Ajoy Mondal and Akash Manna and C V Jawahar},
year={2026},
eprint={2610.35240},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.35240},
}