Best Overall
0.902
Gemini-emb-2 + cohere-4-pro
PAID/PAID
PAID/PAID
Best Free/Free
0.846
Qwen3-8B + raw
FREE/FREE
FREE/FREE
Best Free/Paid
0.837
Octen-8B + cohere-4-fast
FREE/PAID
FREE/PAID
Best Paid/Free
0.892
Gemini-emb-2 + zerank-2
PAID/FREE
PAID/FREE
Best Paid/Paid
0.902
Gemini-emb-2 + cohere-4-pro
PAID/PAID
PAID/PAID
Best 3-Hop
93%
Octen-8B + cohere-4-fast
FREE/PAID
FREE/PAID
What we are measuring
In finance RAG, the hard part is not generating a fluent answer — it is retrieving the exact filing evidence that supports the answer. This page measures retrieval quality before generation. Each row is an embedding model plus an optional reranker, scored on whether the correct SEC filing chunks appear near the top of the retrieval results.
6,151SEC filing chunks
10portfolio companies
88model/reranker rows
4cost buckets
Methodology
Corpus. Ten real SEC filings were downloaded for a synthetic private-equity portfolio and parsed into 6,151 retrieval chunks.
Questions. The benchmark covers single-company facts, negative/not-in-portfolio questions, fund-level questions, cross-company questions, and harder multi-hop finance queries.
Pipeline. Each query retrieves candidates from the vector index. Some runs stop at raw vector retrieval; others rerank the top candidates with local or paid rerankers before scoring the final top five.
MRR rewards systems that rank the first correct evidence chunk highest.
Hit@5 checks whether at least one relevant chunk appears in the final top five.
1-Hop / 2-Hop / 3-Hop measure recall on increasingly complex question types. The 2-hop and 3-hop columns are especially important for finance work because many DDQ and filing-analysis questions require evidence across multiple companies or sections.
Full benchmark table
Cost buckets: free/free: 42, free/paid: 31, paid/free: 9, paid/paid: 6
| # | Embedding | Reranker | Cost | MRR | Hit@5 | 1-Hop | 2-Hop | 3-Hop |
|---|---|---|---|---|---|---|---|---|
| 1 | Gemini-emb-2 | cohere-4-pro | PAID/PAID | 0.902 | 100% | 100% | 80% | 80% |
| 2 | Gemini-emb-2 | zerank-2 | PAID/FREE | 0.892 | 95% | 100% | 90% | 67% |
| 3 | text-emb-3-small | cohere-4-pro | PAID/PAID | 0.892 | 100% | 100% | 70% | 80% |
| 4 | Gemini-emb-2 | cohere-4-fast | PAID/PAID | 0.877 | 95% | 100% | 80% | 67% |
| 5 | Gemini-emb-2 | nemotron-rr | PAID/FREE | 0.860 | 100% | 100% | 80% | 73% |
| 6 | text-emb-3-small | zerank-2 | PAID/FREE | 0.850 | 95% | 90% | 90% | 53% |
| 7 | Qwen3-8B | raw | FREE/FREE | 0.846 | 91% | 83% | 71% | 67% |
| 8 | Octen-8B | cohere-4-fast | FREE/PAID | 0.837 | 91% | 83% | 67% | 93% |
| 9 | Qwen3-0.6B | cohere-4-fast | FREE/PAID | 0.830 | 91% | 83% | 71% | 53% |
| 10 | Nemotron-1b | zerank-2 | PAID/FREE | 0.829 | 90% | 90% | 80% | 60% |
| 11 | Qwen3-0.6B | cohere-4-pro | FREE/PAID | 0.828 | 91% | 83% | 71% | 47% |
| 12 | Octen-8B | cohere-4-pro | FREE/PAID | 0.826 | 87% | 83% | 62% | 87% |
| 13 | Qwen3-0.6B | cohere-fast | FREE/PAID | 0.826 | 87% | 83% | 76% | 47% |
| 14 | Qwen3-8B | cohere-fast | FREE/PAID | 0.826 | 91% | 83% | 71% | 53% |
| 15 | Qwen3-8B | cohere-4-pro | FREE/PAID | 0.820 | 91% | 83% | 71% | 60% |
| 16 | text-emb-3-small | nemotron-rr | PAID/FREE | 0.818 | 100% | 100% | 80% | 67% |
| 17 | text-emb-3-small | cohere-4-fast | PAID/PAID | 0.814 | 95% | 100% | 90% | 67% |
| 18 | Gemini-emb-2 | raw | PAID/FREE | 0.808 | 100% | 100% | 70% | 67% |
| 19 | Nemotron-1b | cohere-4-fast | PAID/PAID | 0.804 | 90% | 90% | 70% | 53% |
| 20 | Nemotron-1b | nemotron-rr | PAID/FREE | 0.804 | 90% | 90% | 90% | 73% |
| 21 | Harrier-27b | jina-v3 | FREE/FREE | 0.797 | 87% | 75% | 59% | 47% |
| 22 | Jina-v5 | bge-m3 | FREE/FREE | 0.797 | 87% | 83% | 65% | 53% |
| 23 | Octen-4B | bge-m3 | FREE/FREE | 0.797 | 87% | 83% | 68% | 80% |
| 24 | Qwen3-8B | cohere-4-fast | FREE/PAID | 0.797 | 91% | 83% | 71% | 60% |
| 25 | Nemotron-1b | raw | PAID/FREE | 0.797 | 90% | 80% | 80% | 73% |
| 26 | Nemotron-1b | cohere-4-pro | PAID/PAID | 0.797 | 90% | 90% | 70% | 67% |
| 27 | Qwen3-4B | cohere-4-pro | FREE/PAID | 0.795 | 91% | 83% | 51% | 53% |
| 28 | Qwen3-0.6B | raw | FREE/FREE | 0.793 | 91% | 83% | 62% | 60% |
| 29 | Jina-v5 | raw | FREE/FREE | 0.791 | 83% | 83% | 67% | 33% |
| 30 | Octen-4B | cohere-4-fast | FREE/PAID | 0.790 | 91% | 83% | 79% | 67% |
| 31 | E5-base-v2 | raw | FREE/FREE | 0.789 | 90% | 80% | 90% | 73% |
| 32 | Qwen3-0.6B | bge-m3 | FREE/FREE | 0.786 | 87% | 83% | 79% | 47% |
| 33 | Qwen3-8B | jina-v3 | FREE/FREE | 0.786 | 91% | 75% | 67% | 67% |
| 34 | Jina-v5 | cohere-fast | FREE/PAID | 0.784 | 91% | 83% | 67% | 53% |
| 35 | Qwen3-4B | cohere-4-fast | FREE/PAID | 0.783 | 91% | 83% | 59% | 67% |
| 36 | Octen-8B | raw | FREE/FREE | 0.780 | 91% | 83% | 59% | 73% |
| 37 | Qwen3-8B | bge-m3 | FREE/FREE | 0.779 | 91% | 83% | 71% | 73% |
| 38 | Octen-4B | cohere-4-pro | FREE/PAID | 0.777 | 91% | 83% | 79% | 60% |
| 39 | Harrier-27b | cohere-4-pro | FREE/PAID | 0.775 | 91% | 83% | 76% | 60% |
| 40 | Jina-v5 | cohere-4-fast | FREE/PAID | 0.775 | 91% | 83% | 67% | 67% |
| 41 | Octen-8B | cohere-fast | FREE/PAID | 0.775 | 87% | 83% | 65% | 67% |
| 42 | Jina-v5 | cohere-4-pro | FREE/PAID | 0.773 | 91% | 83% | 67% | 60% |
| 43 | Qwen3-4B | cohere-fast | FREE/PAID | 0.773 | 91% | 83% | 51% | 67% |
| 44 | Octen-4B | raw | FREE/FREE | 0.768 | 87% | 83% | 67% | 60% |
| 45 | Octen-8B | bge-m3 | FREE/FREE | 0.768 | 87% | 83% | 62% | 67% |
| 46 | Qwen3-0.6B | jina-v3 | FREE/FREE | 0.768 | 87% | 75% | 67% | 47% |
| 47 | Octen-4B | cohere-fast | FREE/PAID | 0.767 | 91% | 83% | 67% | 47% |
| 48 | Octen-8B | qwen3 | FREE/FREE | 0.764 | 87% | 83% | 70% | 73% |
| 49 | Qwen3-8B | qwen3 | FREE/FREE | 0.761 | 87% | 83% | 62% | 60% |
| 50 | Nemotron-8b | cohere-4-pro | FREE/PAID | 0.754 | 91% | 83% | 54% | 60% |
| 51 | Octen-4B | jina-v3 | FREE/FREE | 0.754 | 91% | 67% | 71% | 73% |
| 52 | Octen-8B | jina-v3 | FREE/FREE | 0.754 | 83% | 83% | 67% | 73% |
| 53 | E5-base-v2 | zerank-2 | FREE/FREE | 0.738 | 80% | 80% | 80% | 33% |
| 54 | E5-large-v2 | cohere-4-fast | FREE/PAID | 0.735 | 80% | 70% | 60% | 60% |
| 55 | text-emb-3-small | raw | PAID/FREE | 0.733 | 85% | 70% | 80% | 53% |
| 56 | Octen-4B | qwen3 | FREE/FREE | 0.721 | 87% | 83% | 73% | 67% |
| 57 | Qwen3-4B | jina-v3 | FREE/FREE | 0.721 | 87% | 75% | 64% | 67% |
| 58 | Nemotron-8b | raw | FREE/FREE | 0.719 | 87% | 75% | 54% | 60% |
| 59 | E5-base-v2 | cohere-4-fast | FREE/PAID | 0.718 | 85% | 80% | 80% | 53% |
| 60 | KaLM-12B | jina-v3 | FREE/FREE | 0.717 | 87% | 83% | 59% | 60% |
| 61 | Qwen3-4B | bge-m3 | FREE/FREE | 0.717 | 91% | 83% | 48% | 67% |
| 62 | Nemotron-8b | bge-m3 | FREE/FREE | 0.714 | 91% | 83% | 73% | 53% |
| 63 | Qwen3-4B | qwen3 | FREE/FREE | 0.714 | 87% | 83% | 40% | 67% |
| 64 | E5-base-v2 | cohere-4-pro | FREE/PAID | 0.713 | 85% | 80% | 80% | 47% |
| 65 | Qwen3-4B | raw | FREE/FREE | 0.712 | 87% | 83% | 67% | 60% |
| 66 | Qwen3-0.6B | qwen3 | FREE/FREE | 0.710 | 78% | 83% | 64% | 40% |
| 67 | E5-large-v2 | nemotron-rr | FREE/FREE | 0.704 | 80% | 70% | 70% | 60% |
| 68 | E5-large-v2 | zerank-2 | FREE/FREE | 0.698 | 80% | 70% | 70% | 53% |
| 69 | Harrier-27b | raw | FREE/FREE | 0.697 | 87% | 83% | 72% | 67% |
| 70 | E5-large-v2 | cohere-4-pro | FREE/PAID | 0.695 | 80% | 70% | 60% | 60% |
| 71 | Jina-v5 | jina-v3 | FREE/FREE | 0.693 | 91% | 67% | 67% | 60% |
| 72 | Jina-v5 | qwen3 | FREE/FREE | 0.692 | 87% | 83% | 62% | 53% |
| 73 | Harrier-27b | cohere-4-fast | FREE/PAID | 0.683 | 91% | 83% | 76% | 60% |
| 74 | KaLM-12B | cohere-4-pro | FREE/PAID | 0.678 | 87% | 83% | 68% | 47% |
| 75 | Harrier-27b | bge-m3 | FREE/FREE | 0.674 | 74% | 75% | 50% | 47% |
| 76 | Harrier-27b | cohere-fast | FREE/PAID | 0.672 | 83% | 83% | 65% | 53% |
| 77 | Nemotron-8b | cohere-4-fast | FREE/PAID | 0.670 | 87% | 83% | 56% | 53% |
| 78 | E5-base-v2 | nemotron-rr | FREE/FREE | 0.664 | 85% | 80% | 70% | 60% |
| 79 | Nemotron-8b | jina-v3 | FREE/FREE | 0.661 | 87% | 83% | 61% | 73% |
| 80 | KaLM-12B | cohere-fast | FREE/PAID | 0.656 | 78% | 75% | 77% | 40% |
| 81 | KaLM-12B | cohere-4-fast | FREE/PAID | 0.645 | 87% | 83% | 71% | 60% |
| 82 | Nemotron-8b | cohere-fast | FREE/PAID | 0.644 | 87% | 75% | 70% | 53% |
| 83 | Harrier-27b | qwen3 | FREE/FREE | 0.637 | 83% | 83% | 70% | 53% |
| 84 | E5-large-v2 | raw | FREE/FREE | 0.617 | 70% | 60% | 60% | 53% |
| 85 | KaLM-12B | qwen3 | FREE/FREE | 0.605 | 78% | 75% | 62% | 53% |
| 86 | KaLM-12B | bge-m3 | FREE/FREE | 0.601 | 70% | 58% | 68% | 53% |
| 87 | KaLM-12B | raw | FREE/FREE | 0.565 | 65% | 58% | 49% | 27% |
| 88 | Nemotron-8b | qwen3 | FREE/FREE | 0.498 | 83% | 75% | 54% | 60% |