Careeroza

Re-ranking

Retrieve Wide, Then Re-rank

First-stage search (BM25, vectors) is fast but rough. A re-ranker (a cross-encoder model) reads the query and each candidate together and gives a more accurate relevance score. It is too slow for millions of documents, so you use it only on the top 20-50.

query ──► hybrid search (top 30, ~20 ms) ──► re-ranker (score 30 pairs, ~100 ms) ──► top 4 ──► LLM prompt

Example

Example

The re-ranking pattern. rerank_score is a stand-in for a cross-encoder: it rewards documents that contain the query words close together and in order, something a single embedding cannot see.

import re

def words(t): return re.findall(r"[a-z0-9]+", t.lower())

def rerank_score(query, doc):        # stand-in for a cross-encoder model
    q, d = words(query), words(doc)
    overlap = len(set(q) & set(d)) / len(set(q))
    bigrams = set(zip(q, q[1:])) & set(zip(d, d[1:]))
    return round(overlap + 0.5 * len(bigrams), 3)

query = "refund to original payment method"
candidates = [                       # first-stage top 4, in first-stage order
    "Payment methods accepted: UPI, cards, net banking and COD.",
    "Refunds go back to the original payment method within 5 to 7 working days.",
    "Original invoices are emailed after payment.",
    "Refund status can be checked under My Orders.",
]
reranked = sorted(candidates, key=lambda d: rerank_score(query, d), reverse=True)
for d in reranked:
    print(rerank_score(query, d), "|", d)
print("keep top 2 for the prompt")

Output

1.8 | Refunds go back to the original payment method within 5 to 7 working days.
0.4 | Original invoices are emailed after payment.
0.2 | Payment methods accepted: UPI, cards, net banking and COD.
0.2 | Refund status can be checked under My Orders.
keep top 2 for the prompt

Real Re-rankers

# Option 1: Voyage AI hosted re-ranker
import voyageai
vo = voyageai.Client()                      # reads VOYAGE_API_KEY
result = vo.rerank(query, candidates, model="rerank-2.5", top_k=3)
for r in result.results:
    print(round(r.relevance_score, 3), r.document)

# Option 2: open-source cross-encoder, runs on your own machine
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = model.predict([(query, d) for d in candidates])

Sample output

python rerank.py

Example output

0.912 Refunds go back to the original payment method within 5 to 7 working days.
0.404 Refund status can be checked under My Orders.
0.118 Payment methods accepted: UPI, cards, net banking and COD.