RAG: choosing how to search and answer.

These are the two measures used throughout the project. Recall counts relevant sources among the top ten results; nDCG also rewards their position, giving more credit to a relevant source near the top.

def recall_at(ranked_ids, relevant, k):
    return len(set(ranked_ids[:k]) & relevant) / len(relevant)


def ndcg_at(ranked_ids, relevant, k):
    dcg = sum(1 / math.log2(rank + 2)
              for rank, pid in enumerate(ranked_ids[:k]) if pid in relevant)
    ideal = sum(1 / math.log2(rank + 2) for rank in range(min(k, len(relevant))))
    return dcg / ideal if ideal else 0.0

From scripts/run_d1_pilot.py.

Each corpus passage is encoded as a vector once. For each question, only the question is encoded and compared with all passage vectors by dot product. The model version is fixed because updated weights would change the results.

self.model = SentenceTransformer(model_name, revision=revision, device="cpu")

# Vettori normalizzati a lunghezza 1: il prodotto scalare è la similarità coseno.
self.embeddings = self.model.encode(
    [p.indexed_text for p in passages],
    batch_size=batch_size,
    normalize_embeddings=True,
)

def search(self, query, top_k=10):
    query_vector = self.model.encode(
        [self.query_instruction + query], normalize_embeddings=True)[0]
    scores = self.embeddings @ query_vector      # la domanda contro tutti i brani
    order = np.argsort(-scores, kind="stable")[:top_k]
    return [RankedPassage(self.passages[i].passage_id, float(scores[i])) for i in order]

From src/retrieval.py.

In the final configuration, I kept models and questions fixed and varied only the number of passages passed to the reranker.

for d in RERANK_DEPTHS:  # 20, 30, 50, 60
    per_q = []
    for i, g in enumerate(gold_ranks):
        order = np.argsort(-sc[i][:d], kind="stable")
        pos = {int(p): r + 1 for r, p in enumerate(order)}
        per_q.append(metrics([pos[r - 1] if r <= d else r for r in g]))
    record(f"reranking sui primi {d}", per_q)

Adapted from esperimenti/studio-embedding-retrieval/16_configurazione_finale.py.