Building Enterprise AI Search: Hybrid Dense & Sparse Retrieval with Reciprocal Rank Fusion

Why Pure Vector Embeddings Fall Short

The 2-Stage Hybrid Search Pipeline

Implementing Reciprocal Rank Fusion in TypeScript

RRF normalizes and merges rank positions without needing to calibrate raw disparate score ranges:

// hybrid-search.ts - Reciprocal Rank Fusion (RRF) with Sparse & Dense Retrieval
interface SearchResult {
  id: string;
  score?: number;
  document: string;
}

export function reciprocalRankFusion(
  sparseResults: SearchResult[], // e.g., BM25 or SPLADE
  denseResults: SearchResult[],  // e.g., OpenAI / Voyage embeddings
  k: number = 60 // RRF smoothing constant
): SearchResult[] {
  const rrfScores = new Map<string, { score: number; doc: string }>();

  // Accumulate sparse reciprocal rank
  sparseResults.forEach((item, rank) => {
    const existing = rrfScores.get(item.id) || { score: 0, doc: item.document };
    existing.score += 1.0 / (k + rank + 1);
    rrfScores.set(item.id, existing);
  });

  // Accumulate dense reciprocal rank
  denseResults.forEach((item, rank) => {
    const existing = rrfScores.get(item.id) || { score: 0, doc: item.document };
    existing.score += 1.0 / (k + rank + 1);
    rrfScores.set(item.id, existing);
  });

  // Sort by fused reciprocal rank score
  return Array.from(rrfScores.entries())
    .map(([id, data]) => ({
      id,
      document: data.doc,
      score: data.score,
    }))
    .sort((a, b) => (b.score ?? 0) - (a.score ?? 0));
}

Cross-Encoder Neural Re-Ranking

While bi-encoder embeddings process query and document independently to allow fast vector indexing, cross-encoders evaluate the query and candidate documents simultaneously via full multi-head self-attention. Applying a cross-encoder to the top 25 candidates yields state-of-the-art precision:

// cohere-rerank.ts - Cross-Encoder Neural Re-Ranking
import { CohereClient } from "cohere-ai";

const cohere = new CohereClient({ token: process.env.COHERE_API_KEY });

export async function reRankSearchResults(
  query: string,
  candidateDocs: Array<{ id: string; text: string }>,
  topN: number = 5
) {
  const response = await cohere.v2.rerank({
    model: "rerank-v3.5",
    query: query,
    documents: candidateDocs.map((d) => d.text),
    topN: topN,
    returnDocuments: false,
  });

  return response.results.map((result) => ({
    id: candidateDocs[result.index].id,
    text: candidateDocs[result.index].text,
    relevanceScore: result.relevanceScore,
  }));
}

Relevance Metrics: Evaluating Search Accuracy

Never tune search parameters on intuition alone. Track industry-standard ranking metrics:

• NDCG@10 (Normalized Discounted Cumulative Gain): Measures graded relevance quality across top 10 positions.
• MRR@10 (Mean Reciprocal Rank): Tracks how frequently the first correct answer appears in the #1 or #2 spot.
• Zero-Result Query Rate: Percentage of user queries that returned zero hits (target < 2%).

Enterprise Search Checklist

Frequently Asked Questions

What value of 'k' is best for Reciprocal Rank Fusion?

A smoothing constant of $k = 60$ is the standard empirical default proposed in academic literature, preventing outlier high ranks from dominating the fused score.

How much latency does a neural re-ranker add?

Re-ranking the top 20 documents typically takes 15ms to 35ms using lightweight quantized models or cloud APIs like Cohere Rerank.

Can PostgreSQL do hybrid search natively?

Yes! PostgreSQL can execute to_tsvector (Full-Text Search) and pgvector in a single SQL query, merging the results using RRF directly inside a Common Table Expression (CTE).