RAG 流水线可以很快却依然出错：嵌入评估开发者指南

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. A vector database can return 20 documents in 15 milliseconds and still give your RAG system a terrible answer. The hard part of retrieval is rarely generating vectors. It is deciding whether the vectors are putting the right documents near the query. That makes embedding evaluation an information-retrieval problem. Once you see it that way, three metrics become particularly useful: Recall@k: did we retrieve the relevant information at all? MRR: how quickly did we find the first relevant result? NDCG: did we put the most useful results near the top? Then there is a second problem. What happens when your users search in Hindi for English documentation? Or type a product name in Latin script inside a Hindi sentence? Or use internal terminology that never appeared in the embedding model's training data? And finally: when should you fine-tune the embedding model instead of changing something else? Let's build the answer from the ground up. 1. An embedding model is really a ranking system It is tempting to think of an embedding as a semantic fingerprint. You encode: "How do I rotate an API key?" Enter fullscreen mode Exit fullscreen mode and get a vector such as: [0.18, -0.07, 0.42, ...] Enter fullscreen mode Exit fullscreen mode You encode every document chunk in your corpus, store those vectors, and retrieve the closest ones. Usually the similarity is cosine similarity: cos(q, d) = (q . d) / (||q|| * ||d||) Enter fullscreen mode Exit fullscreen mode When vectors are normalized to unit length, cosine similarity becomes equivalent to the dot product: cos(q, d) = q . d Enter fullscreen mode Exit fullscreen mode So the system is doing something conceptually simple: query | v embedding | v rank documents by similarity | v top-k documents Enter fullscreen mode Exit fullscreen mode The important word is rank. Your application does not really care whether document A has a cosine similarity of 0.83 and document B has 0.79. It cares whether A should appear before B. This distinction has been present in information retrieval for decades. In 2002, Kalervo Järvelin and Jaana Kekäläinen published work on evaluating retrieval systems using graded relevance: a document can be irrelevant, somewhat useful, highly useful, and so on. Their motivation was practical: retrieval systems produce too many results, so evaluation has to reward systems that put highly relevant material near the top. Modern embedding search has the same problem. The vector database is merely the mechanism. Your real system is: semantic query -> ranking -> useful context Enter fullscreen mode Exit fullscreen mode That means you need a test set. 2. Build a retrieval test set before touching the model Suppose you have a documentation system with 100,000 chunks. You collect 500 real user queries: "how to rotate api keys" "why does websocket authentication fail" "can I run the agent without public ports" "delete an organization" ... Enter fullscreen mode Exit fullscreen mode For each query, you identify the documents that actually answer it. For example: Query: "how to rotate api keys" Relevant: doc_1842 doc_7119 doc_9320 Enter fullscreen mode Exit fullscreen mode That gives you a small retrieval benchmark. You can now compare embedding models using the exact same queries. This is much more useful than saying: Model A "looks more semantic." A useful benchmark should contain the ugly cases. Include short queries. Include vague queries. Include terminology that only your users understand. Include queries with multiple correct documents. Include queries where the correct answer is buried inside a large document. Include typos. Include multilingual queries. And keep a separate test set that you do not use while tuning the model. A simple evaluation dataset might look like: query_id: 17 query: "rotate api key" relevant_docs: - doc_1842 - doc_7119 query_id: 18 query: "websocket auth failure" relevant_docs: - doc_921 Enter fullscreen mode Exit fullscreen mode Now your embedding problem becomes measurable. 3. Recall@k answers the first question: "Did we find it?" Suppose there are five relevant documents for a query: Relevant = {A, B, C, D, E} Enter fullscreen mode Exit fullscreen mode Your embedding search returns: Top 5 = [X, A, Y, Z, B] Enter fullscreen mode Exit fullscreen mode You found 2 of the 5 relevant documents. So: Recall@5 = 2 / 5 = 0.40 Enter fullscreen mode Exit fullscreen mode In general: Recall@k = relevant documents retrieved in top-k ------------------------------------- total relevant documents Enter fullscreen mode Exit fullscreen mode The intuition is: Recall@k measures how much of the answer space you managed to bring into view. Consider a RAG system retrieving 10 chunks. If the answer requires information from one specific chunk and that chunk never appears in the top 10, your LLM cannot recover it. A stronger language model does not magically fix missing context. Why recall@k matters for RAG Imagine 1,000 evaluation queries, with an average of 4 relevant chunks each. That is approximately: 1,000 * 4 = 4,000 relevant chunks Enter fullscreen mode Exit fullscreen mode If Recall@20 is 0.80, your retriever brought back roughly: 4,000 * 0.80 = 3,200 Enter fullscreen mode Exit fullscreen mode relevant chunks. About 800 relevant chunks were missed. That is a concrete failure budget. Recall@k has an important limitation It treats all relevant documents equally. Suppose the relevant set is: [A, B, C] Enter fullscreen mode Exit fullscreen mode and two systems return: System 1: [A, X, Y, Z, B] System 2: [X, A, B, Y, Z] Enter fullscreen mode Exit fullscreen mode At k=5, both have identical recall. Yet System 1 puts the strongest result first. Recall tells you whether useful material entered the candidate set. It says much less about ordering. That is where MRR and NDCG enter. 4. MRR asks: "How quickly did we find something useful?" MRR stands for Mean Reciprocal Rank. For one query: reciprocal rank = 1 / rank_of_first_relevant_result Enter fullscreen mode Exit fullscreen mode So if the first relevant result appears at: rank 1 -> 1.00 rank 2 -> 0.50 rank 3 -> 0.33 rank 10 -> 0.10 Enter fullscreen mode Exit fullscreen mode For multiple queries, take the average. Example: Query 1 -> first relevant at rank 1 -> 1.00 Query 2 -> first relevant at rank 2 -> 0.50 Query 3 -> first relevant at rank 4 -> 0.25 Query 4 -> no relevant result -> 0.00 Enter fullscreen mode Exit fullscreen mode Then: MRR = (1.00 + 0.50 + 0.25 + 0.00) / 4 = 0.4375 Enter fullscreen mode Exit fullscreen mode MRR is especially useful when your query usually has one obvious answer. Think of questions such as: "what is the default timeout?" "where is the config file?" "how do I reset my password?" "what command starts the server?" Enter fullscreen mode Exit fullscreen mode For these, getting the correct result at rank 1 is much better than getting it at rank 15. But MRR ignores everything after the first relevant document. Consider: System A: [A, X, X, X, X] System B: [A, B, C, D, E] Enter fullscreen mode Exit fullscreen mode If A is relevant, both have: RR = 1.0 Enter fullscreen mode Exit fullscreen mode MRR sees them as identical. That is clearly wrong for many RAG applications. If your answer depends on several pieces of evidence, you care about the whole ranking. This is where NDCG becomes useful. 5. NDCG asks the more realistic question: "Did we rank the useful stuff properly?" NDCG stands for Normalized Discounted Cumulative Gain. The useful idea is simpler than the name. You assign each retrieved result a relevance grade. For example: 0 = irrelevant 1 = somewhat useful 2 = useful 3 = directly answers the question Enter fullscreen mode Exit fullscreen mode Now imagine your ranking is: rank 1 -> relevance 3 rank 2 -> relevance 2 rank 3 -> relevance 0 rank 4 -> relevance 1 rank 5 -> relevance 0 Enter fullscreen mode Exit fullscreen mode NDCG gives more credit to highly relevant documents near the top. The underlying DCG calculation is: DCG@k = sum from i=1 to k of (2^rel_i - 1) / log2(i + 1) Enter fullscreen mode Exit fullscreen mode The pieces have intuitive meanings. 2^rel_i - 1 Enter fullscreen mode Exit fullscreen mode makes relevance 3 much more valuable than relevance 1. And: log2(i + 1) Enter fullscreen mode Exit fullscreen mode discounts lower-ranked results. So relevance at rank 1 counts more than relevance at rank 10. Then normalize against the ideal ordering: NDCG@k = DCG@k / ideal_DCG@k Enter fullscreen mode Exit fullscreen mode Therefore: NDCG@k = 1.0 Enter fullscreen mode Exit fullscreen mode means your results are ordered exactly like the ideal ranking. A concrete example Suppose the ideal relevance grades are: [3, 2, 2, 1, 0] Enter fullscreen mode Exit fullscreen mode Your system returns: [1, 0, 3, 2, 0] Enter fullscreen mode Exit fullscreen mode The system still retrieved good documents. Recall might look decent. MRR might also look decent because a relevant document appears early. But NDCG drops because the grade-3 answer was pushed to rank 3. That is often exactly what you want to punish in a RAG pipeline. The metrics are telling you different things Think of them as three different questions: Recall@k -> Did we retrieve enough of the answer? MRR -> Did we find an answer quickly? NDCG -> Did we order useful answers correctly? Enter fullscreen mode Exit fullscreen mode A good retrieval benchmark often uses all three. A simple Python implementation makes the distinction concrete: import math def recall_at_k(ranked_ids, relevant_ids, k): relevant_ids = set(relevant_ids) retrieved = set(ranked_ids[:k]) return len(retrieved & relevant_ids) / len(relevant_ids) def reciprocal_rank(ranked_ids, relevant_ids): relevant_ids = set(relevant_ids) for rank, doc_id in enumerate(ranked_ids, start=1): if doc_id in relevant_ids: return 1.0 / rank return 0.0 def ndcg_at_k(relevances, k): def dcg(values): return sum( (2 ** rel - 1) / math.log2(i + 2) for i, rel in enumerate(values[:k]) ) actual = dcg(relevances) ideal = dcg(sorted(relevances, reverse=True)) return actual / ideal if ideal else 0.0 Enter fullscreen mode Exit fullscreen mode Notice something important: NDCG requires graded judgments. If every document is simply relevant or irrelevant, NDCG still works, but you are throwing away useful information. For many production systems, a practical judgment scheme is: 0 = does not answer the query 1 = related but insufficient 2 = useful evidence 3 = directly answers the query Enter fullscreen mode Exit fullscreen mode That is often enough to make ranking failures visible. 6. Multilingual retrieval changes the problem Now suppose your documentation is multilingual. A user asks: "How do I reset my password?" Enter fullscreen mode Exit fullscreen mode The relevant document is written in Hindi: "पासवर्ड भूल जाने पर आप अपना पासवर्ड रीसेट कर सकते हैं..." Enter fullscreen mode Exit fullscreen mode A genuinely multilingual embedding model should place those texts near each other in vector space. This is a harder problem than ordinary monolingual similarity because the model has to learn a language-independent semantic representation. This became a major research direction before today's LLM era. Mikel Artetxe and Holger Schwenk, for example, developed LASER, a multilingual sentence-embedding system covering 93 languages. Their work explicitly evaluated cross-lingual similarity and showed that a shared embedding space could support semantic search across languages. But "multilingual" does not mean "equally good in every language pair." You should test that yourself. For example, your evaluation matrix might look like: Query Document Recall@10 English English 0.91 Hindi Hindi 0.87 English Hindi 0.64 Hindi English 0.59 Tamil English 0.52 Hinglish English 0.47 These numbers are illustrative, but the structure of the test is important. The aggregate score could hide a serious problem. Suppose: English-English: 0.92 English-Hindi: 0.61 Hindi-English: 0.58 Enter fullscreen mode Exit fullscreen mode and 90% of your benchmark queries are English-English. Your overall recall could still look excellent. Your Hindi users would experience something very different. What tends to break multilingual retrieval? There are several recurring cases: Different scripts English: "insurance claim" Hindi: "बीमा दावा" Enter fullscreen mode Exit fullscreen mode Transliteration "बीमा" "beema" "bima" Enter fullscreen mode Exit fullscreen mode Code-switching "UPI ka transaction fail ho raha hai" Enter fullscreen mode Exit fullscreen mode Named entities A company, API, medicine, place, or product name may appear unchanged across languages. Domain vocabulary Your users may mix natural language with terms such as: OAuth JWT webhook protobuf nginx Enter fullscreen mode Exit fullscreen mode These are neither cleanly English nor cleanly Hindi. For multilingual retrieval, evaluate by language pair and query type rather than relying on one global score. The broader lesson came through later benchmark work too. The MTEB benchmark evaluated embedding models across many tasks and languages and found that no single embedding method dominated every task. Embedding quality is task-dependent. That observation matters enormously when somebody says: "We use the best embedding model." Best for what? 7. When should you fine-tune the embedding model? Fine-tuning is attractive because it feels like the direct solution: retrieval is bad | v fine-tune embedding model | v retrieval gets better Enter fullscreen mode Exit fullscreen mode Sometimes that is exactly right. Often it is premature. The first question should be: What kind of retrieval error are you trying to fix? Consider this failure: Query: "How do I rotate an API key?" Returned: "API authentication concepts" "API key configuration" "Creating API credentials" Enter fullscreen mode Exit fullscreen mode The system understands the general topic but misses the specific operation. This could be an embedding-model problem. Now consider another failure: Query: "How do I rotate an API key?" Relevant chunk exists. The embedding retriever finds the correct document. But your chunk is 3,000 tokens long and the relevant sentence is buried near the bottom. Enter fullscreen mode Exit fullscreen mode Fine-tuning the embedding model will not solve the real problem. You have a chunking problem. Or perhaps: Query: "delete organization" Enter fullscreen mode Exit fullscreen mode The system retrieves documentation for both: delete organization delete organization member Enter fullscreen mode Exit fullscreen mode The semantic representation may be perfectly reasonable. You may need metadata filters, query rewriting, or a reranker. A useful decision sequence Start with the cheapest intervention. 1. Check chunking 2. Check metadata filters 3. Check query preprocessing 4. Try a stronger embedding model 5. Add a reranker 6. Fine-tune the embedding model Enter fullscreen mode Exit fullscreen mode The exact order can change, but the principle is: Fix the smallest component that explains the failure. Fine-tuning becomes attractive when errors are systematic Suppose you have 5,000 real query-document judgments. You repeatedly see: "policy renewal" Enter fullscreen mode Exit fullscreen mode being ranked below: "policy purchase" Enter fullscreen mode Exit fullscreen mode even though your users consider renewal documents clearly relevant. That tells you something useful. Your production relevance function differs from the semantic relationships learned by the base embedding model. Now fine-tuning has a concrete target. You can train on examples such as: query: "policy renewal" positive: "Renewing an existing insurance policy" hard negative: "Purchasing a new insurance policy" Enter fullscreen mode Exit fullscreen mode The hard negative matters. Random negatives are usually too easy. If the model already knows that: "dog food" Enter fullscreen mode Exit fullscreen mode is unrelated to: "database replication" Enter fullscreen mode Exit fullscreen mode training on that pair teaches it almost nothing. You want confusing examples: positive: "reset password for an existing account" hard negative: "change account email address" Enter fullscreen mode Exit fullscreen mode The model needs to learn the boundary. Fine-tuning has an economic cost Suppose you want 2,000 evaluation queries. You inspect the top 20 retrieved chunks for each query. That is: 2,000 * 20 = 40,000 judgments Enter fullscreen mode Exit fullscreen mode At just 10 seconds per judgment: 40,000 * 10 seconds = 400,000 seconds ~= 111 hours Enter fullscreen mode Exit fullscreen mode The expensive part of embedding work is often not GPU training. It is producing trustworthy relevance data. That means your benchmark itself is an asset. Once you have it, you can evaluate: Embedding A Embedding B Embedding C Embedding C + reranker Fine-tuned C Fine-tuned C + reranker Enter fullscreen mode Exit fullscreen mode using the same dataset. There is also a production cost Suppose you store 1 million chunks with 1,536-dimensional float32 embeddings. Raw vector storage is approximately: 1,000,000 * 1,536 * 4 bytes = 6.144 GB Enter fullscreen mode Exit fullscreen mode before vector-index overhead, metadata, replication, and backups. A 768-dimensional representation cuts the raw vector storage roughly in half. This is why embedding evaluation is not purely about "which model has the highest score?" You are optimizing a system with several variables: retrieval quality latency embedding generation cost storage index size reranking cost engineering complexity Enter fullscreen mode Exit fullscreen mode Suppose a new model improves Recall@20 from: 0.84 -> 0.87 Enter fullscreen mode Exit fullscreen mode but doubles embedding generation cost and increases query latency. That improvement might still be worthwhile. Or it might be irrelevant if a cheap reranker gets you: 0.84 -> 0.91 Enter fullscreen mode Exit fullscreen mode at acceptable latency. You need measurements rather than intuition. 8. The practical evaluation loop A production embedding evaluation system does not need to be elaborate. Start with perhaps 300-1,000 real queries. For each query, store: query language relevant document IDs optional relevance grades Enter fullscreen mode Exit fullscreen mode Then run every candidate retriever over the same benchmark. Track at least: Recall@5 Recall@10 Recall@20 MRR NDCG@10 Enter fullscreen mode Exit fullscreen mode For multilingual systems, break them down by language pair. For domain-heavy systems, break them down by query category. For example: Recall@10 MRR NDCG@10 -------------------------------------------- Authentication 0.91 0.88 0.86 Billing 0.87 0.79 0.81 Deployment 0.83 0.76 0.77 Multilingual 0.62 0.51 0.55 Enter fullscreen mode Exit fullscreen mode Now you have something far more actionable than: "The embeddings seem pretty good." You can also investigate individual failures. For every bad query, log: query top-k documents similarity scores relevance labels language chunk metadata Enter fullscreen mode Exit fullscreen mode Then ask: Was the answer absent from top-k? Was the right document present but badly ranked? Was the chunk itself poor? Was the query ambiguous? Was this a language-specific failure? Enter fullscreen mode Exit fullscreen mode Those questions lead to different engineering fixes. That is the main mindset shift. Embedding evaluation is not a model leaderboard exercise. It is debugging a ranking system. The embedding model is one component inside that system. And once you have a real benchmark, the fine-tuning question becomes much easier. You can make the change, rerun the exact same queries, and ask: Did Recall@20 improve? Did NDCG improve? Did multilingual retrieval improve? Did we make another language worse? Did latency or cost change? Did the improvement survive on the held-out test set? Enter fullscreen mode Exit fullscreen mode That is much more reliable than choosing an embedding model because its benchmark score looks good. Conclusion Embeddings turn semantic similarity into geometry. Retrieval turns that geometry into a ranking. Evaluation tells you whether that ranking actually serves your users. Recall@k tells you whether the useful information entered the candidate set. MRR tells you how quickly the first useful result appears. NDCG tells you whether the most useful results were placed where users can actually benefit from them. Multilingual evaluation tells you whether a single aggregate number is hiding failures across languages. And a good benchmark tells you when fine-tuning is justified rather than merely tempting. The most useful question is therefore not: "Which embedding model should I use?" It is: "What retrieval behavior do my users actually need, and can I measure whether my system produces it?" What retrieval metric or failure mode has mattered most in the RAG systems you have built? Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: HexmosTech / LiveReview Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. blast-radius-demo.mp4 LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score How does Blast Radius scoring work? (a more technical explanation) Here's the goal: A 3-line fix in a function used by 40 other files, that also writes to a database, should score high. A 300-line UI change in one file, fully covered by… View on GitHub Click below to try LiveReview with your codebase: