The right embedding model for RAG is the smallest one that clears your retrieval-accuracy bar at acceptable latency and cost — not whichever model tops a public leaderboard. Because the embedding model decides which chunks your system retrieves, it sets the ceiling on answer quality: a weak embedding model means the LLM never even sees the right context. Choosing well is a trade-off across four things — accuracy, speed, cost, and vector dimensionality — tuned to your data and traffic.
This guide breaks down the factors that actually matter, how to measure accuracy on your own data instead of trusting benchmarks blindly, the dimensionality trade-off that quietly drives storage and search cost, and a decision framework by use case. It assumes you already know the basics of embeddings and vector search; here we focus purely on picking the model.
TL;DR: Judge an embedding model on retrieval accuracy (on your data), embedding/query speed, cost per million tokens plus vector-storage cost, and dimensionality. Higher dimensions can help accuracy but cost more to store and search. Start from a strong general model, evaluate 2–3 candidates on a small labeled set, and pick the cheapest/fastest one that hits your accuracy target. Always use the same model for indexing and querying.
What an Embedding Model Does in RAG
An embedding model converts text into a vector — a list of numbers that captures meaning, so semantically similar passages land near each other in vector space. In RAG, you embed your documents into a vector collection, embed the user's query the same way, and retrieve the nearest chunks to feed the model as context.
That makes the embedding model the gatekeeper of retrieval quality. Every downstream improvement — better prompts, a bigger LLM, reranking — is capped by whether the right chunk was retrieved in the first place.
Key takeaway: The embedding model sets your retrieval ceiling. Optimizing prompts or the LLM can't fix context the retriever never found.
The Factors That Actually Matter
Factor | What it affects | Watch out for |
|---|---|---|
Retrieval accuracy | Whether the right chunk is retrieved | Benchmark scores ≠ your data |
Speed / latency | Query latency and indexing throughput | Big models slow both |
Cost | Per-token embedding + vector storage | Re-embedding and storage add up |
Dimensionality | Storage size and search speed | Higher isn't always better |
Max input tokens | How big a chunk you can embed | Truncation silently drops content |
Language / domain | Accuracy on your content | General models weaken on niche jargon |
Open vs. hosted API | Control, cost model, ops | Trade control for convenience |
Key takeaway: There's no "best" embedding model — only the best fit for your accuracy target, latency budget, data, and cost model.
Accuracy: Measure It on Your Own Data
Public leaderboards like MTEB are a reasonable starting shortlist, but they're averaged across tasks that may look nothing like yours. A model that ranks #1 on general benchmarks can underperform a smaller model on your domain.
Evaluate candidates properly:
Build a small eval set — 50–200 real queries with the chunk(s) that should be retrieved for each.
Index your corpus with each candidate model.
Measure retrieval metrics — recall@k (did the right chunk appear in the top k?) and nDCG (was it ranked well?).
Compare on your numbers, not the leaderboard's.
Key takeaway: A 2-hour evaluation on your own queries beats any leaderboard. Benchmarks shortlist; your data decides.
Speed: Latency and Throughput
Two speeds matter. Query embedding latency is on the critical path of every request — the user waits for it — so an oversized model adds latency to every search. Indexing throughput governs how fast you can embed (and re-embed) your corpus. Larger models and higher dimensions slow both, and bigger vectors also make the nearest-neighbor search itself slower. For interactive RAG, a fast, smaller model often wins the overall experience even if it's a hair less accurate.
Cost: More Than the Per-Token Price
Embedding cost has three parts people routinely miss:
Embedding compute — the per-token price to generate vectors, paid on every document and every query, and again on any re-embedding when content changes.
Vector storage — proportional to dimensions × number of chunks. A 1024-dim model stores ~2.7× the bytes of a 384-dim model for the same corpus.
Search compute — higher dimensions mean more work per query at scale.
With usage-based pricing, a smaller-dimension model can cut both storage and query cost substantially — which is why dimensionality is a cost decision, not just an accuracy one.
The Dimensionality Trade-Off
Embedding models output vectors of a fixed size — commonly 384, 768, or 1024 dimensions. More dimensions can capture more nuance and lift accuracy, but they cost more to store and are slower to search. The sweet spot is usually the lowest dimensionality that still hits your accuracy target.
Some modern models support Matryoshka embeddings (or dimension truncation), letting you shorten the vector to a smaller size with only a small accuracy loss — a useful lever to trade a little recall for big storage and speed savings.
Key takeaway: Don't default to the highest-dimension model. Pick the smallest vector that passes your accuracy test — storage and search cost scale with it.
A Decision Framework by Use Case
Rather than chase a single "best" model, match the model class to your situation:
Interactive chat / low latency → a small, fast general model (e.g., a 384–768-dim open model). Speed matters more than the last point of recall.
High-accuracy retrieval (legal, research) → a larger, higher-dimension model, validated on your eval set; accept the extra cost where correctness is critical.
Multilingual content → a model explicitly trained multilingual; a strong English-only model will quietly underperform on other languages.
Niche domain (medical, code, finance) → test domain-tuned models, and consider fine-tuning an embedding model on your data if general ones fall short.
Budget- or scale-constrained → a smaller-dimension open model; the storage and compute savings compound at millions of chunks.
Browse the available models — their dimensions, context limits, and intended use — in the model catalog before you shortlist.
One Rule You Can't Break: Consistency
Whatever you choose, embed your documents and your queries with the same model. Vectors from different models live in different spaces and aren't comparable, so a mismatch quietly destroys retrieval quality. And if you later switch embedding models, you must re-embed the entire collection — it's a deliberate migration, not a gradual swap. The embeddings API docs cover how to generate vectors consistently across indexing and querying.
How NevTan Cloud Helps
NevTan Cloud provides hosted embeddings and vector collections, so you can generate vectors through an API and store them next to your inference and retrieval on one platform — no separate embedding service or vector database to wire together. You can compare models from the catalog, and usage-based pricing means a smaller, cheaper embedding model directly lowers both your per-token and storage bills. The documentation walks through generating embeddings and loading them into a collection.
Common Mistakes
Picking by leaderboard alone. Benchmarks are averaged across tasks unlike yours. Evaluate on your own queries.
Mismatched query and document models. Different models = incomparable vectors = broken retrieval. Use one model for both.
Over-dimensioning "to be safe." Higher dimensions cost more storage and slower search for often-marginal accuracy gains.
Ignoring re-embedding cost. Switching models or refreshing content means paying to embed again. Factor it in.
Wrong language or domain fit. A top English model underperforms on multilingual or niche-jargon corpora. Match the model to the content.
Frequently Asked Questions
How do I choose an embedding model for RAG? Shortlist a few strong general models, then evaluate them on a small labeled set of your own queries using recall@k and nDCG. Pick the cheapest, fastest model that clears your accuracy target — not necessarily the highest-ranked on a public benchmark.
Does the embedding model really affect RAG quality that much? Yes — it sets the retrieval ceiling. If the right chunk isn't retrieved, no prompt or LLM upgrade recovers it. The embedding model is often the highest-leverage choice in a RAG stack.
Do more dimensions mean better accuracy? Sometimes, but with diminishing returns and real costs — higher dimensions mean more storage and slower search. Choose the lowest dimensionality that still passes your accuracy test; Matryoshka models let you trim vectors with minimal loss.
Can I use different embedding models for queries and documents? No. Documents and queries must be embedded with the same model, because vectors from different models aren't comparable. A mismatch silently wrecks retrieval.
What happens if I switch embedding models later? You must re-embed the entire collection — the old and new vectors live in different spaces. Treat it as a one-time migration, and plan for the embedding cost.
Should I use an open model or a hosted API? Open models give you control and can be cheaper at scale; a hosted embedding API is simpler to operate. On NevTan Cloud you can run embeddings through its API and keep them beside your vector collections either way.
How do I measure embedding quality? Build an eval set of real queries with their correct chunks, index with each candidate, and compare recall@k and nDCG. Repeat whenever your data or model changes.
Pick for Your Data, Not the Leaderboard
Choosing an embedding model for RAG comes down to a disciplined trade-off: enough accuracy on your content, fast enough for your users, at a dimensionality and price you can afford to run and re-run. Shortlist from benchmarks, prove it on your own queries, favor the smallest vector that hits your target, and keep one model across indexing and querying.
NevTan Cloud gives you hosted embeddings, a model catalog to compare options, and vector collections to store the results — all on usage-based pricing. Start with the docs to generate your first embeddings and build a collection you can trust. Repo to production.
