Mistral's open-weight models are among the most practical options for teams that want strong language model performance without per-token pricing or sending data to a third party. They're small enough to run on a single GPU, permissively licensed in most releases, and well supported by the major serving runtimes.
The harder question isn't how to run them. It's which approach fits your situation — and that decision has more impact on cost, latency, and operational burden than any configuration choice you'll make afterward.
This guide covers both paths on NevTan Cloud: calling Mistral through the hosted inference API, and self-hosting on dedicated GPU infrastructure. It explains the trade-offs, the technical concepts that matter, and the failure modes worth knowing about before you commit.
If you want a Mistral endpoint and don't have hard residency or customization requirements, the hosted inference API is OpenAI-compatible and needs no infrastructure. Self-host when you need data residency, a custom fine-tune, unusual serving configuration, or sustained utilization high enough that dedicated hardware makes sense. Self-hosting means choosing a serving runtime, sizing VRAM to your model and context length, setting scaling bounds with a warm floor, and monitoring time-to-first-token alongside GPU utilization.
The First Decision: Hosted or Self-Hosted
Most teams that set out to self-host an open-weight model don't need to. Work through this before provisioning anything.
Your situation | Approach that fits |
|---|---|
Want a Mistral endpoint, standard use case | |
Need Mistral adapted to your domain or tone | Fine-tune a LoRA adapter, serve it managed |
Need answers grounded in your own documents | Vector RAG over a hosted model |
Strict data residency or air-gap requirement | Self-host on GPU instances |
Custom serving configuration or research work | Self-host |
Sustained high utilization | Self-host — dedicated hardware amortizes better |
Want dedicated hardware without building a container |
NevTan Cloud's inference API is OpenAI-compatible, so pointing an existing integration at a Mistral model is typically a base-URL and model-name change rather than a rewrite. The model catalog lists what's currently available.
The honest framing. Self-hosting is an ongoing commitment, not a setup task. You take on provisioning, driver compatibility, VRAM management, cold starts, capacity planning, and an on-call surface that now includes GPU nodes. That's the right trade in plenty of situations. It just shouldn't be the default.
A common misconception worth clearing up: a lot of teams assume self-hosting is obviously cheaper because they're not paying per token. Whether that holds depends on utilization. A dedicated GPU costs the same whether it's saturated or idle, so a model serving sporadic traffic can easily cost more than the API it replaced. The comparison also needs to include engineering time, which never appears on an infrastructure invoice but is usually the largest line item in the first few months.
Everything below assumes you've decided self-hosting fits.
Understanding the Model Requirements
Memory is the binding constraint
For LLM inference, VRAM determines what hardware you need more than anything else. Rough planning figures:
Model class | Precision | Approximate VRAM for weights |
|---|---|---|
7B parameters | FP16 | ~15–16 GB |
7B parameters | 8-bit | ~8 GB |
7B parameters | 4-bit | ~5–6 GB |
Mixtral 8x7B | FP16 | ~90 GB |
Mixtral 8x7B | 4-bit | ~25–30 GB |
These cover weights only. Actual memory use is higher, sometimes considerably, because of three additional consumers:
KV cache — grows with context length and concurrent requests, often the largest variable component
Activation memory — transient, scales with batch size
Framework overhead — CUDA kernels, fragmentation, allocator headroom
This is why a model whose weights "fit" in a card's capacity can still fail under concurrent load. Size against measured peak usage during a realistic load test, not against the weight figure.
Choosing a model size
A 7B-class instruct model handles most production tasks well: summarization, classification, extraction, structured output, straightforward question answering. Mixture-of-experts models like Mixtral offer stronger reasoning and multilingual performance at substantially higher hardware requirements.
The useful discipline is to start with the smaller model and prove it insufficient on your actual task before moving up. Benchmarks measure general capability; your workload is specific. A 7B model that's been shown the right examples frequently outperforms a much larger one used naively.
Mistral ships new releases regularly, so check their current lineup rather than working from a version string in any guide — including this one. Model names and licensing terms both change.
Serving Runtimes
The runtime you choose determines throughput more than the GPU does.
Runtime | Strength | Fits |
|---|---|---|
vLLM | Continuous batching, paged KV cache, high concurrency | Production serving with multiple simultaneous users |
Ollama | Simple setup, easy local parity | Prototyping, single user, low-traffic internal tools |
TGI | Production-oriented, strong streaming support | Teams already in that ecosystem |
Framework-native | No extra dependency | Batch jobs, experimentation |
Why continuous batching matters
A naive serving loop processes one request at a time. Twenty concurrent users means nineteen waiting, and GPU utilization sits low because the hardware is idle between requests.
Continuous batching changes the shape of this. vLLM's paged KV cache manages attention state in non-contiguous memory blocks — conceptually similar to how an operating system handles virtual memory — which lets new requests join a batch already in flight rather than queueing behind it. Requests that finish early release their slots immediately instead of holding the batch open.
The practical effect is that concurrency stops being a hard wall and becomes a throughput curve. How much you gain depends on sequence lengths, concurrency level, and batch configuration, so benchmark against your own traffic pattern rather than relying on a general multiplier.
AI model servers explained covers runtime selection in more depth.
Quantization: The Other Lever
Quantization reduces numerical precision of model weights, cutting memory footprint and often allowing a smaller GPU.
Precision | Memory vs. FP16 | Typical quality impact |
|---|---|---|
FP16 | Baseline | None |
8-bit | Roughly half | Usually minimal |
4-bit | Roughly a quarter | Modest, task-dependent |
The trade-off is real but uneven across tasks. Quantization loss tends to concentrate in specific capabilities rather than degrading everything uniformly:
Tolerates it well: summarization, classification, sentiment, extraction, simple Q&A
More sensitive: multi-step reasoning, code generation, mathematical work, long-context tasks
This is why published accuracy deltas are a poor guide. A benchmark average can look fine while the specific capability you depend on degrades noticeably. Validate on your own evaluation set — a few hundred representative inputs with known-good outputs, compared before and after.
Quantization also interacts with throughput. Smaller weights mean more room for KV cache, which means more concurrent requests on the same card. Sometimes the throughput gain matters more than the hardware saving.
How Deployment Works
The general shape of a self-hosted deployment on NevTan Cloud:
1. Code in a repository. Your serving application, Dockerfile, and configuration live in Git. Connect via the GitHub integration or Bitbucket, and enable auto-deploy so changes build automatically. Keeping the model version in Git rather than in a console setting means every change is reviewable and revertible.
2. A container defining the runtime. A CUDA base image, your serving runtime, and your application code. Deploy from a Dockerfile gives you the control GPU workloads need.
3. GPU infrastructure sized to the model. GPU instances provide the hardware; regions and availability matters because GPU supply varies by location and affects both availability and data locality.
4. Configuration through environment variables. Model identifier, context length limits, memory settings, and credentials belong in environment variables rather than baked into the image — so the same tested artifact promotes between environments unchanged.
5. Scaling rules with a floor and a ceiling. Scaling presets handle elasticity.
6. Monitoring across infrastructure and model behavior. Container metrics plus custom metrics through metric ingestion.
Three container requirements specific to LLM serving
Load the model once at startup. Loading weights takes tens of seconds to minutes. Doing it per request makes the service unusable. Load at container start and hold the model in memory.
Bind to all interfaces, not localhost. A container listening only on localhost is unreachable from outside itself. This trips up more first deployments than it should.
Separate health from readiness. Health means the process is alive. Readiness means weights are loaded and the service can actually respond. If your health check passes while a 15 GB model is still loading, traffic routes to a container that can't serve it, and requests time out.
That last one deserves emphasis: conflating health and readiness is one of the most common causes of mystery timeouts in LLM deployments. Gate readiness on a completed warm-up inference — not just on the process starting — so a replica only receives traffic once it can genuinely handle it. Troubleshooting AI deployment issues covers the diagnostic pattern.
On version pinning
Pin your CUDA base image and serving runtime versions, and verify their compatibility before building. Runtime and driver compatibility changes across releases, and a mismatch produces errors that look like model failures but are actually driver problems — cryptic, misleading, and a frequent source of lost hours.
Check the current release notes for whichever runtime you're using rather than copying a version pair from a guide. This is the fastest-aging part of any deployment tutorial.
Scaling Considerations
GPU workloads scale differently from CPU services, in ways that matter.
Keep a warm floor. Scaling to zero means the next request waits through a full weight load. For anything user-facing, keep at least one replica running. The idle cost buys you predictable latency.
Set a ceiling. GPU hourly rates are an order of magnitude above CPU instances, so an uncapped autoscaler responding to a retry loop gets expensive fast. A ceiling is cheap insurance.
Scale on the right signal. CPU utilization is misleading for GPU workloads — a saturated GPU can sit behind idle CPU. Better signals:
Signal | What it indicates |
|---|---|
GPU compute utilization | Whether the accelerator is actually working |
Request queue depth | The clearest scale-up trigger |
Time to first token | User-perceived degradation |
GPU memory utilization | Proximity to OOM risk |
Account for startup time in your thresholds. If a replica takes minutes to become ready, scaling at the moment you're already saturated means minutes of degraded service. Scale earlier than feels necessary.
How to scale AI applications on demand covers traffic patterns in more detail.
Monitoring LLM Serving
Standard application metrics don't capture what matters for language model inference.
Metric | Why it matters |
|---|---|
Time to first token | What users experience as responsiveness |
Tokens per second | Throughput once streaming begins |
GPU compute utilization | Whether you're paying for idle hardware |
GPU memory utilization | Leading indicator of OOM risk |
Request queue depth | Clearest signal that capacity is short |
Error rate by status code | Timeouts and application errors have different causes |
Output length distribution | Sudden shifts often indicate a prompt or model change |
Set targets from your own baseline. Latency and throughput vary enormously with model size, quantization, context length, batch configuration, and GPU type — any published figure is a different configuration than yours. Measure your deployment under realistic load, establish a baseline, then alert on deviation from it.
Alert on p99, not average. Averages hide the tail, and the tail is what users describe as "it's broken."
Monitoring AI inference performance covers threshold-setting, and GPU utilization and cost optimization covers reading utilization data to spot waste.
Securing the Endpoint
An inference endpoint is attack surface, and GPU-backed endpoints have a cost dimension that ordinary APIs don't.
Authenticate every request. Scoped API keys per client, revocable individually.
Rate limit per key. A single misbehaving client can saturate a GPU, degrading service for everyone and generating cost while doing it. Edge-level throttling is the cheapest protection available.
Keep secrets out of image layers. Anything in a Docker layer is readable by anyone who can pull the image. The same applies to Git history — a committed token is compromised even after removal.
Validate and bound inputs. Cap context length and output tokens at the boundary. Unbounded generation requests are both a cost risk and a latency risk.
Consider a gateway for multi-model setups. An AI gateway centralizes routing, key management, and usage tracking — which also means one place to look when routing is the problem.
Further reading: protecting AI APIs and endpoints and secure AI deployment best practices. Platform-level details are in the security overview and trust center.
On data handling: self-hosting means prompts and responses stay within your deployment rather than going to a third-party API — that's the main reason teams choose it. But it's an architectural property, not a guarantee by itself. Authentication, TLS, secret handling, and access controls all still apply.
Common Mistakes
Claiming all GPU memory. Leaving no headroom for CUDA overhead and activation memory produces out-of-memory crashes under concurrent load — which means they surface in production, not in testing. Leave meaningful headroom and tune down if you still see OOM.
Loading the model per request. Load once at startup.
Conflating health and readiness. Covered above, and worth repeating: it's the most common cause of timeouts that make no sense.
Ignoring cold starts. The first request after scaling from zero waits through a full weight load.
Sizing for imagined peak. The most expensive error in GPU inference. Start with the smallest configuration that could work, measure real traffic, scale on evidence.
Hardcoding credentials. Image layers and Git history are both permanent.
Skipping rate limits. On GPU hardware, an availability incident is also a cost incident.
Copying version pins from guides. Runtime, CUDA, and model releases all move. Verify current compatibility before building.
Assuming self-hosting is cheaper. It depends entirely on utilization. Do the arithmetic with your actual volume, and include engineering time.
Frequently Asked Questions
Should I use the hosted Mistral API or self-host?
Use the hosted API unless you have a specific reason not to — it requires no infrastructure and no ongoing operational commitment. Self-host when you need data residency, a custom fine-tune, unusual serving configuration, or have sustained utilization high enough that dedicated hardware makes sense. The comparison should include engineering time, which is the cost most teams leave out and which usually dominates early on.
Can I run Mistral without a GPU?
Technically yes, and it will be slow. CPU inference for a 7B model produces a small number of tokens per second against tens on a GPU. That's workable for overnight batch jobs or very low-traffic internal tools, and unusable for anything interactive. If you want to avoid GPU infrastructure but need responsive inference, a hosted API is a better answer than CPU self-hosting.
Which Mistral model should I start with?
A 7B-class instruct model is the usual starting point — it balances quality, speed, and hardware requirements well for most tasks. Move to a larger mixture-of-experts model only after measuring that 7B is genuinely insufficient for your workload, since the hardware requirement increases substantially. Check Mistral's current releases for available versions.
How much VRAM do I need?
Weights are the starting point — roughly 15–16 GB for a 7B model in FP16, considerably less quantized. But KV cache and activation memory add to that, and both scale with context length and concurrency. A model whose weights fit can still fail under load. Size against measured peak usage during a realistic load test.
Does quantization hurt model quality?
It costs some accuracy, and the amount depends heavily on the task. Summarization, classification, and extraction typically tolerate 4-bit well. Multi-step reasoning, code generation, and mathematical work are more sensitive. Published benchmark averages can mask this, because the loss concentrates in specific capabilities — validate on your own evaluation set rather than trusting a general figure.
How many concurrent users can one GPU handle?
It depends on model size, quantization, context length, and batching configuration — the range is wide enough that any single number would mislead. Measure with your own traffic: watch request queue depth and time-to-first-token, and scale when queue depth grows persistently. That's a more reliable signal than any benchmark concurrency figure.
How do I update the model version?
Keep the model identifier in configuration and change it via commit rather than editing a console setting. The pipeline rebuilds and rolls out, and with more than one replica you can roll through without a service gap. The real benefit is auditability — a version change becomes a reviewable, revertible commit.
What causes a deployed model to time out even though the container looks healthy?
Almost always a readiness versus health mismatch. Health means the process started; readiness should mean weights are loaded and the service can respond. If traffic routes on the health signal, requests arrive at a container still loading the model and the proxy times out before it finishes. Gate readiness on a completed warm-up inference.
Getting Started
Two paths, and which one fits depends on whether you need dedicated hardware.
For a Mistral endpoint without infrastructure: the hosted inference API is OpenAI-compatible and works with your existing SDK.
For dedicated hardware: connect your repository, build a container with your chosen serving runtime, size the GPU against measured memory use, set scaling bounds with a warm floor, and instrument time-to-first-token and GPU utilization before routing real traffic.
Either way, start with the smallest configuration that could plausibly work and let measurements drive the next decision. The most expensive mistake in GPU inference isn't choosing the wrong card — it's provisioning for traffic that hasn't arrived yet.
Inference docs · GPU instance docs · App platform docs
Key Takeaways
Decide hosted vs. self-hosted first. It affects cost and operational burden more than any later configuration choice.
Self-hosting isn't automatically cheaper. It depends on utilization, and the comparison must include engineering time.
VRAM is the binding constraint, and weights are only part of it — KV cache scales with context and concurrency.
The serving runtime determines throughput more than the GPU does. Continuous batching is the main lever.
Quantization loss is task-specific. Validate on your own evaluation set, not on benchmark averages.
Health and readiness are different signals. Conflating them is the most common cause of unexplained timeouts.
Keep a warm floor and a hard ceiling on GPU autoscaling.
Measure before optimizing. Every performance figure you read was produced on a different configuration than yours.
