Mistral Small 3
A 24B Apache-2.0 model tuned for speed and local agentic loops, fitting in 16 GB at 4-bit for real-time workflows.
The brutal truth
The only sub-16GB model on this list that runs real-time agent loops. Trade: weaker reasoning than 27B+ models.
Files and Versions
Direct inspected tree artifacts from Hugging Face. Verified filenames, exact byte sizes, model weight shards, and configuration definitions.
Files and versions are hosted on Hugging Face. You can view all branches, commits, model weights, and tokenizer artifacts directly.
Inspect mistralai/Mistral-Small-3-24B-Instruct-2501 /tree/main ↗Inspect /tree/main Repository ↗Decision matrix
✓ RUN / USE THIS IF…
- •You're on a 16GB GPU
- •You want Apache 2.0
- •Your 24GB GPU comfortably fits the model weights (~15GB VRAM) with dedicated headroom for inference.
- •You require 100% offline data privacy with zero third-party API logging, subscriptions, or prompt retention.
✕ WALK AWAY IF…
- •You need 70B-class reasoning
- •You want 128K+ context
- •Your GPU has less than 15GB VRAM and you cannot tolerate slow CPU/system RAM offload paging.
- •You prefer a plug-and-play web app interface and don't want to manage local model checkpoints, drivers, or runtimes.
True cost of ownership
About
A 24B Apache-2.0 model tuned for speed and local agentic loops, fitting in 16 GB at 4-bit for real-time workflows.
What it does
Download the weights from Hugging Face, pick a quantisation that matches your GPU, and run it locally with Ollama, llama.cpp, vLLM, or a compatible runtime. Local agents runs on your own hardware — no per-token bill, no third-party data handling.
Key features
Fast enough for real-time agent loops
Apache 2.0 with no additional restrictions
Use cases
Enrich new leads and route them to the right owner.
Chain tools and models into repeatable workflows.
Classify and draft responses to inbound tickets.
Pull metrics and generate weekly summaries automatically.
Quantization & Hardware Fit
VRAM Hardware Compatibility & Memory Allocation
| Quant | VRAM | Quality / Fit on 24GB | Install / Run Command (OLLAMA) | |
|---|---|---|---|---|
| Q4_K_M● ACTIVE | 15 GB | Standard · ✓ Fits (9GB spare) | ollama run mistral-small3:24b | |
| Q5_K_M | 18 GB | Standard · ✓ Fits (6GB spare) | ollama run mistral-small3:24b | |
| Q8_0 | 27 GB | Standard · ✕ Needs 27GB | ollama run mistral-small3:24b | |
| FP16 | 48 GB | Standard · ✕ Needs 48GB | ollama run mistral-small3:24b |
Buyer doubts, cleared
Can I actually run Mistral Small 3 on my 24GB GPU?
What's the fine print on the license?
Does the 32K tokens actually hold up in practice?
User reviews
Reasoning quality is close to the bigger models for structured tasks. Open-ended planning still needs a frontier model.
The output reads naturally once you tune the prompt. It has cut my first-draft time roughly in half.
Clean interface, consistent results once you land on a prompt you like. Wish there were more export presets for client handoffs.
Mistral Small 3 handled our batch jobs without surprises. The API is well documented and the rate limits are reasonable.
