Best Local RAG Models for Ollama in 2026

Strong local RAG results come from pairing a fast embedding model with a generation model that fits your VRAM. Use this page to choose a 16GB/24GB-friendly stack before moving to cloud GPUs.

Embedding models for local retrieval quality

Model VRAM min/optimal Category Detail
MXBAI Embed Large 335M FP16 2GB / 10GB embedding Open
Snowflake Arctic Embed 335M FP16 2GB / 10GB embedding Open
Nomic Embed Text 137M FP16 2GB / 10GB embedding Open
Snowflake Arctic Embed 137M FP16 2GB / 10GB embedding Open
Snowflake Arctic Embed 110M FP16 2GB / 10GB embedding Open
All-MiniLM 33M FP16 2GB / 10GB embedding Open
Snowflake Arctic Embed 33M FP16 2GB / 10GB embedding Open
All-MiniLM 22M FP16 2GB / 10GB embedding Open
Snowflake Arctic Embed 22M FP16 2GB / 10GB embedding Open
BGE-M3 567M FP16 4GB / 12GB embedding Open

Generation models that fit local VRAM

Model VRAM min/optimal 3090 tok/s Data Detail
Ministral 3 14B Q4 12GB / 14GB 21 Measured Open
Ministral 3 14B Q5 14GB / 16GB 18.9 Measured Open
Glm 4.7 Flash 7B Q4 6GB / 16GB 30 Measured Open
Glm 4.7 Flash 7B Q5 8GB / 18GB 27 Measured Open
GPT-OSS 20B Q4 16GB / 20GB 11 Measured Open
DeepSeek-R1 14B Q4 10GB / 20GB 21 Measured Open
GPT-OSS 20B Q5 20GB / 22GB 9.9 Measured Open
DeepSeek-R1 14B Q5 12GB / 22GB 18.9 Measured Open
Glm 4.7 Flash 7B Q8 12GB / 22GB 21.6 Measured Open
Qwen3.5 35B Q4 22GB / 24GB 6.8 Measured Open
Qwen3.6 35B Q4 22GB / 24GB 6.8 Measured Open
Nemotron 3 Nano 30B Q4 20GB / 24GB 11 Measured Open

Recommended local RAG flow

  1. Pick an embedding model that fits your latency and memory budget.
  2. Use a 7B to 32B generator model that can sustain your expected context length.
  3. Tune retrieval quality first, then upgrade generation model size if needed.
Read RAG field notes Open embedding group Estimate VRAM for your stack