没人谈 RAM：本地 LLM 的每一次后悔都是内存问题

Originally published at mrsaynothing.dev. Walk into any local-LLM thread and the arguments are about GPUs. VRAM benchmarks, 24 GB cards, CUDA versus ROCm, whether the 3060 is still the people's card. Meanwhile the number that actually decides whether your model runs is sitting in the other slot, unbenchmarkable and unmarketed: how much RAM the machine has. VRAM sells the dream. RAM decides whether the model boots at all — and how much context survives when it does. I tested this on the box in front of me while writing this post: a Ryzen desktop with 32 GB of RAM and a GeForce RTX 3060 with 12 GB of VRAM. Pulled llama3.1:8b — the download page says 4.9 GB. Watch what it actually reserved: That is the whole argument in one screen. A "4.9 GB model" reserved 7.0 GB before answering a single prompt — a 43% context tax — and kept 7,963 of 12,288 MiB of VRAM to itself. The weights were never the budget. The context was. Where the extra gigabytes come from The llama.cpp memory notes spell out the arithmetic that download pages omit: total memory = model weights + KV cache + compute buffer. Only the first term is constant. The KV cache grows linearly with context length, and the compute buffer grows with batch and graph shape. Ollama wraps llama.cpp, so the same law applies — which is why ollama ps reported 7.0 GB for a 4.9 GB tag at a 32,000-token window, all of it resident on the GPU. A quantized model isn't a compromise. It's an admission that memory was always the real budget. This is also why the GGUF quantization ladder exists at all. Q4 isn't a religion; it's what makes the memory arithmetic land inside hardware people own. And it's why two "identical" 8B setups behave nothing alike: same model, different context length, different machines entirely. Mr Say Nothing Mr Say Nothing Mr Say Nothing Follow Sep 13 GGUF Quantization: Which Level Should You Use? #ai #gguf #quantization #localllm Comments Add Comment 5 min read The table nobody runs before buying Same three model classes, both memory types, one 12 GB card and 32 GB of RAM — the configuration thousands of developers actually have: Model class (Q4) Download Loaded + 32k ctx On 12 GB VRAM On 32 GB RAM 7–8B (llama3.1:8b) 4.9 GB 7.0 GB (measured) 100% GPU, ~8 GiB used barely noticed 13–14B (qwen2.5:14b) 9.0 GB ~12 GB offload begins fine 27–32B (gemma3:27b) 17 GB ~20+ GB CPU pulls the weight the only reason it runs Read the last two rows again. On VRAM alone, a 14B model at a real context window is already a split-offload job, and a 27B is impossible. On 32 GB of system RAM, both are merely slow. That difference — between impossible and slow — is the entire practical difference between RAM and VRAM. If it fits in VRAM, it's fast. If it fits in RAM, it works. If it fits in neither, you're swapping to an NVMe drive and time loses meaning. Offload sits on PCIe, and the Ollama FAQ is straight about the cost: layers that don't fit on the GPU run on the CPU, and throughput drops hard as the GPU share shrinks. Nobody chooses that trade-off consciously. It happens silently, one layer at a time, and the symptom is just "local models feel overrated." The honest ledger Things that broke or surprised me while writing this piece, in order: The 43% number itself. I expected the 8B model to occupy "about its file size." It reserved 7.0 GB against a 4.9 GB download. If that surprised me, it surprises everyone who reads spec sheets instead of ps output. The screenshot session. My first capture grabbed the wrong window. My second worked, which is the one above. Receipts are a workflow, not a vibe — and a pull → capture → ollama rm cycle keeps the disk honest. What didn't break: the GPU never spilled. 8B at 32k context on 12 GB is genuinely comfortable. The card is fine. The discourse around the card is what's miscalibrated. None of this means GPUs don't matter — the 100% GPU line in that capture is why generation felt instant. It means the GPU is the second question. The runner choice comes after you know what fits, not before. Mr Say Nothing Mr Say Nothing Mr Say Nothing Follow Sep 18 llama.cpp vs Ollama: Which Should You Run in 2026? #ai #llamacpp #ollama #localllm Comments Add Comment 5 min read The rule of thumb worth keeping RAM you need = model file + KV cache for your real context window + 4 GB for being a computer. For 7–8B at Q4, 16 GB is comfortable. For 14B–32B, 32 GB stops being a luxury and starts being the point. Buy VRAM for the speed you want at the context you use; buy RAM for everything you'll ever load. Check ollama ps once and the spec-sheet religion quietly ends. So, two questions. When you spec your next dev box, are you buying VRAM for the benchmarks you'll post — or RAM for the models you'll actually run? And honestly: how many of the models you pulled at 2 a.m. did you delete before breakfast? I did, tonight, mid-post. Tell me I'm not the only one in the comments — and tell me which side of the RAM/VRAM split your build sits on. More field notes at mrsaynothing.dev · code at GitHub · say hi: contact@mrsaynothing.dev