A local LLM on a six-year-old graphics card can write your code all day, for free. So why isn’t every developer running one? The answer comes down to a single number: memory.
This year local models quietly crossed the line where you stop double-checking their code against a cloud model, at least for a defined slice of work. The unlock wasn’t raw intelligence, it was reliable tool-calling. But crossing that line is a hardware purchase, not a software install, and consumer hardware got more expensive right as the capability arrived. This video maps exactly where the line sits, what it costs to cross it, how to actually run a local coding model end to end, and whether you’re one of the people who should.
CHAPTERS
00:00 Why a free local model suddenly looks tempting
00:44 When running your own model stopped being a hobby question
01:31 Did local LLMs actually get good enough for coding?
02:54 The real wall: why VRAM decides almost everything
04:59 How to run a local coding model (engine, harness, editor)
07:58 The honest trade: what you gain and what you give up
09:45 Where this is heading, and whether it’s worth it for you
WHAT THIS COVERS
• The “good enough” bar: when a local LLM is worth trusting for coding
• Why tool-calling, not intelligence, was the thing that flipped
• The VRAM math: roughly half a gig of memory per billion parameters
• Why 24GB is the number everyone keeps repeating
• The mixture-of-experts memory trap that fools people
• Quantization: how hard you can compress weights before quality breaks
• The three pieces of a local setup: the engine, the harness, and your editor
• Why pointing a big cloud coding agent at a small local model backfires
• The hybrid play: plan with the frontier, run the cheap work local
• Who should build a local daily driver, and who should stay on the cloud
📬 Get the hotter takes in your inbox every Tuesday:
WHAT IS A LOCAL LLM?
A local LLM is a language model that runs entirely on your own machine instead of a company’s servers. You download the model weights, run them through an inference engine, and your code never leaves your computer. The appeal is control: a local model can’t be rate-limited at the worst moment, silently swapped, repriced, deprecated, or switched off by a policy you never saw. The cost is hardware, because the whole model has to fit in fast memory the GPU can reach, which is why VRAM (or a Mac’s unified memory) is the real gate.
RELATED VIDEOS
• Ollama vs LM Studio vs llama.cpp: which local runtime should you use? →
• The open-weight moment: why free models caught up so fast →
• Why AI coding agents give you surprise bills →
SOURCES & FURTHER READING
• Vicki Boykis, “Running local models is good now” (vickiboykis.com)
• Hacker News discussion: “Has anyone replaced Claude/GPT with a local model for daily coding?” (news.ycombinator.com)
• r/LocalLLaMA community threads on hardware, quantization, and runtimes (reddit.com)
• Mitchell Hashimoto, “My AI Adoption Journey” (mitchellh.com)
• Stanford AI Index 2026, on the closed-vs-open model gap (hai.stanford.edu)
• Anthropic’s statement on the Fable 5 and Mythos 5 suspension (anthropic.com)
• Simon Willison’s analysis of the model suspension (simonwillison.net)
• llama.cpp and GGUF quantization documentation (github.com)
• OpenRouter model usage rankings (openrouter.ai)
#LocalLLM #AICoding #SelfHosted #LLM #Devsplainers
コメント