I’ve been running local models on and off for over a year. Not because I have strong ideological feelings about open source AI, but because I wanted to understand what the gap actually looks like between running something yourself and using a hosted API. The gap is real, it’s been shrinking, and the tradeoffs are more interesting than either side of the debate usually admits.
A bit of context on my setup: I’ve been testing primarily on a MacBook Pro with Apple Silicon. I use Ollama for local inference because the ergonomics are good and the model library is solid. I’ve also done some testing on a Linux machine with a consumer GPU. Your experience will vary significantly based on hardware.
The Models Worth Knowing
Meta’s Llama 3 family is probably the most widely used open-weight collection right now. The 8B model runs comfortably on most modern hardware, is fast on Apple Silicon, and is genuinely good for a lot of general tasks — coding help, summarization, drafting. The 70B model needs more resources but produces noticeably better output on complex reasoning and writing. The 405B model requires serious hardware (or cloud inference) and competes with frontier closed models on many tasks.
Mistral’s models are worth knowing about. Mistral 7B punched above its weight when it released in late 2023 and sparked a lot of interest in smaller, efficient models. Mixtral 8x7B uses a mixture-of-experts approach — at inference time, it activates a subset of its parameters, so you get a larger effective model capacity without proportionally higher compute cost. That architecture has influenced a lot of what came after.
Microsoft’s Phi-3 family takes a different angle. Rather than competing on raw parameter count, Phi-3 models are trained on high-quality synthetic data specifically designed to produce strong reasoning in small model sizes. Phi-3 Mini (3.8B) is genuinely impressive at coding and step-by-step reasoning for its size. It runs fast. On tasks that suit it, it beats much larger models. On tasks that need broad world knowledge or nuanced writing, it struggles — the small parameter count catches up with you.
Google’s Gemma models are solid, especially for developers already in the Google ecosystem. Qwen from Alibaba has picked up attention for strong multilingual performance and good benchmark scores on coding tasks.
What Running Local Models Is Actually Like
With Ollama, getting a model running takes about two minutes. You install Ollama, run ollama pull llama3, and you’re generating. The CLI is clean. The API is a local HTTP endpoint that mimics the OpenAI API format, so most tools that support OpenAI can point at Ollama instead. That’s a practical advantage that matters — you can swap local inference into existing workflows without rewriting integration code.
The hardware reality: a quantized 8B model at 4-bit quantization needs roughly 5-6GB of VRAM or unified memory. On a Mac with 16GB of unified memory, that leaves plenty of headroom. Inference is fast. A quantized 70B model needs 40+ GB, which immediately limits your options. Most consumer hardware can’t do it usefully. The gap between “what the model is capable of” and “what you can actually run” is mostly a hardware problem.
Quantization is the other thing you quickly learn about. Running a full-precision model locally is generally not feasible on consumer hardware for anything beyond small models. Quantization reduces precision (usually from 16-bit to 4-bit or 8-bit floats), which shrinks memory requirements significantly but does degrade quality. How much degradation you see depends on the model and the task. For straightforward coding help and summarization, 4-bit quantized Llama 3 8B is fine. For subtle reasoning or nuanced writing, you start noticing the edges.
The Quality Gap
I want to be honest about this because a lot of open source AI coverage isn’t: the best open-weight models are genuinely very good, but the frontier closed models are still better on complex tasks. Using Llama 3 70B locally versus Claude Sonnet or GPT-4o is a real difference, not a slight one. For many tasks — especially those requiring long, coherent reasoning, deep coding understanding, or nuanced instruction following — the frontier models still have a meaningful lead.
The gap has shrunk considerably over the past year. It used to be that the best open model was a generation behind the best closed model. Now it’s more like: competitive on many tasks, behind on the hardest ones. That’s progress worth acknowledging.
Where open models are competitive or close enough that the tradeoffs favor them: shorter tasks, tasks where the prompt fits comfortably in context, coding assistance for common languages and frameworks, summarization, classification. Where they’re still behind: very long context tasks, multi-step reasoning chains, tasks requiring synthesis of specialized knowledge.
Why You Might Choose Local
Privacy is the clearest reason. If you’re working with code or documents you can’t send to an external API, local inference removes that concern entirely. Your data stays on your machine. No logs, no training data collection, no terms of service questions. For work in regulated industries or with sensitive proprietary code, this matters a lot.
Cost at scale is another real consideration. API calls add up. If you’re running high volume — thousands of requests a day — the economics of local inference look different than if you’re using AI intermittently. Compute you own doesn’t have per-token pricing.
Offline use. This sounds minor until you need it. Running an AI model without internet access is genuinely useful for travel, environments with restricted connectivity, or just working from a coffee shop with bad wifi.
Customization. You can fine-tune open models on your own data in ways you can’t with closed APIs. The tooling for this has gotten dramatically easier over the past year. Whether you actually need fine-tuning depends on your use case, but the option existing matters.
The Honest Tradeoff
I don’t run local models exclusively. For most of my day-to-day work — coding, writing, complex reasoning — I use Claude or GPT-4. The quality is better and I care about that. For specific use cases where privacy matters, where I want to experiment with model behavior without burning API credits, or where I’m working offline, I reach for Ollama and Llama 3.
The framing of “local vs cloud” as a philosophical choice misses that they serve different purposes and that most people who care about AI will end up using both. The more interesting question is which tasks belong in which category for your specific situation.
The open-weight ecosystem in mid-2026 is genuinely impressive. The tooling (Ollama, LM Studio, llama.cpp) has made local inference accessible to anyone with a decent laptop. The models are real. The gap to frontier models is real too. Worth knowing what you’re getting into on both sides.