“Which Model Is Best” Is the Wrong Question

I've been playing around with running AI models locally on my own PC lately, and it got me thinking about how the question we ask about these models has quietly changed. It used to be “which model is the best?” I wrote about this last year, and my answer back then was that the best LLM is the one you use the most, because that's the model that has accumulated the most context about you, your preferences, your work, and your way of thinking. But these days I don't think best is even the right frame anymore. The real question is which model is appropriate for the job in front of you.

Here's what's driving this. The open weight models are getting very close to the frontier models on benchmarks, but at a fraction of the cost. DeepSeek's new V4 Flash scores within a few points of the top proprietary models on reasoning and coding, comes with a 1M token context window, and costs about 8 cents per million input tokens. For comparison, OpenAI's Sol is priced at $5 per million input tokens and Anthropic's Fable at $10, which works out to 60 to over 100 times the price for models that sit only a few benchmark points apart. When the price gap is that wide and the capability gap is that narrow, paying the frontier premium stops being a default and becomes a decision you actually have to justify.

Additionally, some of the smaller models can now be hosted on your own PC. I ran Qwen 3.6 35B, quantized down to about 15GB, on my PC, which has 12GB of VRAM and 24GB of RAM, nothing special by today's standards. Setup took about twenty minutes, and now I have a personal LLM on my hard drive that I can run as much as I want, and whatever I feed into it stays on my machine.

Now, none of these models beat the best frontier models (i.e. Fable, Sol, etc.). But when you think about the work most of us actually do with AI, summarizing documents, troubleshooting IT problems, answering everyday questions, drafting the first pass of a routine email, most of it doesn't need the best model, it just needs a model that's good enough. Just like how in a law firm or legal department, you don't send every matter to your most senior counsel. You match the work to the resource and escalate when the stakes call for it. Personally, I think model selection is heading the same direction, where open models handle the volume and the frontier models become the escalation tier you reach for when it actually matters.

I think the future may be machines built to run these open models locally, with zero marginal cost per token, and the data kept safe in the local hard drive. For those of us whose profession runs on confidentiality, that second part might end up mattering more than the first.