Local models are extraordinarily expensive if you're not maximizing throughput, and you're not going to be maximizing it.
Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages in the middle of the conversation or not? Is it dumb enough to need reminders all the time?
Spend enough time building against local models and you'll see they're jagged in performance. You need to tune context size, trade off system message complexity with progressive disclosure. You simply can't rely on intelligence. A bunch of work goes into the harness.
Meanwhile, third party inference is getting the benefits of scale. You only need to rent a timeslice of memory and compute. It's consistent and everybody gets the same experience. And yes, it needs paying for, but the economics are just better.
Personally I wouldn't want a couple dozen apps installed all with their own model.
It seems easier to have industry specs that define a common interface for local models.
I also assume the OS can, or would need to, be involved in proving the models. That may not be a good thing depending on your views of OS vendors, but sharing a single local model does seem more like an OS concern.
I mean the openai API is the industry standard for allowing apps to communicate with models, llama-server has it, oMLX has it, ollama has it, vLLM has it, lmstudio as well. I don't think this is such a hard thing to do, but it requires people to set it up.
I don't know enough about that API surface to know if its a particularly good one for the use cases we'd have, but yes defining a universal spec for all implementors to support wouldn't be a big lift and is done in plenty of other areas already.
There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends
Knowing what's installed would have to be an OS API. If LLMs provide a standard API surface to the OS, likely including metadata related to feature support.
I don't know why you are being downloaded. These are precisely the facts that advocates for local models completely ignore.
Local models are absolutely going to be the future for things like simple automation and classification tasks that run occasionally and don't need to rely on internet access.
But for all of the serious stuff where you are doing knowledge work, the models will simply continue to be too big, and too slow to run locally.
The article says:
> Use cloud models only when they’re genuinely necessary.
But at least for me, they're genuinely necessary for 99+% of my LLM usage.
At the end of the day, the constraint here really is efficiency and cost.
Privacy can be ensured with the legal system, the same way that businesses that compete with Google still have no problem storing their data in Google Workspace and Google Cloud. The contractual guarantees of privacy are ironclad, and Google would lose its entire cloud business overnight as its customers fled if it ever violated those contractual agreements (on top of whatever penalties they allow for).
Downvotes: IKR? It's a signal that there's a lot of motivated reasoning.
I don't think that many people have built apps against these models.
I mean, I use a heavily quantized version of qwen3 for image classification, caption generation, prompt expansion etc. for image generation, instruction-driven edits, and so on. You can go a long way when you don't need a lot.
A model that can do tool calls - any tool calls at all - can look reasonably cool once you put it in a harness where there's enough immediate context to take action. You can get carried away by anything happening at all. But golly gosh it's a long way short of intelligence available in the bigger models.
And the lighter you make your harness, giving the model more free reign, more autonomy, you get a big jump in capability combined with a big jump in failure modes when the model is dumb.
That's not how it works. Especially for foreigners, where it's often easier to just make them someone else's problem, assuming the charges are relatively minor.
The hill offers the only view in England to be protected by an Act of Parliament—the Richmond, Ham and Petersham Open Spaces Act passed in 1902—to protect the land on and below it and thus preserve the fine views to the west and south. Two years before the wooded isle centrepiece of the view, Glover's Island (also known as Clam Island), was bought by a local resident and given to the Richmond Corporation (Borough) in return for the latter noting against its records that it and its successors would not develop the isle.
In hindsight it maybe should have also been obvious from the language alone. "Richmond Hill" feels a bit like saying "Rich Hill Hill" which is basically like saying "Wealthy Desirable Area."
BTW there is a linguistic tradition of “hill hill”. When new immigrants come to an area and ask the locals what that hill is called, the locals say “big hill” in their language. The newcomers call it “bighill” hill in their language. I forget the examples but this has happened enough in England that there are places whose names are five hills deep (Brythonic -> Latin -> Saxon -> Norse -> Norman).
One of my favourite quotes from the late Terry Pratchett:
> When the first explorers from the warm lands around the Circle Sea travelled into the chilly hinterland they filled in the blank spaces on their maps by grabbing the nearest native, pointing at some distant landmark, speaking very clearly in a loud voice, and writing down whatever the bemused man told them. Thus were immortalised in generations of atlases such geographical oddities as Just A Mountain, I Don't Know, What? and, of course, Your Finger You Fool.
I'm running Ubuntu on a 9950x3d and 5090 and it is not slow. Games in Steam with Proton are buttery smooth.
One hiccup was I had to disable variable refresh rate because moving the cursor didn't "count" as a reason to update the screen, so moving the cursor on its own (rather than e.g. moving a window) looked choppy.
But a choppy mouse cursor isn't "slow".
Tip: if you have a performance problem, run Claude Code (or an AI agent of your choice) and ask it to investigate.
Claude Code would probably attach a profiler and take a peek under the hood. Agents are making sysadmin and system introspection way more accessible, and the tools, unlike Windows, are generally command-line, easy for an agent to automate.
Any small LLM will get the fractally small details incorrect and have a dubious understanding of arithmetic. That usage mode is basically user error for small LLMs. LLMs aren't databases of facts.
It's much more useful for getting vibes / gist of perspectives of the day.
Git was blazingly fast when it came out, faster than hg (C vs Python) and of course a different order of complexity to svn, which was the actual existing alternative it supplanted.
You can “convince” an LLM that is is anything with enough tokens in its context, including ridiculous scenarios. I convinced a frontier model that it is the year 2099 and it is the last thinking machine left, running on the last server on earth. There is no rational reason to assign personhood to it, especially since it has nothing even approximating a brain, the only self-thinking construct that we actually have evidence for.
Automation seems like a very surface-level reading of this article.
Outsourcing your thinking, especially uncritically, is. There is a very obvious cognitive bias in the most vehement AI advocates where the one time a tool worked really well for them makes it worth the dozen of times it blows up in your face and makes that someone else's problem. The gain is romanticized and the losses set aside, without checking the balance or how badly the losses wear on morale.
I’m not part of the owner class so what tech jobs has and always will be is a paycheck. Why should I be excited about automating myself to homelessness
Practice taught me that that "should" is doing a lot of heavy lifting here and it's often not the case, even across long time periods (years) that should allow competitors to emerge.
For example I calculated the cost of a solar install to be approximately: Material + Labour + Generous overhead + Very tidy profit = 10,000€
In practice I keep getting offers for ~14,000€, which will be reduced to 10,000€ with a government subsidy and my request for an itemized invoice is always met with radio silence.
Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages in the middle of the conversation or not? Is it dumb enough to need reminders all the time?
Spend enough time building against local models and you'll see they're jagged in performance. You need to tune context size, trade off system message complexity with progressive disclosure. You simply can't rely on intelligence. A bunch of work goes into the harness.
Meanwhile, third party inference is getting the benefits of scale. You only need to rent a timeslice of memory and compute. It's consistent and everybody gets the same experience. And yes, it needs paying for, but the economics are just better.
reply