IME tok/s is only useful with the additional context of ttft and total latency. At this point a given closed-model does not exist in a vaccuum but rather in a wider architecture that affects the actual performance profile for an API consumer.
This isn't usually an issue comparing models within the same provider, but it does mean cross-provider comparison using only tok/s is not apples-to-apples in terms of real-world performance.
Exactly. Really frustrating they don't advertise TTFT and etc, and that it's really hard to find any info in that regard on newer models.
For voice agents gpt-4.1 and gpt-4.1-mini seem to be the best low latency models when you need to handle bigger data more complex asks.
But they are a year old and trying to figure out if these new models(instant, chat, realtime, mini, nona, wtf) are a good upgrade is very frustrating. AFAICT they aren't; the TTFT latencies are too high.
More often than not I've seen this be the case. Refactoring as "rewrite using my idiomatic style, so that I can understand it", which does not scale across the team so the next engineer does the same thing.
I like guns and cars, but not sports. How exactly is it performative? Both are engineering marvels and fascinating to watch videos about, and they also happen to be a load of fun.
I watch hours of videos on both with nobody else around and don't really talk about those topics with others much. So in the spirit of HN, I'm actually curious to know what about those interests is performative?
I think it's beneficial to hear this, as I've definitely been on the other side of this before now. So, thanks for sharing.
As much as we can fault the technology and the hype around it, this as much a people problem as anything else. Before AI, this same problem happened with architecture/PoC to implementation hand-offs.
AI is a new tool that a lot of us are still figuring out, but that doesn't excuse poor communication.
The argument here is to validate those possibilities before acting on them.
reply