I know a few people that are more or less prototyping little agents with it that monitor stuff for them and make some kind of discernment/decision about alerting them.
This was miserably failing for me on a new install. Guess the reason why may have been less to do with my skill and more to do with being on a broken version (or some mix of it).
This is the most basic level of eval, of whether they can produce output that will be considered by someone somewhere (usually a young urban US American) as informal toned. Real human communication is far more nuanced than this, different groups have different linguistic registers they're used to and things outside it sound odd even if they can't articulate why. You could also want to be informal but not over-familiar with the other person (for eg. in a discord chat to a new acquaintance) - actually looking at the outputs here, the Claude output seems best fitting for that (in my subjective view anyway) than to the one you gave it - or want many other little variations.
What makes one cringe and another recognize as familiar and comfortable is also pretty subtle and hard to define. These things need nuanced descriptions and examples to actually get right, and it's in understanding those nuances and figuring out the register of the examples that Grok outshines the others.
Claude 4.7 is the clear winner to me for manager and formal report updates.
As an ex-senior exec (hundreds of staff), the bolded timeline impact is a particular nuance that I would expect a Lead/Director to format for a VP+ audience. Interesting none of the other models did that. My eyes immediately went to impact statement, then worked back to context to grasp the whole situation.
Thanks from where I'm looking Grok 4.3 and Claude 4.7 do a better job on the informal close friend/coworker vibe.
ChatGPT sounds fake / formal phrasing (for the specific close friend context) and has em-dashes and uses capitalization. Hence, ChatGPT does not, imo grok the assignment ;)
Is it me or did GPT get noticeably more natural in word choice recently? You can see it between 4.1 and 5.5 here, but I'm not sure when that happened. (My guess would be one of the recent 5.x releases.)
Edit: I meant specifically the absence of bizarre phrasing. That seems to have improved.
Wow, I'm surprised. Grok 4.3 actually is noticeably better than the other two for the close-friend variant. Surprisingly I found Claude the cringiest of the three!
I know it's just an evaluation, but seeing an informal message and a prompt to ask to rewrite this informal message to the tone of an "informal message" when the original one sounds just fine, just makes me sad... Not because of this evaluation, but because it reminds me that this is how some people use LLMs, basically asking it to remove your own voice from texts that are generally fine already.
My sister in law is a pharmacist and the heaviest non-dev ChatGPT user I know and her main use case is writing professionally polite messages to doctors on how the drugs they prescribed to a patient would have killed them had she not caught a particular interaction or common side effect.
There's a lot of "tone" in it as she's not trying to anger these folks, but also it's quite serious, but also there's just everything else happening in medicine.
Pretty neat. This kind of tone self-moderation comes naturally to good communicators, but I know people (on and off the spectrum) who really, really need help with this, and it's cool to see LLMs are able to do this. There are a surprising number of people in the business world who are just totally unable to tone-police themselves. In the medical field I'd be worried about hallucinations, of course, but presumably your SIL fact-checks the output.
That makes it more sad, to me. Someone with those credentials should be able to communicate with their colleagues effectively. I wonder if she used to be able to.
It appears Hacker News disagrees that social skills are valuable skills. Mea culpa, I should have guessed.
There's something ironic about complaining about other people's social skill while you couldn't be bothered to make a point without sounding dismissive and condescending.
Navigating tough conversations takes time, attention, and mental energy. I’d rather a pharmacist spend that time on catching another dangerous contraindicated combo of drugs for a different patient. Actually, AI should soon be checking for that, too.
All three did well, and while I'm a Claude user, I found the Opus reply here added some unnecessary detail, like "Impact: Minimal; no downstream dependencies are currently at risk". Downstream dependencies weren't mentioned in the original message; for all we know downstream could be relying on a poorly performing API and is impacted by waiting another week for replacement.
Seeing this makes me wonder if Grok uses Claude conversations for training.
It's otherwise kind of surprising that they both converge on very similar phrases (e.g. "API integration is kicking my ass") that aren't anywhere in the prompt.
All of these were frankly terrible. I guess Grok’s “informal” version sounded the most like a real human, but only because it reads exactly like an Elon tweet (including his favorite emoji!). It’s obvious what they’ve been training on.
Fwiw - I did a fairly large comparison of Gemini Nano (the in browser ai model) vs a comparable free hosted model of Gemma (from OpenRouter) and the hosted model absolutely trashed the local model on every aspect of speed, reliability, availability, etc. [1]
I'm not particularly happy about that outcome as I wish we had more locally run AI models for reasons of privacy and efficiency, so this is more just a warning that at present there are some severe tradeoffs.
Thanks for the write-up and the comparison, but more importantly for using the API in production!
You’re highlighting the "state of the art" gap we’re working to close. Cloud models will always have the advantage of massive parameter counts, but our bet is that for a huge class of simpler or high-volume tasks, the upsides of on-device (e.g. zero-cost, permission-less start with no quotas/infra, network-resilience, privacy) make it a compelling trade-off.
The models have been getting better at a rapid clip, and the team is heads-down on optimizing performance and reliability. To that end, we're always grateful for feedback. If you hit specific bugs, crashes, or quality regressions, filing a report with repro steps is the best way to help us improve. You can file those on crbug.com under the "Chromium > Blink > AI" component.
There's a push and pull here, Typescript + React + Vercel are also very amenable to LLM driven development due to a mix of the popularity of examples in the LLMs dataset, how cheap the deployment is and how quick the ecosystem is to get going.
Memory makers make capital investements (build different factories, convert physical production lines, etc.) to meet orders that have been place for the next ~5 years.
OpenAI (or whoever) crashes and can't pay for the order leaving the memory makers in a tough spot.
The amount of money flowing both from the AI bubble and from quite literally scalping both the server and consumer market... They gambled on the opportunity and if they fail - it's their problem.
And how is that a problem and more importantly how's that a problem of Average Joe?
Capitalists did their gamble things. If they fail in that gamble what forbids them to sell the regular RAM they made for AI bubbleists to the regular consumers? Besides HBM it's just the regular chips which are exactly the same for the consumer/server market, why it would be any different?
So I made an IOS app just for us that does face detection and won't take the actual photo until all of the detected faces have their eyes open.
reply