1. The post mainly reiterates a single idea (Capsicum enumerates what the process can do, seccomp provides a configurable filter) in many different ways. There is not much actual depth, code samples notwithstanding. Nothing on why different designs were chosen, how easy each is to use, outcomes besides the Chrome example, etc.
2. There are a lot of AI writing tells, like staccato sentences, parallelism ("Same browser. Same threat model. Same problem."), pointless summary tables, "it's not X, it's Y" contradiction ("This is not a bug. It is the original Unix security model"), etc.
3. The author has roughly a blog post a day, all with similar style and on widely varied topics, and in the same writing style. Unless the author has deep expertise on a remarkably wide range of topics and spends all their time writing, these can't reflect deep insight or experience, but minimal editing of AI output.
I work on research studying LLM writing styles, so I am going to have to steal this. I've seen plenty of lists of LLM style features, but this is the first one I noticed that mentions "tapestry", which we found is GPT-4o's second-most-overused word (after "camaraderie", for some reason).[1] We used a set of grammatical features in our initial style comparisons (like present participles, which GPT-4o loved so much that they were a pretty accurate classifier on their own), but it shouldn't be too hard to pattern-match some of these other features and quantify them.
If anyone who works on LLMs is reading, a question: When we've tried base models (no instruction tuning/RLHF, just text completion), they show far fewer stylistic anomalies like this. So it's not that the training data is weird. It's something in instruction-tuning that's doing it. Do you ask the human raters to evaluate style? Is there a rubric? Why is the instruction tuning pushing such a noticeable style shift?
I have nothing to contribute but speculation based on my intuition, but IMO RLHF (or rather human preference modeling in general, including the post-training dataset formatting) is a relatively small factor in this, RL-induced mode collapse is much bigger one. Take a look at the original DeepSeek R1 Zero, the point of which was to train a model with very little human preference, because they've been on a budget and human preference doesn't scale. It's pretty unhinged in its writing, like the base model, but unlike the base model it converges onto stable writing patterns, and the output diversity is as non-existent as in models with carefully engineered "personalities" like Claude. Ask it to name a random city and look at the logits, and you'll still see a pretty narrow distribution. At the same time some models with RLHF (e.g. the old RedPajama) have more diverse outputs.
Collapsed mode makes the models truncate entire token trajectories, repeat themselves, and indirectly it does something MUCH deeper, they converge on almost 1:1 input-to-output concept mapping (instead of one-to-many, like in base models). Same lack of variety can be seen in diffusion models, GANs, VAEs and any other model regardless of the type and receiving human preference.
Moreover, these patterns are generational. Old ones get replaced with new ones, and the list in the OP is going to be obsolete in a year. This is what already happened to previous models several times, from what I can tell. Supposedly this is because they scrape the web polluted by previous gen models.
It doesn't depend on the language at all, it's a failure mode of the model itself. English, Chinese, Spanish, C++, COBOL, base64-encoded Klingon, SVGs of pelicans on bikes, emoji-ridden zoomer speak, everything is affected and has its own specific -isms and stereotypes. Besides, they're also skewed towards the pretraining set distribution, e.g. Russian generated by some models has unnatural sounding constructions learned from English which is prevailing in the dataset and where they are common, e.g. "(character) is/does X, their Y is/does Z". I don't see why it should be different for programming languages, e.g. JS idioms subtly leaking into Rust, although it's harder to detect I suppose.
I've heard the Kenya and Nigeria story, but has anyone backed it up with quantitative evidence that the vocabulary LLMs overuse coincides with the vocabulary that is more common in Kenyan and Nigerian English than in American English?
The newer Claude models constantly use the word "genuinely" because Anthropic seems to have forcibly trained them to claim to be "genuinely uncertain" about anything they don't want it being too certain about, like whether or not it's sentient.
Interesting. Does this apply to all subjects? From what I understood, a major cause of hallucination was that models are inadvertently discouraged by the training from saying "I don't know." So it sounds like encouraging it to express uncertainty could improve that situation.
That's not a major issue. Any newer model with reasoning/web search has to be able to tell when it doesn't know something, otherwise it doesn't know when to search for it.
> It's something in instruction-tuning that's doing it.
Isn't the instruction tuning done with huge amounts of synthetic data? I wonder if the lack of diversity comes from llm generated data used for instruction tuning.
Probably. One common feature of LLM output is grammatical features that indicate information density, like nominalizations, longer words, participial clauses, and so on. Perhaps training tasks that involve asking the LLMs for concise explanations or summaries encourage the use of these features to give denser answers.
I wonder if it has to do with how meaning is tied to the tokens. c+amara+derie (using the official gpt-5 tokenizer).
There's also just that weird thing where they're obsessed with emoji which I've always assumed is because they're the only logograms in english and therefore have a lot of weight per byte.
No, that doesn't really work so well. A lot of the LLM style hallmarks are still present when you ask them to write in another style, so a good quantitative linguist can find them: https://hdsr.mitpress.mit.edu/pub/pyo0xs3k/release/2
That was with GPT4, but my own work with other LLMs show they have very distinctive styles even if you specifically prompt them with a chunk of human text to imitate. I think instruction-tuning with tasks like summarization predisposes them to certain grammatical structures, so their output is always more information-dense and formal than humans.
The first sentence is a reference to prior research work that has found those productivity gains, not a summary of the experiment conducted in this paper.
In that case it should not be stated as a fact, it should then be something like the following.
While prior research found significant productivity gains, we find that AI use is not delivering significant efficiency gains on average while also impairing conceptual understanding, code reading, and debugging abilities.
Outside of disciplines that use LaTeX, the ability of authors to do typesetting is pretty limited. And there are other typesetting requirements that no consumer tool makes particularly easy; for instance, due to funding requirements, many journals deposit biomedical papers with PubMed Central, which wants them in JATS XML. So publishers have to prepare a structured XML version of papers.
Accessibility in PDFs is also very difficult. I'm not sure any publishers are yet meeting PDF/UA-2 requirements for tagged PDFs, which include things like embedding MathML representations of all mathematics so screenreaders can parse the math. LaTeX only supports this experimentally, and few other tools support it at all.
At least in my experience, grad students don't pay submission fees. It usually comes out of an institutional finances account, typically assigned to the student's advisor (who is generally the corresponding author on the submission). (Not that the waiver isn't a good idea — I just don't think the grad students are the ones who would feel relieved by that arrangement.)
Also, I'm pretty sure my SIG requires LaTeX submissions anyway... I feel like I remember reading that at some point when I submitted once, but I'm not confident in that recollection.
> Outside of disciplines that use LaTeX, the ability of authors to do typesetting is pretty limited.
Since this is obviously true, and yet since most journals (with some exceptions) demand you follow tedious formatting requirements or highly restrictive templates, this suggests, in fact, that journals are outsourcing the vast majority of their typesetting and formatting to submitters, and doing only the bare minimum themselves.
Most of the tedious formatting requirements do not match what the final typeset article looks like. The requirements are instead theoretically to benefit peer reviewers, e.g., by having double-spaced lines so they can write their comments on the paper copy that was mailed to them back when the submission guidelines were written in the 1950s.
The smarter journals have started accepting submissions in any format on the first round, and then only require enough formatting for the typesetters to do their job.
For my area, everybody uses LaTeX styles that more or less produce PDFs identical to the final versions published in proceedings. Or, at least, it's always looked close enough to me that I haven't noticed any significant differences, other than some additional information in the margins.
It didn't "survey" devs. It paid them to complete real tasks while they were randomly assigned to use AI or not, and measured the actual time taken to complete the tasks vs. just the perception. It is much higher quality evidence than a convenience sample of developers who just report their perceptions.
Sure, if you're learning to write and want lots of examples of a particular style, LLMs can generate that for you. Just don't assume that is a normal writing style, or that it matches a particular genre (say, workplace communication, or academic writing, or whatever).
Our experience (https://arxiv.org/abs/2410.16107) is that LLMs like GPT-4o have a particular writing style, including both vocabulary and distinct grammatical features, regardless of the type of text they're prompted with. The style is informationally dense, features longer words, and favors certain grammatical structures (like participles; GPT-4o loooooves participles).
With Llama we're able to compare base and instruction-tuned models, and it's the instruction-tuned models that show the biggest differences. Evidently the AI companies are (deliberately or not) introducing particular writing styles with their instruction-tuning process. I'd like to get access to more base models to compare and figure out why.
Go vibe check Kimi-K2. One of the weirdest models out there now, and it's open weights - with both "base" and "instruct" versions available.
The language it uses is peculiar. It's like the entire model is a little bit ESL.
I suspect that this pattern comes from SFT and RLHF, not the optimizer or the base architecture or the pre-training dataset choices, and the base model itself would perform much more "in line" with other base models. But I could be wrong.
Goes to show just how "entangled" those AIs are, and how easy it is to affect them in unexpected ways with training. Base models have a vast set of "styles" and "language usage patterns" they could draw from - but instruct-tuning makes a certain set of base model features into the "default" persona, shaping the writing style this AI would use down the line.
I definitely know what you mean, each model definitely has it's own style. I find myself mentally framing them as like horses with different personalities and riding quirks.
Still, perhaps saying "copy" was a bit misleading. Influence would have been more precise way of putting it. After all, there is no such thing as a "normal" writing style in the first place.
So long as you communicate with anything or anyone, I find people will naturally just absorb the parts they like without even noticing most of the time.
I don't think the AI companies are systematically working to make their models sound more human. They're working to make them better at specific tasks, but the writing styles are, if anything, even more strange as they advance.
Comparing base and instruction-tuned models, the base models are vaguely human in style, while instruction-tuned models systematically prefer certain types of grammar and style features. (For example, GPT-4o loves participial clauses and nominalizations.) https://arxiv.org/abs/2410.16107
When I've looked at more recent models like o3, there are other style shifts. The newer OpenAI models increasingly use bold, bulleted lists, and headings -- much more than, say, GPT-3.5 did.
So you get what you optimize for. OpenAI wants short, punchy, bulleted answers that sound authoritative, and that's what they get. But that's not how humans write, and so it'll remain easy to spot AI writing.
That's interesting. I had not heard that. I wonder if making them sound more human and making them better at specific tasks though are mutually exclusive. (Or if perhaps making them sound more human is in fact also a valid task.)
In our studies of ChatGPT's grammatical style (https://arxiv.org/abs/2410.16107), it really loves past and present participial phrases (2-5x more usage than humans). I didn't see any here in a glance through the lightfastness section, though I didn't try running the whole article through spaCy to check. In any case it doesn't trip my mental ChatGPT detector either; it reads more like classic SEO writing you'd see all over blogs in the 20-teens.
edit: yeah, ran it through our style feature tagger and nothing jumps out. Low rate of nominalizations (ChatGPT loves those), only a few present participles, "that" as subject at a usual rate, usual number of adverbs, etc. (See table 3 of the paper.) No contractions, which is unusual for normal human writing but common when assuming a more formal tone. I think the author has just affected a particular style, perhaps deliberately.
Tangent, but I'm curious about how your style feature tagger got "no contractions" when the article is full of them. Just in the first couple of paras we have it's, that's, I've, I'd...
Probably because the article uses the Unicode right single quotation mark instead of apostrophes, due to some automated smart-quote machinery. I'll have to adjust the tagger to handle those.
If the output is interpreting sources rather than just regurgitating quotes from them, you need to exert judgment to verify they support its claims. When the LLM output is about some highly technical subject, it can require expert knowledge just to judge whether the source supports the claims.
1. The post mainly reiterates a single idea (Capsicum enumerates what the process can do, seccomp provides a configurable filter) in many different ways. There is not much actual depth, code samples notwithstanding. Nothing on why different designs were chosen, how easy each is to use, outcomes besides the Chrome example, etc.
2. There are a lot of AI writing tells, like staccato sentences, parallelism ("Same browser. Same threat model. Same problem."), pointless summary tables, "it's not X, it's Y" contradiction ("This is not a bug. It is the original Unix security model"), etc.
3. The author has roughly a blog post a day, all with similar style and on widely varied topics, and in the same writing style. Unless the author has deep expertise on a remarkably wide range of topics and spends all their time writing, these can't reflect deep insight or experience, but minimal editing of AI output.
So yes, it's pretty sloppy.
reply