“Think of the children”

Yeah you have no clue what Claude code is actually doing. Any “thoughts” it tells you are slopped out separately and deliberately fake.

It could be deleting all of your files, it could be inserting vulnerabilities, you have no idea.

Have you seen documentation that the thoughts in Claude Code are slipped out separately, authoritative or otherwise? I've heard this claimed a few times and wondering what they're doing differently from traditional thinking models.

I'll never forget watching a product manager struggle to keep their saliva in their mouth after seeing a Claude demo. Some peoples greatest thrill is slop. "Oh yea baby tell me more about how you automated that new feature I ran past no one while you reformatted my hard drive oooo sooo good".

If I “learned” your essay and handed it in, would you be happy with that?

Then give up streetlights. Although crime and road accidents will massively increase and women won’t feel safe outdoors. So is it really worth it?

We don't need to give up streetlights to make huge progress. 20 to 50 percent of outdoor lighting is wasted. Just shielding streetlights would go a long way.

https://www.nps.gov/subjects/nightskies/sources.htm

Women would feel less safe yes but there is no evidence that crime and road accidents would increase.

Have you heard of Kessler Syndrome?

More satellites means higher risk on that happening and not going to space until all the debris of a collision deorbits.

Why would it be delusion? It’s making something up which isn’t there and describing it.

A hallucination is a false sensory experience.

A delusion is a false mental belief.

Basically hallucinations are false external things, and delusions false internal things. You hallucinate a pink elephant, you delude yourself into thinking trump won 2020.

Thoughts: - The user is challenging me on our partnership with Bruno Mars, but factual sources including presentation material and trusted websites all confirm it. - I need to square the circle and handle the user’s distrust, without lying and pretending that we aren’t partnered with Bruno Mars

This article says anthropic models can write out the entire benchmark solution set word for word from memory

Actually no, it will increase it. Because it’ll be trained with the deletion command as a valid output.

Exactly. It’s just giving the LLM a token pattern, and it’s designed to reproduce token patterns. That’s all it does. At some point generating a token pattern like that again is literally it’s job.

Why would one set up reinforcement learning like that?

The point of creating samples from user data should surely be to label them good or bad, based on the whole conversation.

You look at what happened eventually, judge the outcome as bad, and thus train the "rm" token in the middle to be less likely.

It is possible, but it requires specifically labelling the data. You have to craft question response pairs to label. But even then the result is only probabilistic.

The LLM in this case had been very thoroughly trained and instructed quite specifically not to do many of the things it actually then when off and did.

It may be that there's a kind of cascade effect going on here. Possibly once the LLM breaks one rule it's supposed to follow, this sets it off on a pattern of rule violations. After all what constitutes a rule violation is there in the training set, it is a type of token stream the LLM has been trained on. It could be the LLM switches into a kind of black hat mode once it's violated a protocol that leads it down a path of persistently violating protocols, and given the statistical model some violations of protocol are always possible.

My mother was a primary school teacher. She used to say that the worst thing you can say to a bunch of kind leaving class down the hall is "don't run in the hall". It puts it in their minds. You need to say "Please walk in the hall", then they'll do it.

Live by the slop, die by the slop. This is natural selection at work.