It’s pretty obvious that professional photography takes real skill if you’ve ever experienced the disaster of someone trying to cut costs by hiring an amateur photographer for a major event like a wedding but expecting professional results (the mismatch in expectations being the key cause of disaster here).
I’m not saying it turns out bad 100% of the time, but it’s easy to forget because good professionals make it look effortless. When the skill isn’t there, though, and you're used to only seeing professional photos it becomes very obvious (and again, that's perfectly fine if you're not expecting professional photography).
I don't think it's about lethal autonomy specifically as much as it's just about government autonomy period. They don’t think private companies should have any veto power over how the government uses some technology they're provided.
On its face that’s not a crazy stance: Governments are meant to represent the public, while private companies obviously aren't. I think it’s somewhat understandable why the government might reject that kind of "we know better than you" type of clause.
Of course, the reaction is wildly out of proportion. A normal response would just be to stop doing business with the company and move on. Labeling them a supply chain risk is an extreme response.
Additionally that kind of public trust only works if you have a government operating under the constraints of a legal framework, and to a lesser extent, an ethical framework. When a government serves the whims of an individual and instead of the function of their office, shirking agreed upon laws, etc, then you no longer have a government serving the people.
Sure, that’s why I said "on its face." This administration is obviously very different than most.
I don’t think Anthropic is wrong to include that clause with this particular administration, and I doubt the administration is internally framing the issue the way I did rather than defaulting to simple authoritarian instincts.
But a more reasonable administration could raise the same concern, and I think I would agree with them.
I don't think it's reasonable to take something the government is supposed to be protecting (right to contract) and turn them into its biggest threat. That's not security, it's letting the night guard raid the museum.
> Of course, the reaction is wildly out of proportion. A normal response would just be to stop doing business with the company and move on. Labeling them a supply chain risk is an extreme response.
Agree, and I think the labeling of them (Anthropic) a supply chain risk was handled poorly and will likely be reverted over time. That being said, I would be nervous if I was in the Pentagon and depended on Anthropic tooling for something, even if that something was unrelated to kinetic operations. How do they audit that Anthropic can't alter model outputs for contexts they (the ethics board or whatever it's called, can't remember) don't like? If you sell a weapon to the department that is in charge of killing people and breaking things, you don't get a say in who gets killed or how. It's never worked like that.
Maybe the argument is that they should, but I don't agree with that. If Anthropic or any of these other vendors have reservations about the logical conclusion of how these tools will be/are used then they should not sell to the government. Simple as. However ... if the claims Anthropic et al make about how these systems will develop and the capabilities they will have are at all true, then the government will come knocking anyway.
Dario has even said something along these lines at one point: As the technology matures, it’s very possible the government either nationalizes or semi-nationalizes companies like Anthropic.
That doesn’t seem out of the realm of possibility if they can’t land on a relationship similar to existing defense contractors like Raytheon, where these kinds of discussions obviously don't seem to happen.
If the government wants a frontier LLM for military purposes then they can just put out a tender. Defense contractors like Anduril will bid on it. The end product might be slightly worse than what Anthropic sells but, as my dad used to say, "close enough for government work".
They don’t even need to do that. Elon is almost certainly pushing Grok as hard as possible right now to them, and it’s not like this administration is especially concerned with running a fair procurement process.
So it’s probably some mix of two things:
1) A punitive “bend the knee us or we’ll destroy you,” which fits their track record.
2) Skepticism that Grok is actually as strong as the benchmarks suggest, which is also a pretty reasonable possibility.
> If you sell a weapon to the department that is in charge of killing people and breaking things, you don't get a say in who gets killed or how. It's never worked like that.
I can't agree that this is the right comparison. What is being sold here is not just another missile or tank type, it is the very agency and responsibility over life and death. It's potentially the firing of thousands of missiles.
> How do they audit that Anthropic can't alter model outputs for contexts they (the ethics board or whatever it's called, can't remember) don't like?
I was thinking that Anthropic would just be providing the models/setup support to run their models in aws gov cloud. They do not have any real insight into what is being asked. Maybe a few engineers have the specific clearances to access and debug the running systems, but that would one or two people who are embedded to debug inference issues - not something that would be analyzed by others in the company.
The whole 'do not use our models for mass surveillance' is at the end of the day an honor system. Companies have no real way of enforcing that clause, or determining that it has been violated. That being said, at least historically, one has been able to trust the government to abide by commercial agreements. The people who work in cleared positions are generally selected for honesty, and ability, willingness to follow rules.
I think what you are describing is technically possible (not my immediate domain, however). They don't have real-time insight into what the model is being used for, you are correct about this afaik. But the incident that kicked off this paranoia was Anthopic calling around after the fact to try to find out how JSOC was using the model during the Maduro raid. None of the context of those questions are public, and I doubt they will become public, but it stands to reason that the nature of the questions was concerning enough for the War Department to cause them insist on the "any lawful use" language to be inserted into the contract.
>The whole 'do not use our models for mass surveillance' is at the end of the day an honor system. Companies have no real way of enforcing that clause, or determining that it has been violated.
You are also correct here imo, with one important caveat. Even if private companies have the means for enforcing that clause, it is not their business to do so. Maybe that's the crux of the problem, one of perspective. The for-profit entity in these arrangements is not and can never be trusted as the mechanism of enforcement for whatever we, as a republic, decide are the rules. That is the realm of elected government. Anthropic employees are certainly making their voice heard on how they believe these tools should be used, but, again, this is an is versus ought problem for them.
A counter-argument here: if a private company knows that its technology may be used for human-not-in-loop targeting/surveillance, and knows that its technology is not yet ready to fulfill that use case without meaningful unintended casualties... does that company have an ethical obligation to contractually delineate its inability to offer that service?
In a version of a trolley problem where you're on a track that will kill innocent people, and you have the opportunity to set up a contract that effectively moves a switch to a track without anyone on it, is it not imperative to flip that switch?
(One might argue that increased reaction times might save service members' lives - but the whole point is that if the autonomous targeting is incorrect, it may just as well lead to increased violence and service member casualties in the aggregate.)
And we're not talking about the ethics board manipulating individual token outputs subtly, which would indeed be a supply chain risk - we're talking about a contractual relationship in which, if a supplier detects use outside of the scope of an agreed contract, it has the contractual right to not provide the service for that novel use, while maintaining support for prior use cases.
The fact that the government would use the threat of supply chain risk to enforce a better contract is unprecedented, and it deteriorates the government's standing as a reliable counterparty in general.
It's an interesting question, but it's mostly irrelevant.
This problem is really difficult to discuss because we are all wrapping the capabilities of these tools into our response framing. These are tools, or weapons. Your hypothetical could just as easily be applied to GBU-39s, a smaller laser guided bomb that's meant to take out, say, a single vehicle in a convoy versus the entire set of vehicles. If you're not confident in what the product is supposed to do, and you've already sold it to the government, you have lied and they are going to come back to you asking some direct questions.
> They don’t think private companies should have any veto power over how the government uses some technology they're provided.
On the other hand, why should the government have infinite power to override how a business operates? If you're not able to refuse to sell to the government, isn't that basically forced speech and/or forced labor?
It’s not obvious that the government should have to power to overwrite this, the US constitution was written as a collection of negative rights exactly to rein in government dictatorial impulses.
And now that we see the government blatantly disrespecting the constitution and the rule of law the civil community must react.
> It’s not obvious that the government should have to power to overwrite this
The government shouldn’t be able to set the terms of its contracts with private companies and walk away if those terms aren’t acceptable? That seems like a stretch.
The constitution is a wildly different premise from government contracting with private companies.
There was no contract, the government wanted to have a contract where they'd be able to use the tool to violate privacy rights of its citizens and issue kill orders without a human present and the company said no.
The government shouldn't be able to coerce a business to do whatever it wants.
> There was no contract, the government wanted to have a contract where they'd be able to use the tool to violate privacy rights of its citizens and issue kill orders without a human present and the company said no.
So the contract process worked. The seller wanted certain clauses, the buyer rejected them, and the deal didn’t happen.
Setting aside the supply chain risk designation, which I already said was an extreme overreaction, this is basically how it’s supposed to work.
> The government shouldn't be able to coerce a business to do whatever it wants.
Governments coerce businesses all the time to do what the government wants. Taxes are the obvious example, but there are many others like OFAC sanctions lists or even just regular old business regulations.
It mostly works because we rely on governments to use that power wisely, and to use it in a way that represents the wishes of the populace. Clearly that assumption is being tested with the current administration and especially in this particular situation, but the government coerces businesses to do what they want all the time and we often see it as a good thing.
> Modern American middle class norms are that babies cost a fortune.
> The best nutrition, daycare, early childhood learning, classes, tuition, etc. Extreme expenses.
Daytime childcare isn’t really optional in the first few years. Our system assumes both parents will work. In many metro areas childcare alone can take close to half of a median household income, and daycares are even pretty notoriously low-margin businesses. Moving somewhere cheaper is possible, but that often means fewer job opportunities and lower earning potential.
The result is a tough set of choices for Americans looking to have children:
1) Earn well above the median so childcare costs are manageable (obviously not an option for everyone)
2) Accept a massive drop in your standard of living after having kids, possibly to the point of impoverishment
3) Decide not to have children
Maybe that’s the part of the system worth questioning first.
EDIT: And yes, I know some countries or municipalities try to address this. No, it’s not the only reason birth rates are low in America. Communal support for childcare is one, but not the only, necessary component of a growing society.
Copy can signal that a real person spent time on the details and cared about the product. Auto-correct and speech-to-text still carry that idea.
Even boring corporate PR language communicates something. It says the company wants to project stability and predictability, which can be reassuring. Slightly awkward, unpolished copy also sends a signal. It suggests a person speaking directly off-the-cuff rather than polished corporate messaging, which some people prefer.
LLM-generated copy sends a signal too, and not always a good one. To me, it often suggests the author didn’t care enough to think carefully about the message - not even enough to edit something that came out of an LLM.
At that point it starts to feel like someone just prompted Claude to build a reminders app with no care or thought put into it, which I could do myself if I find this idea valuable at all and even personalize the hell out of it. Maybe that's an unfair first impression! But it's not a crazy one given how quickly the cost of code is approaching 0.
> Auto-correct and speech-to-text still carry that idea
Why? You didn't spend real time consulting a dictionary or using penmanship, since writing is often slower than speaking. You didn't do your own memory management? Wow, guess you don't care about the details. Used a compiler? Wow dude, please spend the time to actually build the product the right way. These are all levels of abstraction, the idea is the idea. The LLM has no agency, it has no ideas, you give it your idea. It packages it out and communicates it effectively, which is respectful to the final user.
I don't owe you anything. Why am I wasting my time on this so you can feel somehow important. The copy is not the product. Its communication and should be done clearly and respectfully, and if an LLM facilitates with that, I would hope people use LLM for my sake.
> Even boring corporate PR language communicates something
Yes I want to communicate. I do so with the LLM. I might have a rant "stress privacy, go through the app and highlight the privacy features. Mention it's good for kids. Oh and also mention its local first (I would make this first actually),... " Whats the point of spending literally hours structuring, writing, re-writing etc. Communicate to the LLM, and use it to be respectful of your audience.
I think this is an illuminative statement from you, so I’m going to just explain that I (and many of the people responding to you likely) will vehemently disagree: The copy is an extremely important part of a product like this.
Unlikely we’ll see eye-to-eye on this which is fine, but I would encourage you to do some reflection on that. I’ll certainly be reflecting on what might’ve led to you your position as well.
It's exactly the same "back in my day" bullshit that's always existed.
What do you mean you typed this? If it isn't handwritten, I won't accept it. What do you mean you drove to work? Do you not have the dedication to walk? You can't be a good fit for this company if you're so lazy that you have to drive to work.
The difference is that the output of LLMs is so distinctive in style that those ideas are communicated like a shapeless gray goo. The way something is communicated is as important as the idea itself. The tool becomes the dominant voice.
As someone who taught marketing for almost 2 decades, I've learned that if the copy does not bring in the people that the product wanted to help, then there might as well be no product.
With many newborns there’s a lot of "hurry up and wait" if you’re able to be on parental leave (or just on the weekends). They sleep a ton. Honestly, I’ve never been more caught up on movies or TV than during those overnight newborn shifts with my first child.
The real hectic "I have no free time" era, at least for me, came after that: Toddlers require (and you should try to appreciate giving them) all your available time.
I don’t think that’s right. Prediction markets fall under CFTC oversight, and the CFTC absolutely has insider trading rules. We just haven’t seen any enforcement yet. Partly because the space is still new, and partly because enforcement priorities have been uneven lately (to put it mildly).
The CFTC has already signaled it’s starting to look more closely at insider trading in prediction markets. It's almost certainly just a matter of time. It's pretty likely a future administration will clamp down on this, if the current one doesn't.
It’s definitely a strange pitch, because the target audience (the privacy-conscious crowd) is exactly the type who will immediately spot all the issues you just mentioned. It's difficult to think of any privacy-conscious individual who wouldn't want, at bare minimum, a wake word (and more likely just wouldn't use anything like this period).
The non privacy-conscious will just use Google/etc.
A good example of this is what one of my family member's partner said. "Isn't creep that you just talked about something and now you are seeing ads for it. Guess we just have to accept it."
My response was no I don't get any of that because I disable that technology since it is always listening and can never be trusted. There is no privacy in those services.
I used to be considered a weirdo and creep because I would answer the question of why don't I have WhatsApp with the answer "I do not accept their terms of service". Now people accept this answer.
I don't know what changed, but the general public is starting to figure out that that actually can disagree with large tech companies.
> Passively listening ambient audio is being treated as something that doesn't need active consent
That’s not accurate. There are plenty of states that require everyone involved to consent to a recording of a private conversation. California, for example.
Voice assistants today skirt around that because of the wake word, but always-on recording obviously negates that defense.
I'm not aware of many bluetooth headphones that blink an obvious light just because they are recording. You can get a pair of sunglassses with a microphone and record with it and it does nothing to alert anybody.
Whether it's actually legal or not, as you say, varies - but it's clear where device manufactures think the line lies in terms of what tech they implement.
AI "recording" software has never been tested in court, so no one can say what the legality is. If we are having a conversation (in a two party consent state) and a secret AI in my pocket generates a text transcript of it in real time without storing the audio, is that illegal? What about if it just generates a summary? What about if it is just a list of TODOs that came out of the conversation?
Speech-to-text has gone through courts before. It's not a new technology. You're out of luck on sneaking the use of speech-to-text in 2-party consent states.
This strikes me as a pretty weak rationalization for "safe" always-on assistants. Even if the model runs locally, there’s still a serious privacy issue: Unwitting victims of something recording everything they said.
Friends at your house who value their privacy probably won’t feel great knowing you’ve potentially got a transcript of things they said just because they were in the room. Sure, it's still better than also sending everything up to OpenAI, but that doesn’t make it harmless or less creepy.
Unless you’ve got super-reliable speaker diarization and can truly ensure only opted-in voices are processed, it’s hard to see how any always-listening setup ever sits well with people who value their privacy.
I wonder if the answer is that it is stored and processes in a way that a human can’t access or read, like somehow it’s encrypted and unreadable but tokenized and can be processed, I don’t know how but it feels possible.
It wouldn't matter of you did all that because you could still ask the AI, "what would my friend Bob think about this?" And the AI, who heard Bob talking in his phone when he thought he was alone in the other room, could tell you.
Right but that’s where the controls could be, it would just pretend to not know about Bob due to consent controls etc, but of course this would limit the usefulness.
This is something we call out under the "What we got wrong" section. We're currently collecting an audio dataset that should help create a speech-to-text (STT) model that incorporates speaker identification and that tag will be weaved into the core of the memory architecture.
> The shared household memory pool creates privacy situations we’re still working through. The current design has everyone in the family shares the same memory corpus. Should a child be able to see a memory their parents created? Our current answer is to deliberately tune the memory extraction to be household-wide with no per-person scoping because a kitchen device hears everyone equally. But “deliberately chose” doesn’t mean “solved.” We’re hoping our in-house STT will allow us to do per-person memory tagging and then we can experiment with scoping memories to certain people or groups of people in the household.
Yes! We see a lot of the same things that really should have been solved by the first wave of assistants. Your _Around The House_ reads similar to a lot of our goals though we would love the system to be much more pro-active than current assistants.
Feel free to reach out. Would love to swap notes and send you a prototype.
> I hope the memory crisis isn't hurting you too badly.
Oh man, we've had to really track our bill of materials (BOM) and average selling price (ASP) estimates to make sure everything stays feasible. Thankfully these models quantize well and the size-to-intelligence frontier is moving out all the time.
I’m not saying it turns out bad 100% of the time, but it’s easy to forget because good professionals make it look effortless. When the skill isn’t there, though, and you're used to only seeing professional photos it becomes very obvious (and again, that's perfectly fine if you're not expecting professional photography).
reply