I think we are in largely uncharted territory here, especially given the implications. Is Anthropic's approach optimal? Probably not. But given the stakes involved, gating access seems like a reasonable place to start.
I'm curious about how gated access actually holds over time, especially given that historically with dual-use capabilities containment tends to erode, whether through leaks, independent rediscovery, or gradual normalization of access.
In case it helps, we open-sourced a tool to audit dependencies for this kind of supply-chain issue. The motivation was that there is a real gap between classic “known vulnerability” scanning and packages whose behavior has simply turned suspicious or malicious. We also use AI to analyze code and dependency changes for more novel or generic malicious behavior that traditional scanners often miss.
I think that concern is valid in general terms, but it’s not clear to me that it applies here.
The goal here seems to be removing low-value output; e.g., sycophancy, prompt restatement, formatting noise, etc., which is different than suppressing useful reasoning. In that case shorter outputs do not necessarily mean worse answers.
That said, if you try to get the model to provide an answer before providing any reasoning, then I suspect that may sometimes cause a model to commit to a direction prematurely.
> Answer is always line 1. Reasoning comes after, never before.
> No explaining what you are about to do. Just do it.
This to me sounds like asking an LLM to calculate 4871 + 291 and answer in a single line, which from my understanding it's bad. But I haven't tested his prompt so it might work. That's why I said be aware of this behavior.
Have had similar issues with costs sometimes being all over the map. I suspect that the major providers will figure this out as it’s an important consideration in the enterprise setting
My sense is that the asymmetry is non-trivial issue here. In particular, a threat actor needs one working path, defenders need to close all of them. In practice, patching velocity is bounded by release cycles, QA issues / regression risk, and a potentially large number of codebases that need to be looked at.
My sense is that if a threat actor were able to build a quantum computer to the scale of being able to compromise public-key primitives based on the difficulty of integer factorization and discrete logarithms under the key sizes used in practice today, one of the highest-valued targets will be Bitcoin.
Q-day estimates are sensitive to several factors; e.g., hardware qubit counts, error correction overhead, and algorithmic efficiency (e.g., better factoring approaches could compress the timeline meaningfully without any hardware breakthrough).
Migration complexity side is also not straightforward. Cryptographic primitives tend to be deeply embedded in ways that are not always easy to find. FWIW, we built a free scanning tool for developers to find and remediate cryptographic vulnerabilities in their repos (still in beta: https://app.threatpoint.com).
2029 might be conservative or optimistic depending on which variable moves first.
I wonder if there will be a different phenomena — namely everyone just developing their own personal version of what they want rather than relying on what someone else built. Nowadays, if the core functionality is straightforward enough, I find that I just end up building it myself so I can tailor it to my exact needs. It takes less time than trying to understand and adapt someone else’s code base, especially if it’s (mostly) AI generated and contains a great deal of code slop.
My prompts to Claude has evolved from "what program / data source do I need to do this" to "what program / data source do I need, to make you do this for me".
After a few iterations, any data source without a free API feed, or any program without a free CLI interface are edited out of the gene pool, so to speak.
Nice weekend project! Even though there are copious resources out there (textbooks, videos, etc.), those may not appeal to everyone. People have different preferred modalities for consuming information and there is always value in (correctly) reframing concepts in a way that can be better understood by people who don’t resonate with traditional textbooks and YouTube videos. I’m
glad you found a formulation that works for you, and judging by the number of upvotes, it resonated with others as well. At the very least, I’m sure that working on this improved your understanding as well!
I'm curious about how gated access actually holds over time, especially given that historically with dual-use capabilities containment tends to erode, whether through leaks, independent rediscovery, or gradual normalization of access.
reply