Many of us remember that OpenAI was also started by people with strong personal values. Their charter said that they would not monetize after reaching AGI, their fiduciary duty is to humanity, and the non-profit board would curtail the ambitions of the for-profit incentives. Was this not also believed by a sizeable portion of the employees there at the time? And what is left of these values after the financial incentives grew?
The market forces from the huge economic upside of AI devalues individual values in two ways. It rewards those that choose whatever accelerates AI the most over any individuals who are more careful and act on individual values--the latter simply loses power in the long run until their virtue has no influence. As Anthropic says in their mission statements, it is not of much use to humanity to be virtuous if you are irrelevant. The latter, as is true for many technologies, is that economic prosperity is deeply linked to human welfare. And slowing or limiting progress leads to real immediate harm to the human population. And thus any government regulations which are against AI progress will always be unpopular, because those values which are arguing future harm of AIs is fighting against the values of saving people from diseases and starvation today.
If you have more money than god, you no longer get to play the "I didn't know" game. You have the resources. If you don't know, you made a choice to not know.
Chinese models are developed by Chinese corporate. they are free and open weight because they are the underdog atm. they are not here for fun, they are here to compete.
The competition is good though, it will push down the prices for all of us. At some point being behind 5% won’t have much practical difference. Most people won’t even notice it.
I will gladly switch to that one if their CEO is less of sociopath than Altman and god forbid Amodei. In fact I use some of the new Chinese models at home and compared to Opus 4.6 AGI, the difference is getting less. Codex 5.3 xhigh is already better than opus anyway.
Neither of these things are useful signals. Other labs surely trained on similar material (presumably not even buying hard copies). Also how "bothered" someone is about their predictions is a bad indicator -- the prediction, taken at face value, is supposed to be trying to ask people to prepare for what he cannot stop if he wanted to.
None of this means I am a huge fan of Dario - I think he has over-idealization of the implementation of democratic ideals in western countries and is unhealthily obsessed with US "winning" over China based on this. But I don't like the reasons you listed.
Re: But they aren't alive, they don't live in the world and have experiences, and they can't create something truly new.
Is it possible for a character in a novel to have novel experiences? Or for you to experience a novel dream? I would argue yes. You can know the rules of the environment and the starting conditions, but with a bit of randomness (or not) you can generate from that novel experiences which were unexpected - so too from the data & distribution that AIs are already trained on they can experience new experiences.
Another source of novelty is from good verifiers/recognition of a class of object which is hard to construct but easy to verify - here the AI can search and from that obtain novel solutions which were unthought of before.
N.B novelty itself is basically trivial - just generate random strings. But both of the above are mechanisms to generate novel samples inside some constraint of "meaningfulness"
Once the majority of work at a company can be done by AI, Anthropic has an alternative revenue stream to selling AIs to that company--directly competing with that company with a completely integrated AI system. There's of course many barriers to entry/various advantages of incumbents--but it's possible to see a world in which the company selling the AI has a huge advantage too.
Well I do think there's some degree of unsafeness which is inexorably linked to capability--if the model when deployed with full control of a machine is capable of large scale cyberattacks and blackmailing for example.
There's a lot of value in the implementation of many strong and fast algeorithms in computer algebra in proprietary tools such as Maple, Wolfram, Matlab. However, I (though of course believe that such work needs to be compensated) find it against the spirit of science to keep them from the general public. I think it would be good service to use AI tools to bring open source alternatives like sympy and sage and macaulay to par. There's really A LOT of cool algorithms missing (most familiar to me are some in computational algebraic geometry)
Additionally I think because of how esoteric some algorithms are, they are not always implemented in the most efficient way for today's computers. It would be really nice to have better software written by strong software engineers who also understands the maths for mathematicians. I hope to see an application of AI here to bring more SoTA tools to mathematicians--I think it is much more value than formalization brings to be completely honest.
> against the spirit of science to keep them from the general public
Within science, participants have always published descriptions of methodology and results for review and replication. Within the same science, participants have never made access to laboratories free for everyone. You get blueprints for how to build a lab and what to do in it, you don't get the building.
Same for computation. I'm fairly sure almost all (if not all) algorithms in these suites are documented somewhere and you can implement them if you want. No one is restricting you from the knowledge. You just don't get the implementation for free.
Generally I agree up until now where we appear to be treading on the threshold of AI being orders of magnitude more powerful. Given that, which has potential to displace large swaths of the labor force, I feel as though society deserves a larger return on investment.
Software is fundamentally different than lab equipment, just like PDFs are not paper journals that have to be printed, stored, and shipped. Most things in the digital domain have to be treated in a post-scarcity mindset, because they essentially are.
This is why the incoming generation of AI engineers organizing autonomously and openly on git etc will decimate the dusty locked away AI academia generation.
The concept of heavy gatekeeping and attribution chasing seems asinine as knowledge generation and sharing isn't metered.
I would say almost exactly the opposite is happening. Academia generally publishes it's results relatively freely but academic AI research is largely being left in the dust by large corporations who do not find it in their interest to publicly describe the "magic dust" that makes their products work.
> Same for computation....You just don't get the implementation for free.
software packages arent computation... whilst software takes time and effort (and money) to make, the finished product is virtually free to store and distribute. i see it similarly against the spirit of science. how is there more free software in the laymen space?
Society already funds a lot of scientific research. Some of that funding currently goes to private pockets like Wolfram Research, who license out their proprietary tech under expensive and highly limiting licenses (they're licensed per CPU core, Oracle style), so that scientists can do scientific computing.
As a former Mathematica user, a good part of the core functionality is great and ahead of open source, the rest and especially a lot of me-too functionality added over the years is mediocre at best and beaten by open source, while the ecosystem around it is basically nonexistent thanks to the closed nature, so anything not blessed by Wolfram Research is painful. In open source, say Python, people constantly try to outdo each other in performance, DX, etc.; and whatever you need there's likely one or more libraries for it, which you can inspect to decide for yourself or even extend yourself. With Wolfram, you get what you get in the form of binary blobs.
I would love to see institutions pooling resources to advance open source scientific computing, so that it finally crosses the threshold of open and better (from the current open and sometimes better).
Isn't plugging Wolfram algorithms into LLMs basically their current solution for the DX problem?
As far as society funding research, while I'm quite sympathetic to this view, Wolfram also puts in a significant amount of private dollars into the operationalization of their systems. My guess is there's a whole range of algorithms that aren't prominent enough to publish a paper on nor economically lucrative enough to build a company on that Wolfram products sell.
That said I do think LLM coding agents offer a great way forward to implement more papers on a FOSS manner.
Academic institutions have internal IP scouting monitoring every lab for monetizable research.
On top of that, and often competing with the former, professors are constantly exploring (heavily subsidized with public grants and staffed with free grad students) spin-offs to funnel any commercial potential of their research into their own or their buddie's pockets. It's just like in politics with revolving doors and plushy 'speaking engagements' or 'board seats' galore.
It’s hard to distribute equipment, food and shelter at zero marginal cost. It’s easy to distribute software at zero marginal cost. So let’s start there.
No one is stopping you. Build it, then distribute it. You will find that as long as people need to pay for their living, there is no post-scarcity world in any domain, especially not the digital one.
Academic institutions already pay salaries whether they fund open source development or Wolfram Research, so not sure what you’re trying to argue. People haven’t been starving while doing research in the open.
But academic institutions didn’t produce Mathematica. The point is simple: a lot of useful software like Mathematica has not been open-source for a reason. It is not about distribution being free; it is about production being expensive.
Some form of Mathematica has been open for a long time, in the form of Sage etc. There's no reason why academic institutions can't pool more money to develop these further.
Sage is a nice effort, but it isn't Mathematica. I use Mathematica, I don't use Sage. Academic institutions had almost 40 years to "pool more money", yet there is nothing that rivals Mathematica. The reason is simple: open-source doesn't pay well enough.
What does this have to do with anything? We as a culture decided that science is worthwhile, and that it's worth funding it with public money, which I personally strongly support. With that in mind, I want us to continue contributing to making scientific research and the benefits that it provides to be disseminated freely, while also paying good scientists with actual dollars that they could spend in restaurants.
Individuals and small groups make decisions in their own interest. The same is not true of society. That’s the issue that the GP is asking you to respond to
I suppose I might not be understanding your and the GP's intent correctly, but I thought that the question was based on the following sentences:
> I think it would be good service to use AI tools to bring open source alternatives like sympy and sage and macaulay to par.
> It would be really nice to have better software written by strong software engineers who also understands the maths for mathematicians.
And my response is that I think that this sort of work, which is in the public scientific interest should be funded by tax money, and the results distributed under libre licenses.
Nobody is saying "don't pay the developers". Some of us advocate for "pay the developers to develop free and open source software". Rent-seeking is not good for society.
>We as a culture decided that science is worthwhile, and that it's worth funding it with public money, which I personally strongly support.
what country are you in, and what percentage of the public purse goes to funding science? In the U.S about 11%, and with that number I often read articles, linked to from this site, about U.S Scientists quitting and going into private sector work or other non-scientific fields to get adequate compensation.
>while also paying good scientists with actual dollars that they could spend in restaurants.
see, my admittedly vague understanding of how things are structured tells me this part isn't what is happening.
I think the CBPP maybe underplays research under different organizations, for example is DARPA under DOD or is it under science and education? If under DOD then can probably increase the percent by another .5 from DARPA, and so forth with other organizations.
However, I am certainly fine with taking your stats since that just underlines they point I made and evidently got downvoted for, that the U.S does not pay for scientific research at a level where one can blithely assert that it is something considered important by the government.
I think the current generation of tools have a long way to go before I trust any numerical algorithm they implement, based on our recent experiments trying to make it implement some linear algebra by calling LAPACK. When we asked it to write some sparse linear algebra code based on some more obscure graph algorithms it produced some ugly stepchild of dijkstra's algorithm instead, which needless to say did not achieve the desired aim.
Computer algebra of the Mathematica/Maple variety is not formally rigorous: it will get things wrong due to conflating function domains, choices of branch cuts for 'multi-valued functions' and other assumptions that are required for correct results but not exposed or verified. The work of providing "strong and fast algorithms" that are comprehensively described ought to be done as part of building proof systems for the underlying mathematics that will ensure correctness.
That's the main flaw in open source. Yes, its a great idea, but why am I working a real job to eat, and spending nights and weekends on a project just as a hobby.
Science doesn't progress very fast using the 'hobby' model of funding. Unless you are rich, and it is a hobby, much like Wolfram Alpha was. He wanted to play with math/physics stuff and was rich enough to self fund.
But science does progress on the free sharing of information. Academics get paid to produce stuff that's free for everyone all the time.
No one is contesting that people who build these libraries should be compensated.
The argument is that if more scientific tools and knowledge are freely (or cheaply) available you lower the barrier to entry to experiment and play with those tools/concepts, which means more people will, which means you'll get more output. How many billion dollar companies are built on software that is open source? All of them have it somewhere in their stack whether they know it or not.
In science, it is the government that funds a lot of research. Specifically because the free market does fail at this.
A lot of tech success is built on top of government funding. In this analogy, the funding for people to eat while producing the free stuff for others to found tech startups upon.
Many of us remember that OpenAI was also started by people with strong personal values. Their charter said that they would not monetize after reaching AGI, their fiduciary duty is to humanity, and the non-profit board would curtail the ambitions of the for-profit incentives. Was this not also believed by a sizeable portion of the employees there at the time? And what is left of these values after the financial incentives grew?
The market forces from the huge economic upside of AI devalues individual values in two ways. It rewards those that choose whatever accelerates AI the most over any individuals who are more careful and act on individual values--the latter simply loses power in the long run until their virtue has no influence. As Anthropic says in their mission statements, it is not of much use to humanity to be virtuous if you are irrelevant. The latter, as is true for many technologies, is that economic prosperity is deeply linked to human welfare. And slowing or limiting progress leads to real immediate harm to the human population. And thus any government regulations which are against AI progress will always be unpopular, because those values which are arguing future harm of AIs is fighting against the values of saving people from diseases and starvation today.
reply