I've always looked at it as we're not making software that can think, we're (quite literally) demonstrating that vast categories of things don't need thought (for some quality level). The problem is, it's clearly not 100%, maybe it's 90-some percent, but it doesn't matter, we're only outsourcing the unimportant things that aren't definitional for a task.
This is very typical of naive automation, people assume that most of the work is X and by automating that we replace people, but the thing that's automated is almost never the real bottleneck. Pretty sure I saw an article here yesterday about how writing code is not the bottleneck in software development, and it holds everywhere.
The reason management thinks coding is the bottleneck is because they don't understand the first thing abiut code and neither have the ability or temprament to. Their whole professional career is about plausibly convincing other people through jargon, manipulation and popularity contests, which generally oprn up doors, solve problems and provoke seal like clapping from all involved. The idea that the core problem in many systems and software is due to their constitutonal inability to think rigorously to define requirements logically has never crossed their mind: it must be the magic spells those losers we bullied at school use and we are now tragically dependent on.
The discussion is completely useless without defining what thought is and then demostrating that LLMs are not capable of it. And I doubt any definition you come up with will be workable.
This is very typical of naive automation, people assume that most of the work is X and by automating that we replace people, but the thing that's automated is almost never the real bottleneck. Pretty sure I saw an article here yesterday about how writing code is not the bottleneck in software development, and it holds everywhere.