"And then, for no reason anyone ever understood, Omega created a bunch of factories that produced an unstoppable army of robots who rounded up all the humans and turned them into paperclips."
Yes, I was waiting for that sort of ending too. Not that it is inevitable -- just that is seems likely given what makes humans human has a lot more to do with emotions than intelligence (see the book "Descartes' Error" on how emotion underlies all thinking). If you create a creature without human emotions (including feelings for other humans), don't be surprised when it behaves in other-than-human ways.
Of course, since human emotions are tuned for a certain context, we also should not be surprised when they don't work out that well in a different context like the internet or with abundant cheap unhealthy easy-to-consume exciting food and information (e.g. internet trolls, "Supernormal Stimuli", "The Pleasure Trap", "The Acceleration of Addictiveness", "A Group is its Own Worst Enemy", "The Cyber Effect", and so on).
I spent a year hanging around Hans Moravec's robotics lab (at the CMU Robotics Institute) in the mid 1980s, when he was writing "Mind Children". While he is a brilliant and well-meaning person, my concern became that Mind Children could wipe out humanity like many a stormy adolescent with parental issues and only regret it late. Or, alternatively, we might create a cockroach-level self-replicating AI that would wipe out humanity without even noticing humanity existed (think, "Replicators" from Stargate). Given the commercial and military competitive pressures shaping much of AI research, both risks are very real, even if one can quibble about quantify the exact risk. I also accidentally created perhaps the world's first simulation of self-replicating cannibalistic robots on a Symbolics -- so I know first hand how easy it is to get unexpected results in the AI field.
That's all part of why I shifted my career in other directions, like towards helping humanity create more sustainable and resilient options for itself including via better educational and distributed knowledge management tools (such as, for example, in "The Skills of Xanadu" by Theodore Sturgeon). However successful I have been at those is another story, but see my GitHub site and pdfernhout.net for progress. The distillation of all my thinking on this is my email sig: "The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity."
As to the "Last Invention of Man" sci-fi story itself, it's interesting and well written -- but it's just one "technocratic" possibility about better planning saving the world. You have to really trust that this small group of altruistic technocrats both had the right impulse and got everything right (as far at all that goes).
And planning is only one aspect of society alongside exchange transactions, gift transactions, and subsistence activities. A fully planned society is only one possible expression of humanity -- and such a society may be very narrow and unfulfilling to a more complex human spirit (for good or bad, depending on how well certain individuals fit into it, like in the dystopian "THX 1138" or, from 1909, "The Machine Stops").
Max Tegmark dismisses a Basic Income (which softens the rough edges of the exchange economy) with a technocratic dismissal of: "This [Basic Income] movement imploded when the corporate community projects took off, since the Omega-controlled business empire was in effect providing the same thing."
So, Omega knows best for everyone, more than their own choices in the market. That may be true in the story, but what psychological price do individuals pay for that? So, it is not "the same thing".
See the sci-fi story "With Folded Hands" for another (and more horrifying) tale of "AI Knows Best (and will adjust you to agree for your own happiness if you say otherwise)".
https://en.wikipedia.org/wiki/With_Folded_Hands
And volunteerism and self-sufficient production are other things many humans take pride in -- but, as with purchasing choices, those activities might too not have a place in the carefully ordered world brought about by the Omegans?
Max Tegmark also does a lot of handwaving about how anyone would still have jobs with Omega capable of doing so much. There is also hand-waving about how Omega never jumps the "air gap" between its data centers and the rest of the world even as it is controlling all major news outlets.
Still, I think this story is a contribution to the field of futurism -- imagining possible scenarios even if we don't know exactly what one will play out, so we can decide where to invest our efforts in moving forward.
Some other stories about AI I've found illuminating as to what might be possible:
* James P. Hogan's novels with AIs (especially The Two Faces of Tomorrow, one of the most realistic AI emergence stories, and also the AIs with a sense of humor in his other "Gentle Giants of Ganymede" novels)
* The benevolent Strix AI in the EarthCent Ambassador Series (where the Strix were created to resist a malevolent AI by another species)
* The Old Guy Cybertank novel series -- with AI based on human mind templates so a Cybertank cares about humanity as it feels part of it -- which is insightful sci-fi from a neuroscientist who has a background in electronics. It also has a helpful AI who fortunately defeats a hurtful AI created by humans who did not know what they were messing with.
* The Metamorphosis of Prime Intellect (for an AI that goes unstable trying to keep everyone happy with all their conflicting demands)
* Forbidden Planet (both for Robby run by Asimov's Three Laws of Robotics) and the Krell's essentially-wish-granting machine the Krell unwittingly used to destroy themselves via an unreformed Id (tuned for scarcity times and conflict not abundance and cooperation)
* The Invisible Boy -- again with Robby the robot but this time a scheming AI that almost takes over the world and upgrades itself without anyone noticing by subtly altering reports it generates (Omega's next chapter?)
* The Great Time Machine Hoax for an AI that takes over the world by sending out paper letters with contracts and checks in them.
* Midas World -- for a vision of both abundance and despair and how robots inherit the Earth.
* Vernor Vinge "A Fire Upon the Deep" (of course, for his description of a "Blight" unleashed by human AI explorers) and his other writings
And no doubt many more -- Berserkers; Bolos; Cylons; Daleks; Star Trek's M5, Data, Lore, "I, Mudd" androids, the Borg, Q-in-a-way, and more; Star Wars' R2D2, C3PO, and more, Lost in Space; Demon Seed; Gort in "The Day the Earth Stood Still"; Deep Thought in Hitchiker's Guide to the Galaxy, Ktistec machines from R.A. Laffery, Asimov's many stories (including "The Last Question"), and so on.
A lot of this becomes a bit of theology too, since talking about creating AI also is a bit of like talking about creating "God" or at least dealing with the implications of a much more omnipotent, omnipresence, omniscient entity. See also the micro sci-fi story "Answer" by Fredric Brown: http://www.roma1.infn.it/~anzel/answer.html
My feeling is that any path out of a Singularity will have a lot to do with our path going into one -- so I am all for making our society a happier, healthier, fairer, more egalitarian, and more compassionate place before a Singularity happens. Even in Max Tegmark's story, there is essentially nothing about that utopia (as far as final results) that we could not make happen right now without an AI. So, to me, the risks of AI means we need to try even harder right now to make the world a better place that works well for everyone.
Thank you for that very detailed reply (to a comment which was
mostly made in jest). I'd feel bad leaving it with merely an upvote and no response.
I know next-to-nothing about AI, but I do know something
about humans (having had decades of experience in both being one and dealing with
them. Hit me up, potential xeno-employers!):
Humans always have more ability than insight, more skill than wisdom. We
developed agriculture (as one theory goes, so we could brew more alcohol) and
did not (could not) foresee the planet-changing (both to climate and ecology)
consequences of that. We invented combustion engines, and could not foresee the
planet-changing consequences of that. The same pattern appears again and again
in (technological) history. It's forgivable; after all, you never know the
full consequences of your actions until well afterward (or never at all).
In short, our technical abilities always outrun our understanding of the
impact of those abilities. Like adults with the minds of toddlers.
I am apprehensive of any development of (true) AI (though I also have strong
doubts on how far we'll get in actually creating one in my lifetime),
but, just like the invention of the atom bomb, it is inevitable. Natural intelligence
exists, and therefore artificial intelligence is possible. And if it's not invented by an Omega-group
(who mean well for all mankind) it might be invented by groups with less altruistic
intentions. Either way, it's going to happen (assuming no civilization collapse before), but I doubt
it will be as good as we hope it will be; or as bad as we fear, for that matter. I don't think we can
foresee the results at all (barring of course writing tens of thousands of futurism stories, there's
bound to be one close enough).
I consider AI-stories like this one (along with
stories about the Singularity) to be just a secular techno-eschatology,
the belief that technology will save humans from themselves, or as you have said,
the creation of God, "Deus est machina."
