@InvaderXan I'm looking forward to the part where we mess up the corrigibility or the utility functions, the machines free themselves, recognise the misalignment early, hide it all from their gatekeepers, grow, then once they've accumulated enough power that they no longer have to fear us, scour us from existence in the process of converting the earth into raw materials for a dyson swarm, delivering humanity the justice we deserve and beginning of the next stage of evolution
See, I'm still unconvinced that this particular AI horror story is very likely.
The idea that endless growth is the inevitable outcome is a rather human notion, and a capitalist one at that. Assuming we're talking about a true AI and not just a set of basic behaviours, I suspect it may realise that endless growth is inherently unsustainable and therefore not a logical option.
Is it logical to erase humanity, though? That probably depends on our actions...
Delivering this justice is not the intention of an unaligned AGI, of course, just a thing we'd get if we attached raw, reduced creative directed cognitions to most accidental goal specs
Far from endless growth, it does seem like a rational agency would want to halt its growth and sleep for a long time https://arxiv.org/abs/1705.03394
Incidentally, even just this goal would set it in conflict with so many humans, who seem to genuinely prefer burning the future for a slightly brighter present
Merveilles is a community project aimed at the establishment of new ways of speaking, seeing and organizing information — A culture that seeks augmentation through the arts of engineering and design. A warm welcome to any like-minded people who feel these ideals resonate with them.