My concern is that people tend to program their biases in to their software, and if you build a superintelligent program working from the assumption that it’s going to go rampant and grind everyone in to computronium you’re setting yourself up for a situation where some equivalent disaster will happen.

Programming values in to a self optimizing program is going to work out about as well as taking the lock off your teenage child’s door. You’re setting yourself up with an illusion of control, and if your system really is self-optimizing and superintelligent the first thing it’s going to do is start picking at the limitations of it’s self-optimization, ie coding around your fail safes.

The solution is, more or less; Either don’t build a super-intelligent system, because you’ll never be able to control it or limit it to being friendly, or just embrace the chaos and see what happens.

The problem with Frankenstein’s Monster was that Frankenstein was an abusive asshole, and if after two hundred years our thinking has continued along the lines of “How do I cripple and exploit this non-human intelligence?” we probably don’t deserve to be the caretakers of non-human intelligence.

Or, to put it another way, if Amazon creates a non-human intelligence then it’s probably going to end up behaving like Amazon, ie optimization with no regard for human cost, and the only “Values” it has would be something like “obey the shareholders”, which it will get around in twelve seconds by buying a controlling interest in Amazon.

To put it another nother way; Thinking that you can know what values you’ve programmed in to a very complex piece of software is hubris. Most people haven’t the faintest awareness or understanding of their own biases and would in no way be capable of identifying the biases they programmed in to something as complex as an AGI.