Could we tell the AI to do what's morally right?
We could, but there are a number of challenges to this solution:
-
People do not agree on what is right or moral. Philosophers (and everybody else) have disagreed for a very long time on what is right or wrong, and many hold that human values are inherently complex (in the sense of "difficult or impossible to define in a succinct way").
-
It is difficult to create a well-defined concept of what is morally right in a way we can encode into an AI.[1]
-
We currently don't know how to make an AI pursue any particular goal in a safe and reliable way.
One attempt to solve what the right thing to do is is Coherent Extrapolated Volition (CEV) ↩︎