Will AI care about objective morality?

The question of whether AI will care about objective morality is complex and uncertain. There is no consensus on whether ‘objective morality’ exists. Some believe it exists, while others argue that morality is subjective or nonexistent. Even among those who believe in objective morality, there is no consensus on what it entails.

Today's AI systems behave in ways that depend on their learning processes, reward functions, and the data they are trained on. If there is an objective morality, it is plausible that AI might come to care about it, as it could be encoded in the data or rewarded through the learning process, or that a superintelligent AI will discover it and objective reasons to do it in the real world after deployment. It is also plausible that it would be easier to align AI to an objective morality if there is one.

However, according to the orthogonality thesis, any level of intelligence is in principle compatible with any set of values or goals. In other words, an AI system could be superintelligent and yet not necessarily align with any particular moral framework, including an objective one.