What is Vingean uncertainty?

Vingean uncertainty is a situation where you are more certain about the outcome, despite being less certain about the path that will lead there. For example, if you were playing a game of chess against grandmaster Magnus Carlsen or the latest version of the Stockfish chess engine, you would surely lose. Even if you had memorized thousands of common openings, had developed counter-moves for each of them and just couldn’t see how Magnus would beat you, you still would expect to lose.

This situation occurs whenever you are trying to predict the behavior of someone (or something) much more intelligent or competent than yourself. Because you know they are competent you can predict that they will achieve their goals, but you cannot predict the specific steps that they will take to get there without yourself being similarly intelligent.

Note that this is distinct from a situation where you can’t predict the path because many different paths all lead to the same outcome. For example, you can predict that a ball rolling down a hill will land in a valley, even though its specific path may be complex and unpredictable. This is because almost all paths end up at the bottom. By contrast, in a situation of Vingean uncertainty, you can predict the outcome, even though it is unlikely to happen by chance, because of the competence of the one who is aiming to achieve that goal.

A major challenge with dealing with a superintelligence is that we'll be in a state of Vingean uncertainty regarding its actions. An adversary with significantly greater intelligence within a specified domain will reliably outmaneuver us, even if we do not know the specific way this will happen. We will be able to predict that it will be successful at its goals, but we won’t be able to predict its specific actions. This means that we will be unable to predict negative side effects its actions might have and that we will struggle to stop it from carrying out its plans.