What could a superintelligent AI do, and what would be physically impossible even for it?
It's very hard to know the answer to this in advance because a superintelligent AI is capable of doing anything that it understands how to do. That sounds obvious, but it has a hidden nuance there - it knows how to do things that we don't know how to do, and so we cannot predict in advance what it will do because it will do some things that we didn't even know were possible to do. Almost by definition, that makes it impossible to reliably predict.
We can say that it will not be able to do things which are actually not possible to do, and it seems likely that certain rules are always followed (energy must be conserved, global entropy must always increase etc.) but the problem is that we have been wrong about the laws of physics before (see Newton's laws vs relativity and quantum mechanics) and so we're only *mostly* certain that these things are true. If there's any discrepancy between how we think physics works and how physics actually works (and most likely, such a discrepancy does exist somewhere), we can expect a superintelligent AI to be able to exploit that discrepancy in order to serve its goals in lots of weird ways we couldn't possibly have predicted.
We may be able to say with 99% certainty that a superintelligent AI will not be able to violate conservation of energy, but we're not 100% sure, and we can't be 100% sure of anything here.
|Asked by:||Nico Hill2|
OriginWhere was this question originally asked