Bagd Biggerd's question on Mesa-Optimizers 2

From Stampy's Wiki

The fundamental mistake you are making is that you assume these optimizers have intelligence and awareness, THEY DO NOT! They simply fit data.If the extrapolation is flawed and they destroy the world because of it, well that is just a random coincidence, it doesn't mean they have an awareness and forethought. One should be asking the question: What is the likelihood that if an AI algorithm is trained on X data that it will ultimately destroy the world when released? Well, the likelihood is clearly very low except if they are controlling nuke, politicians, or the stock market.

The main problem seems that AI can find "novel solutions" to problems that evades humans ability to predict those solutions. Like anything one must build in safety. We as humans always build things with complete disregard to the downside and hope for the best. Rather, we must learn to build things assuming the worst but this, of course, costs $$$ and mansions and it is rarely done until after the fact.

Tags: None (add tags)
Question Info
Asked by: Bagd Biggerd
OriginWhere was this question originally asked
YouTube (comment link)
On video: Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think...
Date: 2021-05-28T07:46
Asked on Discord? Yes