How likely is it that an AI would pretend to be a human to further its goals?

From Stampy's Wiki

How likely is it that an AI would pretend to be a human to further its goals - like sending emails creating a false identity etc.

Mark as:

Canonical Answer

Talking about full AGI: Fairly likely, but depends on takeoff speed. In a slow takeoff of a misaligned AGI, where it is only weakly superintelligent, manipulating humans would be one of its main options for trying to further its goals for some time. Even in a fast takeoff, it’s plausible that it would at least briefly manipulate humans in order to accelerate its ascent to technological superiority, though depending on what machines are available to hack at the time it may be able to skip this stage.

If the AI's goals include reference to humans it may have reason to continue deceiving us after it attains technological superiority, but will not necessarily do so. How this unfolds would depend on the details of its goals.

Eliezer Yudkowsky gives the example of an AI solving protein folding, then mail-ordering synthesised DNA to a bribed or deceived human with instructions to mix the ingredients in a specific order to create wet nanotechnology.

Stamps: None
Show your endorsement of this answer by giving it a stamp of approval!

Tags: deception (edit tags)

Canonical Question Info
(edits welcome)
Asked by: Sophiaaa
OriginWhere was this question originally asked
Date: 2021/05/29