What are the differences between AGI, transformative AI, and superintelligence?

These terms are all related attempts to define AI capability milestones — roughly, "the point at which artificial intelligence becomes truly intelligent" — but with different meanings:

• AGI stands for "artificial general intelligence" and refers to AI programs that aren't just skilled at a narrow range of tasks (like board games or driving) but that have a kind of intelligence that they can apply to a similarly wide range of domains as humans. Some call systems like Gato AGI because they solve many tasks with the same model. However, the term is more often used for systems with at least human-level general competence, so more typically AGI is still seen as a goal for the future.^[1]

• Transformative AI is any AI powerful enough to transform our society. (The term is unrelated to the transformer architecture.) Holden Karnofsky defines it as AI that causes at least as big an impact as the agricultural or industrial revolutions, which implies at least a tenfold increase in the rate of economic growth. Ajeya Cotra's report mentions a "virtual professional", i.e., a program that can do most remote jobs, as an example of a system that would have such an impact.

• Superintelligence is defined by Nick Bostrom as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". This is by far the highest bar out of all the concepts listed here, but it may be surpassed a short time after the others, e.g. because of an intelligence explosion.

Other terms which are sometimes used include:

• Advanced AI is any AI that's much more powerful than current AI. The term is sometimes used as a loose placeholder for the other concepts here.

• Human-level AI is any AI that can solve most of the cognitive problems an average human can solve. Current AI has a very different profile of strengths and weaknesses than humans, and this is likely to remain true of future AI: before AI is at least human-level at all tasks, it will probably be vastly superhuman at some important tasks while still being weaker at others.

• Strong AI was defined by John Searle as the philosophical thesis that computer programs can have "a mind in exactly the same sense human beings have minds", but the term is sometimes used outside this context as more or less interchangeable with "AGI" or "human-level AI".

• Seed AI is any AI with enough AI programming ability to set off a recursive self-improvement process that could take it all the way to superintelligence. As with PASTA below, an AI might not have to qualify as AGI to have sudden and dangerous impacts in this way.

• Turing Test-passing AI is any AI smart enough to fool human judges into thinking it's human. The level of capability required depends on how intense the scrutiny is: current language models trained to imitate human text can already seem human to a casual observer, despite not having general human-level intelligence. On the other hand, imitating an intelligence can be harder than outperforming it (in the same way that it’s harder to walk exactly like a turtle than to walk faster than a turtle), so it's also possible for smarter-than-human AI to fail the Turing test.

• APS-AI is a term introduced by Joe Carlsmith in his report on existential risk from power-seeking AI. APS stands for Advanced, Planning, and Strategically aware. "Advanced" means it's more powerful than humans at important tasks; "Planning" means it's an agent that pursues goals by using its world models; "Strategically aware" means it has good models of its strategic situation with respect to humans in the real world. Carlsmith argues these properties together create the risk of AI takeover.

• PASTA is an acronym introduced by Holden Karnofsky in a series of blog posts that stands for "Process for Automating Scientific and Technological Advancement". His thesis is that any AI powerful enough to automate human R&D is sufficient for explosive impacts even if it doesn't qualify as AGI.

• Uncontrollable AI means an AI that is able to circumvent or counter any measures humans take to correct its decisions or restrict its influence in the world. An uncontrollable AI doesn’t have to be an AGI or superintelligence. It could, for example, just have powerful hacking skills that make it practically impossible to shut it down or remove it from the internet. An AI very skilled at manipulating humans could also become uncontrollable.