Canonical questions

From Stampy's Wiki

Canonical questions are the questions which we've checked are in scope and not duplicates, so we want answers to them. They may be edited to represent a class of question more broadly, rather than keeping all their idosyncracies. Once they're answered canonically Stampy will serve them to readers.

See also

142 canonical questions, 117 of them are answered, and 83 canonically answered!

All canonical questions

A lot of concern appears to focus on human-level or “superintelligent” AI. Is that a realistic prospect in the foreseeable future?
AIs aren’t as smart as rats, let alone humans. Isn’t it sort of early to be worrying about this kind of thing?
Are Google, OpenAI etc. aware of the risk?
Are robots the real problem? How can AI cause harm if it has no ability to directly manipulate the physical world?
Are there types of advanced AI that would be safer than others?
Aren’t there some pretty easy ways to eliminate these potential problems?
Can an AI really be smarter than humans?
Can humans stay in control of the world if human- or superhuman-level AI is developed?
Can we just tell an AI to do what we want right now, based on the desires of our non-surgically altered brains?
Can we program the superintelligence to maximize human pleasure or desire satisfaction?
Can we specify a code of rules that the AI has to follow?
Can we teach a superintelligence a moral code with machine learning?
Can we tell an AI just to figure out what we want, then do that?
Can we test a weak or human-level AI to make sure that it’s not going to do bad things after it achieves superintelligence?
Can you give an AI a goal of “minimally impact the world”?
Can you stop an advanced AI from upgrading itself?
Can’t we just program the superintelligence not to harm us?
Could an AGI already be at large?
Could we program an AI to automatically shut down if it starts doing things we don’t want it to?
Couldn’t we keep the AI in a box and never give it the ability to manipulate the external world?
Do you need a PhD to work on AI Safety?
How can I collect questions for Stampy?
How can I contact the Stampy team?
How can I contribute to Stampy?
How can I get hired by an organization working on AI alignment?
How could an intelligence explosion be useful?
How could general intelligence be programmed into a machine?
How could poorly defined goals lead to such negative outcomes?
How is AGI different from current AI?
How is ‘intelligence’ defined?
How likely is an intelligence explosion?
How likely is it that an AI would pretend to be a human to further its goals?
How long will it be until AGI is created?
How might AGI kill people?
How might a superintelligence socially manipulate humans?
How might a superintelligence technologically manipulate humans?
How might an AI achieve a seemingly beneficial goal via inappropriate means?
How might an intelligence explosion be dangerous?
How might non-agentic GPT-style AI cause an intelligence explosion or otherwise contribute to existential risk?
How quickly could an AI go from the first indications of problems to an unrecoverable disaster?
I want to work on AI alignment. How can I get funding?
I'm interested in working on AI Safety. What should I do?
If AI takes over the world how could it create and maintain the infrastructure that humans currently provide?
If superintelligence is a real risk, what do we do about it?
If we solve alignment, are we sure of a good future?
Is donating small amounts to AI safety organisations going to make a non-negligible difference?
Is expecting large returns from AI self-improvement just following an exponential trend line off a cliff?
Is humanity doomed?
Is it possible to block an AI from doing certain things on the internet?
Is it possible to code into an AI to avoid all the ways a given task could go wrong - and is it dangerous to try that?
Is the concern that autonomous AI systems could become malevolent or self aware, or develop “volition”, and turn on us? And can’t we just unplug them?
Is the focus on the existential threat of superintelligent AI diverting too much attention from more pressing debates about AI in surveillance and the battlefield and its potential effects on the economy?
Is there a danger in anthropomorphising AI’s and trying to understand them in human terms?
Isn’t AI just a tool like any other? Won’t AI just do what we tell it to do?
Isn’t it immoral to control and impose our values on AI?
I’d like a good introduction to AI alignment. Where can I find one?
I’d like to find people to talk and learn with to help me stay motivated to get involved with AI alignment. Where can I find them?
I’d like to get deeper into the AI alignment literature. Where should I look?
I’m convinced that this is important and want to contribute. What can I do to help?
Might an intelligence explosion never occur?
Once we notice that the superintelligence working on calculating digits of pi is starting to try to take over the world, can’t we turn it off, reprogram it, or otherwise correct its mistake?
People talk about "Aligning AI with human values" but which humans' values are we talking about?
Superintelligence sounds a lot like science fiction. Do people think about this in the real world?
We’re going to merge with the machines so this will never be a problem, right?
What approaches are AI alignment organizations working on?
What are alternate phrasings for?
What are brain-computer interfaces?
What are good external resources to link to when editing Stampy's Wiki?
What are low cost things people who won't become researchers can do to contribute?
What are some important terms in AI alignment?
What are the differences between AGI, TAI, and Superintelligence?
What are the different possible AI takeoff speeds?
What are the potential benefits of AI as it grows increasingly sophisticated?
What can we do to contribute to AI safety?
What can we expect the motivations of a superintelligent machine to be?
What external content would be useful to the Stampy project?
What is 'Transformative AI'?
What is AGI and what will it look like?
What is AI Safety?
What is AI alignment?
What is Codex / Github Copilot?
What is Coherent Extrapolated Volition?
What is GPT-3?
What is MIRI’s mission?
What is Stampy's scope?
What is a duplicate question on Stampy's Wiki?
What is a follow-up question on Stampy's Wiki?
What is a quantilizer?
What is biological cognitive enhancement?
What is greater-than-human intelligence?
What is meant by AI takeoff?
What is narrow AI?
What is superintelligence?
What is the Control Problem?
What is the Stampy project?
What is the general nature of the concern about AI safety?
What is the intelligence explosion?
What is the orthogonality thesis?
What is whole brain emulation?
What organizations are working on AI existential safety?
What should I read to learn about decision theory?
What should be marked as a canonical answer on Stampy's Wiki?
What technical problems are MIRI working on?
What would an actually good solution to the control problem look like?
What's especially worrisome about autonomous weapons?
When should I stamp an answer?
When will an intelligence explosion happen?
When will transformative AI be created?
Where can I learn about interpretability?
Who is Professor Nick Bostrom?
Who is Stampy?
Why can't we just make a child AI and raise it?
Why can't we simply stop developing AI?
Why can't we turn the computers off?
Why can’t we just use Asimov’s 3 laws of robotics?
Why can’t we just use natural language instructions?
Why can’t we just “put the AI in a box” so it can’t influence the outside world?
Why can’t we just…
Why does AI need goals in the first place? Can’t it be intelligent without any agenda?
Why does takeoff speed matter?
Why don't we just not build AGI if it's so dangerous?
Why is AGI dangerous?
Why is AI Safety hard?
Why is AI Safety important?
Why is safety important for smarter-than-human AI?
Why is the future of AI suddenly in the news? What has changed?
Why might a fast takeoff be dangerous?
Why might an AI do something that we don’t want it to, if it’s really so intelligent?
Why might people try to build AGI rather than stronger and stronger narrow AIs?
Why might we expect a fast takeoff?
Why might we expect a moderate AI takeoff?
Why might we expect a superintelligence to be hostile by default?
Why should I worry about superintelligence?
Why should we prepare for human-level AI technology now rather than decades down the line when it’s closer?
Why think that AI can outperform humans?
Why work on AI safety early?
Why would great intelligence produce great power?
Won’t AI be just like us?
Would it improve the safety of quantilizers to cut of the top few % of the distribution?
Wouldn't a superintelligence be smart enough not to make silly mistakes in its comprehension of our instructions?
Wouldn't it be safer to only build narrow AIs?
Wouldn’t it be intelligent enough to know right from wrong?