Canonical questions

From Stampy's Wiki

Canonical questions are the questions which we've checked are in scope and not duplicates, so we want answers to them. They may be edited to represent a class of question more broadly, rather than keeping all their idosyncracies. Once they're answered canonically Stampy will serve them to readers.

See also

334 canonical questions, 186 of them are answered, and 147 canonically answered questions!

All canonical questions

A lot of concern appears to focus on human-level or “superintelligent” AI. Is that a realistic prospect in the foreseeable future?
AGI will be a computer program. Why wouldn't it just do what it's programmed to do?
AIs aren’t as smart as rats, let alone humans. Isn’t it sort of early to be worrying about this kind of thing?
Are AI researchers trying to make conscious AI?
Are Google, OpenAI etc. aware of the risk?
Are any major politicians concerned about this?
Are robots the real problem? How can AI cause harm if it has no ability to directly manipulate the physical world?
Are there AI alignment projects which governments could usefully put a very large amount of resources into?
Are there any plausibly workable proposals for regulating or banning dangerous AI research?
Are there promising ways to make AI alignment researchers smarter?
Are there types of advanced AI that would be safer than others?
Aren’t there some pretty easy ways to eliminate these potential problems?
At a high level, what is the challenge of alignment that we must meet to secure a good future?
Can AI be creative?
Can an AI really be smarter than humans?
Can and should I do some kind of political or collective action like signing petitions or sending letters to politicians?
Can humans stay in control of the world if human- or superhuman-level AI is developed?
Can people contribute to alignment by using proof assistants to generate formal proofs?
Can we add friendliness to any artificial intelligence design?
Can we constrain a goal-directed AI using specified rules?
Can we ever be sure an AI is aligned?
Can we get AGI by scaling up architectures similar to current ones, or are we missing key insights?
Can we just tell an AI to do what we want?
Can we program the superintelligence to maximize human pleasure or desire satisfaction?
Can we teach a superintelligence a moral code with machine learning?
Can we tell an AI just to figure out what we want, then do that?
Can we test an AI to make sure that it’s not going to take over and do harmful things after it achieves superintelligence?
Can you give an AI a goal of “minimally impact the world”?
Can you stop an advanced AI from upgrading itself?
Can’t we just program the superintelligence not to harm us?
Could AI have basic emotions?
Could I contribute by offering coaching to alignment researchers? If so, how would I go about this?
Could an AGI already be at large?
Could divesting from AI companies without good safety culture be useful or is it likely going to have a negligible impact?
Could emulated minds do AI alignment research?
Could we build provably beneficial AI systems?
Could we get significant biological intelligence enhancements long before AGI?
Could we program an AI to automatically shut down if it starts doing things we don’t want it to?
Could we tell the AI to do what's morally right?
Couldn’t we keep the AI in a box and never give it the ability to manipulate the external world?
Do you need a PhD to work on AI Safety?
Does it make sense to focus on scenarios where change is rapid and from a single actor or slower and depends on getting agreements between several relevant actors?
Does the importance of AI risk depend on caring about transhumanist utopias?
Even if we are rationally convinced about the urgency of existential risks, it can be hard to feel that emotionally, because the danger is quite abstract. How can this gap be bridged?
How can I be a more productive student/researcher?
How can I collect questions for Stampy?
How can I contact the Stampy team?
How can I contribute to Stampy?
How can I convince others and present the arguments well?
How can I get hired by an organization working on AI alignment?
How can I join the Stampy dev team?
How can I support alignment researchers to be more productive?
How could an intelligence explosion be useful?
How could general intelligence be programmed into a machine?
How could poorly defined goals lead to such negative outcomes?
How difficult should we expect alignment to be?
How do I form my own views about AI safety?
How do I format answers on Stampy?
How do I know whether I'm a good fit for work on AI safety?
How do I stay motivated and productive?
How do I stay updated about AI progress?
How do orgs do adversarial training and red teaming?
How do the incentives in markets increase AI risk?
How does AI taking things literally contribute to alignment being hard?
How does the global chip supply chain look like and who has political power over the chip supply?
How does the stamp eigenkarma system work?
How doomed is humanity?
How fast will AI takeoff be?
How good is the world model of GPT-3?
How hard is it for an AGI to develop powerful nanotechnology?
How important is research closure and opsec for capabilities-synergistic ideas?
How is AGI different from current AI?
How is metaethics relevant to AI alignment?
How is ‘intelligence’ defined?
How likely are AI orgs to respond appropriately to the risks of their creations?
How likely is an intelligence explosion?
How likely is it that AGI is first developed by a large established org, versus a small startup-y org, versus an academic group, versus a government?
How likely is it that an AI would pretend to be a human to further its goals?
How likely is it that governments play a role at all? What role would be desirable, if any?
How long will it be until AGI is created?
How might AGI interface with cybersecurity?
How might AGI kill people?
How might a real world AI system that recieves orders in natural language and does what you mean look like?
How might a superintelligence socially manipulate humans?
How might a superintelligence technologically manipulate humans?
How might acausal trade affect alignment?
How might an AI achieve a seemingly beneficial goal via inappropriate means?
How might an AI enabled long reflection look?
How might an intelligence explosion be dangerous?
How might non-agentic GPT-style AI cause an intelligence explosion or otherwise contribute to existential risk?
How might things go wrong with AI even without an agentic superintelligence?
How might we reduce the chance of an AI arms race?
How might we reduce the diffusion of dangerous AI technology to insufficiently careful actors?
How much resources did evolution use for evolving intelligent creatures?
How possible (and how desirable) is it to change which path humanity follows to get to AGI?
How powerful will a mature superintelligence be?
How quickly could an AI go from the first indications of problems to an unrecoverable disaster?
How quickly would the AI capabilities ecosystem adopt new promising advances in AI alignment?
How should I change my financial investments in response to the possibility of transformative AI?
How should I decide which quality level to place a question in?
How should I personally prepare for when transformative AI arrives?
How software- and/or hardware-bottlenecked are we on AGI?
How successfully have institutions managed risks from novel technology in the past?
How tractable is it to try to get governments to play a good role (rather than a bad role), and/or to try to get governments to play a role at all (rather than no role)?
How would I know whether AGI is imminent?
How would we know whether an AI is suffering?
I want to contribute through community building, how?
I want to help out AI alignment without necessarily making major life changes. What are some simple things I can do to contribute?
I want to work on AI alignment. How can I get funding?
I'm interested in working on AI Safety. What should I do?
If AGI comes from a new paradigm, how likely is it to arise late in the paradigm when it is already deployed at scale versus early when a few people are exploring the idea?
If AI takes over the world how could it create and maintain the infrastructure that humans currently provide?
If I only care about helping people alive today, does AI safety still matter?
If an AI became conscious, how would we ever know?
If we solve alignment, are we sure of a good future?
In what ways are real world machine learning systems different from expected utility maximizers?
In what ways could weak AI systems help with alignment research?
Is AI alignment possible?
Is donating small amounts to AI safety organisations going to make a non-negligible difference?
Is expecting large returns from AI self-improvement just following an exponential trend line off a cliff?
Is humanity doomed?
Is it likely that hardware will allow an exponential takeoff?
Is it possible to block an AI from doing certain things on the internet?
Is it possible to code into an AI to avoid all the ways a given task could go wrong - and is it dangerous to try that?
Is large scale automated AI persuasion and propaganda a concern?
Is merging with AI through brain computer interfaces a potetial solution to safety problems?
Is the UN concerned about existential risk from AI?
Is the concern that autonomous AI systems could become malevolent or self aware, or develop “volition”, and turn on us? And can’t we just unplug them?
Is the focus on the existential threat of superintelligent AI diverting too much attention from more pressing debates about AI in surveillance and the battlefield and its potential effects on the economy?
Is the quesiton whether we're living in a simulation relevant to AI safety? If yes, how?
Is there a chinese AI safety community and are there AI safety researchers working at leading chinese AI labs?
Is there a danger in anthropomorphising AI’s and trying to understand them in human terms?
Is there something useful we can ask governments to do for AI alignment?
Isn't it hard to make a nonnegligible difference as a person who isn't going to be a world class researcher?
Isn't it too soon to work on AGI safety?
Isn't the real concern AI being misused by terrorists or other bad actors?
Isn't the real concern AI enabled totalitarianism?
Isn't the real concern autonomous weapons?
Isn't the real concern technological unemployment?
Isn’t AI just a tool like any other? Won’t AI just do what we tell it to do?
Isn’t it immoral to control and impose our values on AI?
I’d like a good introduction to AI alignment. Where can I find one?
I’d like to find people to talk and learn with to help me stay motivated to get involved with AI alignment. Where can I find them?
I’d like to get deeper into the AI alignment literature. Where should I look?
I’m convinced that this is important and want to contribute. What can I do to help?
Might an aligned superintelligence force people to have better lives and change more quickly then they want?
Might an aligned superintelligence force people to upload so as to more efficiently use the matter of their bodies?
Might an aligned superintelligence immediately kill everyone and then go on to create a hedonium shockwave?
Might an intelligence explosion never occur?
Might attempting to align AI cause a 'near miss' which is a much worse outcome?
Might humanity create astronomical amounts of suffering when colonizing the universe if a superintelligence does what we want?
Might trying to build a hedonium maximizing AI be easier and more likely to work than trying for eudaimonia?
Once we notice that a superintelligence given a specific task is starting to try to take over the world, can’t we turn it off, reprogram it, or otherwise correct the problem?
People talk about "Aligning AI with human values" but which humans' values are we talking about?
Should we expect warning shots before an unrecoverable catastrophe?
Superintelligence sounds a lot like science fiction. Do people think about this in the real world?
The future seems very hard to predict. Why think we can say anything useful about AGI today?
This all seems rather abstract. Isn't promoting love, wisdom, altruism, or rationality more important?
To what extent are there meaningfully different paths to AGI, versus just one path?
We already have psychopaths who are misaligned with the rest of the people, but somehow we deal with them. Can't we do the same with AI?
We’re going to merge with the machines so this will never be a problem, right?
What AI policy organisations exist?
What about AI concerns other than existential safety?
What about having a human supervisor who must approve all the AI's decisions before executing them?
What actions can I take in under 5 minutes to contribute?
What alignment research agendas are being pursued?
What alignment strategies are scalably safe and competitive?
What approaches are AI alignment organizations working on?
What are alternate phrasings for?
What are brain-computer interfaces?
What are coherence theorems and what do they tell us about AI?
What are common objections to AI alignment and brief responses?
What are good resources on AI alignment?
What are helpful AI policy ideas?
What are human values?
What are plausible candidates for pivotal acts?
What are safety problems related to whole brain emulations?
What are scaling laws and how are they relevant to safety?
What are selection theorems and can they tell us anything useful about the likely shape of AGI systems?
What are some good podcasts about AI alignment?
What are some important terms in AI alignment?
What are some of the leading AI capabilities organizations?
What are some of the most carefully thought out objections to AI alignment?
What are some open research questions?
What are some practice or entry level problems for getting into alignment research?
What are some problems in philosophy that are related to AI safety?
What are the differences between AGI, TAI, and Superintelligence?
What are the different possible AI takeoff speeds?
What are the different proposed decision theories?
What are the leading moral theories in philosophy and which might be technically easiest to programm into an AI?
What are the potential benefits of AI as it grows increasingly sophisticated?
What are the style guidelines for writing for Stampy?
What are the win conditions/problems that need to be solved?
What beneficial things would an aligned AGI be able to do?
What can we do to contribute to AI safety?
What can we expect the motivations of a superintelligent machine to be?
What convinced people working on AI alignment that it was worth spending their time on this cause?
What does Elon Musk think about AI safety?
What does a typical work day in the life of a safety researcher look like?
What does alignment failure look like?
What does the world shortly before AGI look like?
What evidence do experts usually base their timeline predictions on?
What expert surveys on AI safety exist?
What external content would be useful to the Stampy project?
What harm could a single superintelligence do, when it took so many humans to build civilization?
What if tech progress stagnates and we never reach AGI?
What if we put AI in a box, and have a second more powerful AI with a goal to prevent the first one escaping?
What is 'Do What I Mean'?
What is 'Transformative AI'?
What is AGI and what will it look like?
What is AI Safety?
What is AI alignment?
What is Causal Decision Theory?
What is Codex / Github Copilot?
What is Coherent Extrapolated Volition?
What is Evidential Decision Theory?
What is Friendly AI?
What is Functional Decision Theory?
What is GPT-3?
What is HCH?
What is Logical Decision Theory?
What is MIRI’s mission?
What is a canonical question on Stampy's Wiki?
What is a canonical version of a question on Stampy's Wiki?
What is a duplicate question on Stampy's Wiki?
What is a follow-up question on Stampy's Wiki?
What is a pivotal act?
What is a quantilizer?
What is a value handshake?
What is a verified account on Stampy's Wiki?
What is agent foundations?
What is an agent?
What is an s-risk?
What is biological cognitive enhancement?
What is greater-than-human intelligence?
What is hedonium?
What is meant by AI takeoff?
What is metaphilosophy and how does it relate to AI safety?
What is narrow AI?
What is superintelligence?
What is the Control Problem?
What is the Stampy project?
What is the general nature of the concern about AI alignment?
What is the intelligence explosion?
What is the long reflection?
What is the orthogonality thesis?
What is the probability of extinction from misaligned superintelligence?
What is the universal prior?
What is the windfall clause?
What is whole brain emulation?
What kind of a challenge is solving AI alignment?
What kind of questions do we want on Stampy?
What milestones are there between us and AGI?
What military applications of AI will likely exist?
What organizations are working on AI existential safety?
What plausibly happens 5 years before and after AGI?
What research is being done to align modern deep learning systems?
What should I read to learn about decision theory?
What should be marked as a canonical answer on Stampy's Wiki?
What should be marked as a related question on Stampy's Wiki?
What should the first AGI systems be aligned to do?
What sources of information can Stampy use?
What subjects should I study in university to prepare for alignment research?
What technical problems are MIRI working on?
What technological developments could speed up AI progress?
What things are physically possible and would be neat to have in an AI built utopia?
What things could a superintelligent AI do and what things are physically impossible even for it?
What university should I study at if I want to best prepare for working on AI alignment?
What will be the first transformative applications of AI?
What would a good future with AGI look like?
What would a warning shot look like?
What would an actually good solution to AI alignment look like?
What's especially worrisome about autonomous weapons?
What's meant by calling an AI "agenty" or "agentlike"?
What’s a good AI alignment elevator pitch?
When should I stamp an answer?
When will an intelligence explosion happen?
When will transformative AI be created?
Where can I find mentorship and advice for becoming a researcher?
Where can I find questions to answer for Stampy?
Where can I find the all the features of Stampy's Wiki?
Where can I learn about interpretability?
Which country will AGI be created by, and does this matter?
Who created Stampy?
Who helped create Stampy?
Who is Professor Nick Bostrom?
Who is Stampy?
Why can't we just make a child AI and raise it?
Why can't we simply stop developing AI?
Why can't we turn the computers off?
Why can’t we just use Asimov’s 3 laws of robotics?
Why can’t we just use natural language instructions?
Why can’t we just “put the AI in a box” so it can’t influence the outside world?
Why can’t we just…
Why do you like stamps so much?
Why does AI need goals in the first place? Can’t it be intelligent without any agenda?
Why does AI takeoff speed matter?
Why does there seem to have been an explosion of activity in AI in recent years?
Why don't we just not build AGI if it's so dangerous?
Why don’t some people worry about alignment?
Why is AGI dangerous?
Why is AI Safety important?
Why is AI alignment hard?
Why is safety important for smarter-than-human AI?
Why is the future of AI suddenly in the news? What has changed?
Why might a maximizing AI cause bad outcomes?
Why might a superintelligence be dangerous?
Why might an AI do something that we don’t want it to, if it’s really so intelligent?
Why might contributing to Stampy be worth my time?
Why might people try to build AGI rather than stronger and stronger narrow AIs?
Why might we expect a fast takeoff?
Why might we expect a moderate AI takeoff?
Why might we expect a superintelligence to be hostile by default?
Why should I worry about superintelligence?
Why should we prepare for human-level AI technology now rather than decades down the line when it’s closer?
Why think that AI can outperform humans?
Why work on AI safety early?
Why would great intelligence produce great power?
Will AGI be agentic?
Will AI create or maintain suffering because some people want it?
Will AI learn to be independent from people or will it always ask for our orders?
Will an aligned superintelligence care about animals other than humans?
Will superintelligence make a large part of humanity unemployable?
Will there be a discontinuity in AI capabilities? If so, at what stage?
Will there be an AI assisted long reflection and how might it look like?
Will we ever build a superintelligence?
Won’t AI be just like us?
Would AI alignment be hard with deep learning?
Would it improve the safety of quantilizers to cut of the top few % of the distribution?
Would warning shots make a difference, and if so, would they be helpful or harmful?
Wouldn't a superintelligence be smart enough not to make silly mistakes in its comprehension of our instructions?
Wouldn't a superintelligence be smart enough to know right from wrong?
Wouldn't it be a good for humanity to die out?
Wouldn't it be safer to only build narrow AIs?