Upcoming questions

From Stampy's Wiki

These are the upcoming top questions as sorted by the question review page, in the order they will be posted to Discord (sorted by review status then YouTube likes, best first for both).

Answer to the best of your ability, your answers will be reviewed and stamped by others so don't worry about it not being perfect :)

If you want your reply to be posted on YouTube by Stampy rather than by hand by you, it's best to use the Discord interface (post "stampy, reply formatting" in #general for instructions).

Upcoming questions

Question Write answer Mark as
The Great of Beam's question on The Windfall Clause

Money and profits will become obsolete. The first company to discover AGI will not bother making/selling products. Imagine having a wish-granting genie with unlimited wishes. Why would you bother creating and selling products when you could just wish everything you want into existence?

No tags (edit tags)





Quality:






Michael Spence's question on Instrumental Convergence

So, you're only mostly right when you say that modifying human values doesn't come up much. I can think of two examples in particular. First, the Bible passage which states, "The love of money is the root of all evil". (Not a Christian btw, just pointing it out). The idea here is that through classical conditioning, it's possible for people to start to value money for the sake of money - which is actually a specific version of the more general case, which I will get to in a moment.

The second example is the fear of drug addiction. Which amounts to the fear that people will abandon all of their other goals in pursuit of their drug of choice, and is often the case for harder drugs. These are both examples of wireheading, which you might call a "Convergent Instrumental Anti-goal" and rests largely on the agent being self-aware. If you have a model of the world that includes yourself, you intuitively understand that putting a bucket on your head doesn't make the room you were supposed to clean any less messy. (Or if you want to flip it around, you could say that wireheading is anathema to goal-preservation)

I'm curious about how this applies to creating AGIs with humans as part of the value function, and if you can think of any other convergent anti-goals. They might be just as illuminating as convergent goals.

Edit: Interestingly, you can also engage in wireheading by intentionally perverting your model of reality to be perfectly in-line with your values. (You pretend the room is already clean). This means that having an accurate model of reality is a part of goal-preservation.

No tags (edit tags)





Quality:






Sdtok's question on Avoiding Positive Side Effects

Can you tell us more about how a world with a safe AGI would look like? Will the people to invent an AGI rule the world, outperforming everyone at stock trading for instance? Is it profitable to get second (or how big will the head start be when someone invents AGI second like a week later)? I would love to hear this kinds of things from you! But a good reference would make my day too. Keep up the good work!

No tags (edit tags)





Quality:






Jonathan Zea's question on Intro to AI Safety

Is it really that "safe AI is totally possible"?? How can you be so sure???

No tags (edit tags)





Quality:






Daniel Buzovský's question on Where do we go now

Is AGI avoidable? Is there a way to advance in technology and evolve as a humanity in general without ever coming to point where we turn that thing on. More philosophical one.

No tags (edit tags)





Quality:






Melon Collie's question on The Windfall Clause

Well if I ended up with an AGI or more likely ASI that so happened to be hard coded to do what I want (and it actually listens), what's to stop me from just not paying? I mean with an ASI I could very easily take over the world and nobody could do anything about it since I have an ASI and they don't.

Of course I wouldn't actually do that I'm not a psychopath, but I would probably use it to teach certain people a lesson or two.

No tags (edit tags)





Quality:






Loweren's question on Mesa-Optimizers

Great explanation! I heard about these concepts before, but never really grasped them. So on 19:45, is this kind of scenario a realistic concern for a superintelligent AI? How would a superintelligent AI know that it's still in training? How can it distinguish between training and real data if it never seen real data? I assume programmers won't just freely provide the fact that AI is still being trained.

No tags (edit tags)





Quality:






Bloergk's question on Mesa-Optimizers

At the end you write that, when reading the article, this was a "new class of problems" to you... But it just seems like an instance of the "sub-agent stability problem" (not sure of the proper terminology) you've explained before on Computerphile https://www.youtube.com/watch?v=3TYT1QfdfsM.
The only difference is that in this case, we are dumb enough to build the A.I. in a way that forces it to ALWAYS create a sub-agent.

No tags (edit tags)





Quality:






Peter Bonnema's question on The Windfall Clause

Why would a company that develops AGI try to align its goals with those of the world? Why not align it with just their own goals? They are sociopaths after all.

No tags (edit tags)





Quality:






Mera Flynn's question on The Windfall Clause

Question, doesn’t this contract be basically useless in the situation that a company creates a super intelligent AI who’s interests are aligned with theirs? Wouldn’t it very likely try and succeed at getting them out of this contract?

No tags (edit tags)





Quality:






See more...