If you're replying to a question from YouTube and want your reply to be posted on YouTube by Stampy rather than by hand by you, it's best to use the Discord interface (post "stampy, reply formatting" in #general for instructions). The top few questions here are generally manually added rather than from YouTube.
Alternatively go to a random unanswered question.
Some economists say human wants are infinite, and there will always be new and currently unimaginable kinds of jobs for people to do.Others say this won't be true if AGI can do _anything_ human minds can do.
The second example is the fear of drug addiction. Which amounts to the fear that people will abandon all of their other goals in pursuit of their drug of choice, and is often the case for harder drugs. These are both examples of wireheading, which you might call a "Convergent Instrumental Anti-goal" and rests largely on the agent being self-aware. If you have a model of the world that includes yourself, you intuitively understand that putting a bucket on your head doesn't make the room you were supposed to clean any less messy. (Or if you want to flip it around, you could say that wireheading is anathema to goal-preservation)
I'm curious about how this applies to creating AGIs with humans as part of the value function, and if you can think of any other convergent anti-goals. They might be just as illuminating as convergent goals.Edit: Interestingly, you can also engage in wireheading by intentionally perverting your model of reality to be perfectly in-line with your values. (You pretend the room is already clean). This means that having an accurate model of reality is a part of goal-preservation.