Intelligence and Stupidity: The Orthogonality Thesis

From Stampy's Wiki

Intelligence and Stupidity: The Orthogonality Thesis
Channel: Robert Miles
Published: 2018-01-11T19:53:43Z
Views: 269272
Likes: 18642
265 questions on this video!
QuestionYouTubeLikesAsked Discord?AnsweredBy
Slikrx's question on The Orthogonality Thesis88true
Say Hoe Lee's question on The Orthogonality Thesis85false
Chain Chompsky's question on The Orthogonality Thesis62true
Reckless Roges's question on The Orthogonality Thesis13true
Damian Reloaded's question on The Orthogonality Thesis8true
Simon G.'s question on The Orthogonality Thesis6true
SbotTV's question on The Orthogonality Thesis4false
ꗊꗊ's question on The Orthogonality Thesis3true
Davy Jones's question on The Orthogonality Thesis3false
Андрей Шедько's question on The Orthogonality Thesis3true
A8lg6p's question on The Orthogonality Thesis2false
Keith Noah's question on The Orthogonality Thesis2false
Wiczus's question on The Orthogonality Thesis1false
Max Strelnikov's question on The Orthogonality Thesis1false
David DUNAND's question on The Orthogonality Thesis1false
Icebluscorpion's question on The Orthogonality Thesis1false
Sluppie's question on The Orthogonality Thesis1false
Daniel Robinson's question on The Orthogonality Thesis1false
Springlumpy's question on The Orthogonality Thesis1true
Greg Weigner's question on The Orthogonality Thesis1false
GreatMCGamer's question on The Orthogonality Thesis1false
Iliyan Bobev's question on The Orthogonality Thesis1false
Thedenial's question on The Orthogonality Thesis1false
The Fox Among Wolves's question on The Orthogonality Thesis1false
14zRobot's question on The Orthogonality Thesis1false
Echoes's question on The Orthogonality Thesis1trueSudonym's Answer to The Orthogonality Thesis on 2019-04-15T17:07:34 by echoes
Jeremy Rice's question on The Orthogonality Thesis1false
Avian's question on The Orthogonality Thesis1false
Eugene Vovk's question on The Orthogonality Thesis1true
JonnyRobbie's question on The Orthogonality Thesis1false
Max Omnicast's question on The Orthogonality Thesis0false
You tou's question on The Orthogonality Thesis0false
Илья Шаров's question on The Orthogonality Thesis0false
Brandon Bahret's question on The Orthogonality Thesis0true
Milos Marinkovic's question on The Orthogonality Thesis0true
Bojan Marjanovic's question on The Orthogonality Thesis0false
Nevo Krien's question on The Orthogonality Thesis0false
Okay's question on The Orthogonality Thesis0false
Frédéric Schneider's question on The Orthogonality Thesis0false
Andrey Medina's question on The Orthogonality Thesis0false
Sumner Stuart's question on The Orthogonality Thesis0false
M Wing's question on The Orthogonality Thesis0false
Salec's question on The Orthogonality Thesis0false
James Hutchings's question on The Orthogonality Thesis0false
Okan koç's question on The Orthogonality Thesis0false
Nathan Metsinger's question on The Orthogonality Thesis0true
Mr X's question on The Orthogonality Thesis0false
Falconeye4's question on The Orthogonality Thesis0true
David Blette's question on The Orthogonality Thesis0false
Ralph Shively's question on The Orthogonality Thesis0false
... further results


Can highly intelligent agents have stupid goals?
A look at The Orthogonality Thesis and the nature of stupidity.

The 'Stamp Collector' Computerphile video:
My other Computerphile videos:

Katie Byrne's Channel:
Chad Jones' Channel:
With thanks to my wonderful Patreon supporters:
- Steef
- Sara Tjäder
- Jason Strack
- Chad Jones
- Stefan Skiles
- Ziyang Liu
- Jordan Medina
- Jason Hise
- Manuel Weichselbaum
- 1RV34
- James McCuen
- Richárd Nagyfi
- Ammar Mousali
- Scott Zockoll
- Ville Ahlgren
- Alec Johnson
- Simon Strandgaard
- Joshua Richardson
- Jonatan R
- Michael Greve
- robertvanduursen
- The Guru Of Vision
- Fabrizio Pisani
- Alexander Hartvig Nielsen
- Volodymyr
- David Tjäder
- Paul Mason
- Ben Scanlon
- Julius Brash
- Mike Bird
- Tom O'Connor
- Gunnar Guðvarðarson
- Shevis Johnson
- Erik de Bruijn
- Robin Green
- Alexei Vasilkov
- Maksym Taran
- Laura Olds
- Jon Halliday
- Robert Werner
- Roman Nekhoroshev
- Konsta
- William Hendley
- DGJono
- Matthias Meger
- Scott Stevens
- Emilio Alvarez
- Michael Ore
- Dmitri Afanasjev
- Brian Sandberg
- Einar Ueland
- Lo Rez
- Marcel Ward
- Andrew Weir
- Taylor Smith
- Ben Archer
- Scott McCarthy
- Kabs Kabs
- Phil
- Tendayi Mawushe
- Gabriel Behm
- Anne Kohlbrenner
- Jake Fish
- Bjorn Nyblad
- Stefan Laurie
- Jussi Männistö
- Cameron Kinsel
- Matanya Loewenthal
- Wr4thon
- Dave Tapley
- Archy de Berker
- Kevin
- Vincent Sanders
- Marc Pauly
- Andy Kobre
- Brian Gillespie
- Martin Wind
- Peggy Youell
- Poker Chen


hi this video is kind of a response to
various comments that I've got over the
years ever since that video on computer
file where I was describing the sort of
problems that we might have when we have
a powerful artificial general
intelligence with goals which aren't the
same as our goals even if those goals
seem pretty benign we use this thought
experiment of an extremely powerful AGI
working to optimize the simple goal of
collecting stamps and some of the
problems that that might cause I got
some comments from people saying that
they think the stamp collecting device
is stupid and not that it's a stupid
thought experiment but the device itself
is actually stupid they said unless it
has complex goals or the ability to
choose its own goals then it didn't
count as being highly intelligent in
other videos I got comments saying it
takes intelligence to do moral reasoning
so an intelligent AGI system should be
able to do that and a super intelligence
should be able to do it better than
humans in fact if a super intelligence
decides that the right thing to do is to
kill us all then I guess that's the
right thing to do these comments are all
kind of suffering from the same mistake
which is what this video is about but
before I get to that I need to lay some
groundwork first if you like Occam's
razor then you'll love Humes guillotine
also called the is odd problem this is a
pretty simple concept that I'd like to
be better known the idea is statements
can be divided up into two types is
statements and Hort statements these
statements or positive statements are
statements about how the world is how
the world was in the past how the world
will be in the future or how the world
would be in hypothetical situations this
is facts about the nature of reality the
causal relationships between things that
kind of thing then you have the ought
statements the should statements the
normative statements these are about the
way the world should be the way we want
the world to be statements about our
goals our values ethics morals what we
want all of that stuff now you can
derive logical statements from one
another like it's snowing outside
that's a nice statement it's cold when
it snows another s statement and then
you can deduce therefore it's cold
that's another is statement it's our
conclusion this is all pretty obvious
but you might say something like it's
snowing outside therefore you ought to
put on a coat and that's a very normal
sort of sentence that people might say
but as a logical statement it actually
relies on some hidden assumption
without assuming some kind of ought
statement you can't derive another ought
statement this is the core of the Azure
problem you can never derive an ought
statement using only is statements you
ought to put on a coat why because it's
snowing outside so what is the fact that
it's snowing mean I should put on the
coat well the fact that it's snowing
means that it's cold and why should it
being cold mean I should put on a coat
if it's cold and you go outside without
a coat you'll be cold should I not be
cold well if you get too cold you'll
freeze to death okay you're saying I
shouldn't freeze to death
that was kind of silly but you see what
I'm saying you can keep laying out is
statements for as long as you want you
will never be able to derive that you
ought to put on a coat at some point in
order to derive that ought statement you
need to assume at least one other ought
statement if you have some kind of ought
statement like I ought to continue to be
alive you can then say given that I
ought to keep living and then if I go
outside without a coat I'll die then I
ought to put on a coat but unless you
have at least one ought statement you
cannot derive any other ought statements
and Hort statements are separated by
Hume skia T okay so people are saying
that a device that single-mindedly
collects stamps at the cost of
everything else is stupid and doesn't
count as a powerful intelligence so
let's define our terms what is
intelligence and conversely what is
stupidity I feel like I made fairly
clear in those videos what I meant by
intelligence we're talking about a GI
systems as intelligent agents they're
entities that take actions in the world
in order to achieve their goals or
maximize their utility functions
intelligence is the thing that allows
them to choose good actions to choose
actions that will get them what they
want an agent's level of intelligence
really means its level of effectiveness
of pursuing its goals in practice this
is likely to involve having or building
an accurate model of reality keeping
that model up-to-date by reasoning about
observations and using the model to make
predictions about the future and the
likely consequences of different
possible actions to figure out which
actions will result in which outcomes
intelligence involves answering
questions like what is the world like
how does it work what will happen next
what would happen in this scenario or
that scenario what would happen if I
took this action or that action more
intelligent systems are in some sense
better at answering these kinds of
questions which allows them to be better
at choosing actions but one thing you
might notice about these questions is
they're all ears questions the system
has goals which can be thought of as
Hort statements but the level of
intelligence depends only on the ability
to reason about is questions in order to
answer the single ort question what
action should I take next so given that
that's what we mean by intelligence what
does it mean to be stupid well firstly
you can be stupid in terms of those
questions for example by building a
model that doesn't correspond with
reality or by failing to update your
model properly with new evidence if I
look out of my window
and I see there's snow everywhere you
know I see a snowman and I think to
myself oh what a beautiful warm sunny
day then that's stupid right my belief
is wrong and I had all the clues to
realize it's cold outside so beliefs can
be stupid by not corresponding to
what about actions like if I go outside
in the snow without my coat that's
stupid right well it might be if I think
it's sunny and warm and I go outside to
sunbathe then yeah that's stupid but if
I just came out of a sauna or something
and I'm too hot and I want to cool
myself down then going outside without a
coat might be quite sensible you can't
know if an action is stupid just by
looking at its consequences you have to
also know the goals of the agent taking
the action you can't just use is
statements you need a naught so actions
are only stupid relative to a particular
goal it doesn't feel that way though
people often talk about actions being
stupid without specifying what goals
they're stupid relative to but in those
cases the goals are implied we're humans
and when we say that an action is stupid
in normal human communication we're
making some assumptions about normal
human goals and because we're always
talking about people and people tend to
want similar things it's sort of a
shorthand that we can skip what goals
were talking about so what about the
goals then can goals be stupid
well this depends on the difference
between instrumental goals and terminal
this is something I've covered elsewhere
but your terminal goals are the things
that you want just because you want them
you don't have a particular reason to
want them they're just what you want the
instrumental goals are the goals you
want because they'll get you closer to
your terminal goals like if I have a
terminal goal to visit a town that's far
away maybe an instrumental goal would be
to find a train station I don't want to
find a train station just because trains
are cool I want to find a train as a
means to an end it's going to take me to
this town
so that makes it an instrumental goal
now an instrumental goal can be stupid
if I want to go to this distant town so
I decide I want to find a pogo stick
that's pretty stupid
finding a pogo stick is a stupid
instrumental goal if my terminal goal is
to get to a faraway place but if we're
terminal go with something else like
having fun it might not be stupid so in
that way it's like actions instrumental
goals can only be stupid relative to
terminal goals so you see how this works
beliefs and predictions can be stupid
relative to evidence or relative to
reality actions can be stupid relative
to goals of any kind
instrumental goals can be stupid
relative to terminal goals but here's
the big point terminal goals can't be
stupid there's nothing to judge them
against if a terminal goal seems stupid
like let's say collecting stamps seems
like a stupid terminal goal that's
because it would be stupid as an
instrumental goal to human terminal
goals but the stamp collector does not
have human terminal goals
similarly the things that humans care
about would seem stupid to the stamp
collector because they result in so few
stamps so let's get back to those
comments one type of comments says this
behavior of just single mindedly going
after one thing and ignoring everything
else and ignoring the totally obvious
fact that stamps aren't that important
is really stupid behavior you're calling
this thing of super intelligence but it
doesn't seem super intelligent to me it
just seems kind of like an idiot
hopefully the answer to this is now
clear the stamp collectors actions are
stupid relative to human goals but it
doesn't have human goals its
intelligence comes not from its goals
but from its ability to understand and
reason about the world allowing it to
choose actions that achieve its goals
and this is true whatever those goals
actually are some people commented along
the lines of well okay yeah sure you've
defined intelligence to only include
this type of is statement kind of
reasoning but I don't like that
definition I think to be truly
intelligent you need to have complex
goals something with simple goals
doesn't count as intelligent to that I
say well you can use words however you
want I guess I'm using intelligence here
as a technical term in the way that it's
often used in the field you're free to
have your own definition of the word but
the fact that something fails to meet
your definition of intelligence does not
mean that it will fail to behave in a
way that most people would call
if the stamp collector outwits you gets
around everything you've put in its way
and outmaneuvers you mentally it comes
up with new strategies that you would
never have thought of to stop you from
turning it off and stopping from
preventing it from making stamps and as
a consequence it turns the entire world
into stamps in various ways you could
never think of it's totally okay for you
to say that it doesn't count as
intelligent if you want but you're still
dead I prefer my definition because it
better captures the ability to get
things done in the world which is the
reason that we actually care about AGI
in the first place
similarly people who say that in order
to be intelligent you need to be able to
choose your own goals
I would agree you need to be able to
choose your own instrumental goals but
not your own terminal goals changing
your terminal goals is like willingly
taking a pill that will make you want to
murder your children it's something you
pretty much never want to do apart from
some bizarre edge cases if you
rationally want to take an action that
changes one of your goals then that
wasn't a terminal goal now moving on to
these comments saying an AGI will be
able to reason about morality and if
it's really smarter than us it will
actually do moral reasoning better than
so there's nothing to worry about it's
true that a superior intelligence might
be better at moral reasoning than us but
ultimately moral behavior depends not on
moral reasoning but on having the right
terminal goals there's a difference
between figuring out and understanding
human morality and actually wanting to
act according to it the stamp collecting
device has a perfect understanding of
human goals ethics and values and it
uses that only to manipulate people for
stamps it's super human moral reasoning
doesn't make its actions good if we
create a super intelligence and it
decides to kill us that doesn't tell us
anything about morality it just means we
screwed up
so what mistake do all of these comments
have in common the orthogonality thesis
in AI safety is that more or less any
goal is compatible with more or less any
level of intelligence ie those
properties are orthogonal you can place
them on these two axes and it's possible
to have agents anywhere in this space
anywhere on either scale you can have
very weak low intelligence agents that
have complex human compatible goals you
can have powerful highly intelligent
systems with complex sophisticated goals
you can have weak simple agents with
silly goals and yes
can have powerful highly intelligent
systems with simple weird inhuman goals
any of these are possible because level
of intelligence is about effectiveness
at answering is questions and goals are
all about what questions and the two
sides are separated by Humes guillotine
hopefully looking at what we've talked
about so far it should be pretty obvious
that this is the case like what would it
even mean for it to be false but for it
to be impossible to create powerful
intelligences with certain goals the
stamp collector is intelligent because
it's effective at considering the
consequences of sending different
combinations of packets on the internet
and calculating how many stamps that
results in exactly how good do you have
to be at that before you don't care
about stamps anymore and you randomly
start to care about some other thing
that was never part of your terminal
goals like feeding the hungry or
whatever it's just not gonna happen so
that's the orthogonality thesis it's
possible to create a powerful
intelligence that will pursue any goal
you can specify knowing an agent's
terminal goals doesn't really tell you
anything about its level of intelligence
and knowing an agent's level of
intelligence doesn't tell you anything
about its goals
I want to end the video by saying thank
you to my excellent patrons so it's all
of these people here thank you so much
for your support
lets me do stuff like building this
light boy thank you for sticking with me
through that weird patreon fees thing
and my moving to a different city which
has really got in the way of making
videos recently but I'm back on it now
new video every two weeks is the part
anyway in this video I'm especially
Franklin Katie Beirne who's supported
the channel for a long time she actually
has her own YouTube channel about 3d
modeling and stuff so a link to that and
while I'm at it when I think Chad Jones
ages ago I didn't mention his YouTube
channel so link to both of those in the
description thanks again and I'll see
you next time I don't speak cat what
does that mean