Lex Clips
Full episode with Dileep George (Aug 2020): https://www.youtube.com/watch?v=tg_m_LxxRwM
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)
Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41
Podcast website:
https://lexfridman.com/ai
Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr
Podcast on Spotify:
https://spoti.fi/2nEwCF8
Podcast RSS:
https://lexfridman.com/category/ai/feed/
Dileep George is a researcher at the intersection of neuroscience and artificial intelligence, co-founder of Vicarious, formerly co-founder of Numenta. From the early work on Hierarchical temporal memory to Recursive Cortical Networks to today, Dileep’s always sought to engineer intelligence that is closely inspired by the human brain.
Subscribe to this YouTube channel or connect on:
– Twitter: https://twitter.com/lexfridman
– LinkedIn: https://www.linkedin.com/in/lexfridman
– Facebook: https://www.facebook.com/lexfridman
– Instagram: https://www.instagram.com/lexfridman
– Medium: https://medium.com/@lexfridman
– Support on Patreon: https://www.patreon.com/lexfridman
first !!!!
Duos!!!
Honestly, I don't know for sure that doctors wear underwear. Is my world model any better than GPT-3's?
I disagree with the "text world" vs "the world" argument. We see and interact with the world, but we work with abstractions all the time. Are you aware of all the components used to build the floor in which you are right now? Probably not, but you call it a floor and you know that you can safely stand on it. Our world model is not really that different from what we use when writing.We look at things and we see concepts, not really the things we see. That's why most people can't draw. It takes a lot of effort to really "see" things in a way that you can reproduce them on paper.
Thought experiment:
1) Attach a tiny non-obtrusive camera to a baby's head, with auditory, and olfactory sensors.
2) Record everything
3) Spend $100 billion. Have a multi-trillion parameter network try to predict the baby's actions
Sort of like Tesla does, training it's models on what real humans actually did.
Now at best you have a baby simulator.
Lex, you are a stud. Awesome questions.
Doctors wear underwear? The perfect Turing test… lol
CP 4 can simply ask the questions that it does not know the answer in order to create a more detailed world
Happy birthday Dr. Fridman 🙂 Hope you had an amazing day!
I disagree with the guests criticism of the GTP-3 approach. First, he simply doesn't know what the internet doesn't contain in it. Most certainly, there is plain text on the internet that specifically says "doctors have underwear" and "it's easier to walk forward". The amount of plain text that exists in the world today is truly staggering. A person simply can't comprehend what implicit knowledge could be constructed which is not explicitly stated in all that plain-text. Neural nets do indeed build extremely sophisticated models from the training data which is precisely how they are able to compress all the data into their network ( which is much smaller by comparison ). That level of compression is impossible without having effective models. The modeling involves many layers of abstraction so when the network is asked to do a task, the new information is translated into a highly abstract form where it can be compared and related to other concepts, correctly, even if such comparisons and relations were never explicit in the original training data. If this was not so, it would be impossible for GTP-3 to do many of the magical things it does.
Can a system like CPT-3 make assumptions from incomplete data, such as all doctors wear underwear? Can they program reasoning already??
But what if we translate pictures to base64 text and train gpt on it?
World AGI Developer
DeepMind(UK), Google Brain(USA), MSR AI (USA), Numenta(USA),
Vicarious(USA), IBM Research AI(USA), FAIR(USA), OpenAI(USA),
NNAISENSE(Swiss), WBAI(Japan), GoodAI(Czech), 中国科学院CASIA(China),
Human Brain Project(EU), BRAIN Initiative(USA), Brain/MINDS(Japan),
OpenCog(Hong Kong), Nengo(Can), Carboncopies(USA), Micropsi industries(Deutsch)
surly next step for AI is it collects its own real world high fidelity data in the form of drones …GPT-3 would deduce some wear underwear sometimes and some don't
This technology is amazing
To touch on something the guest mentioned:
I asked GPT-3 (using AI-Dungeon) if it was easier to walk backwards or forward. Its response:
"It's harder to walk backward than forward".
I then asked it, "How do you know?". It responded:
"I don't! It's just what I've always known!"
Then I asked, "Do doctors wear underwear?"
It responded, "Of course they do! Everyone does".
Haha!
Commonsense as the hard
problem of cognitive science
• “Do doctors wear underwear?”
• “Is it easier to walk forwards or backward?”
• “If President Obama is in Washington, is his
spleen in Washington?”
• Everything you needed to know to pass the
Turing test you learned before kindergarten
This is taken from the internet:
https://ocw.mit.edu/courses/brain-and-cognitive-sciences/9-85-infant-and-early-childhood-cognition-fall-2012/lecture-notes/MIT9_85F12_lec1_intro.pdf
GPT3 Has most likely seen it. I don't think George understands the scale of the dataset.
Yes I talking to Noam about the lecture analogy as the core of cognition lecture. That makes sense to me that is would not have the answer that make any sense. Schizophrenia is rare I'd be concerned that the software has it. Maybe connect electrodes, shock it with insulin or gas lamp the hell out of it so it forgets every moment it was created.
All doctors wear underwear… lol
Good talk. Ty for sharing
LEX No Affect can't be generated from concept
He was too far from the microphone and the audio was echoey
Parlor games
Hey Lex, my startup is working on creating exactly this kind of DB you were postulating calling to it tagging and annotating of millions of college students worldwide. Wondering what you would think about our model
could a human being born without any senses (no sight, hearing, touch, etc) have a text conversation with an average person? Without the other person knowing?
As a living thing, you are continuous with all of reality. As neural network, you get rows and columns. The amount of information exposed to GPT-3 is laughably small as relative to what living things are exposed to.
There are many ways to do anything in programming, but scaling up GPT-3 to GPT-10 for AGI might not be the most efficient architecture. We are looking for the most efficient AGI, not just any AGI.
GPT is missing a body. We don't often remind people in text of body functions.
We also have unstated premises, cultural differences, and uncomfortable truths.
He's right. The world is the world and our text is nothing but a very low res representation of small parts of it that we use to communicate with each other. Right now you are doing most of the work in deciphering the meaning of my text and you bring what is basically a massive look-up table to the process that will never be represented by (or reverse engineered from ) just the text alone. Think of all the things that bestow some sort of significance on even just simple objects – maybe we've written about some of the contexts we find that object in and how we feel about and respond to it and what meanings and significance it has for us, but maybe we've barely scratched the surface of all that in our text. And even if we can assume we'd done a good descriptive job for one object, how about the next individual object? How about a world full of limitless objects where new things are constantly being invented?
Is it just me or is anyone else wondering whether that coffee in his cup was maybe a little heavy on the Baileys? 😉
My thoughts exactly! Correlation doesn't imply causation. GPT-3 is very good at capturing correlations, but that will not be enough to learn causality.
2:55 "its a model of the textworld, not a model of the world", he is acting like we have a model of the real world and not just some subset
I don’t understand why he claims that the text corpus doesn’t contain the information needed to form some amount of world model.
Of course, I do expect that, at least for GPT-3, that if it has a model of the world, that said world model would be lacking in many ways. But I find it hard to believe that it doesn’t in some sense have a world model. I guess it is necessary to be more precise in what we mean by it having a world model.
The people who produced the text which was the training set for the model, of course had models of the world, and they used these models of the world in order to produce the text.
What is a model of the world? I suppose it is something like a probability distribution over ways the world could be. I think a Newtonian physics model could be called “a model of the world” in a way, even though it would be very limited in its applicability. Even though Newtonian physics is deterministic, we can still at least treat it as having conditional probabilities of the form “the probability of configuration X at time t, conditional on configuration Y at time 0, is [either 0 or 1 depending on whether that is what the physics model predicts]”, though it would be lacking an unconditional probability distribution over states of the world.
It seems that if I could predict anything that Joe would say in response to a prompt, and if Joe is the kind of person who has expertise in some area and is willing and able to use that expertise to explain things, that I would then also have understanding, in some sense, of the thing Joe understands?
Well, sort-of ?
I suppose this gets into the question of the Chinese room.
If I (and this is obviously impossible for me to do, but for the moment pretend it isn’t) memorize an algorithm and data for how to simulate the brain of person who speaks Chinese, even if this allows me to use this simulation to carry on a conversation in Chinese between some other person and the simulation, I would be kind of acting as a relay, and wouldn’t necessarily understand any of the Chinese speech(I might just be simulating some Turing machines at impossible speeds or whatever, and then have some method of converting between the input/output of the simulation to sounds).
Ok. So this suggests a way that such models might lack understanding of the world, in some sense.
But if I had such a simulation of a person who spoke Chinese in my head,
the information of an understanding of how to speak Chinese, would still be in my head even if I didn’t know how to use it as such an understanding.
I think it would make sense to say that in such a hypothetical, I would still have a model of Chinese, even though I would not understand Chinese.
So, in this sense, perhaps GPT-3 has some amount of a model of the world, even though it doesn’t understand the world?
Thoughts?
5:33 text limitation, so what about blind people ?! hmm…
It depends what you think of language. Is it a logical or creative endeavor?
👏👏👏👏👏👏👍👌
very interesting perspective, I think models like GPT-3 fill one gap in achieving natural language modeling. The most important part that is missing can be learnt from toddlers…. ask "WHY?" and then "WHY?" again until you get to the limit of human understanding and knowledge. the problem is we are building structures to understand how TEXT is related to different parts of a sentence but we are not tackling the concept of learning concepts from that text. In order to do that we have to teach it how to learn, and the best way to learn is to ask WHY? until you get to first principal on everything. We have to be able to build a model that can ask why and analyze the answers to be able to properly " ask the right question", what these models are trying to do is "find the right answer" but we are not teaching our models to learn to ask the right question.
reminds me of Alice in wonderland when she meets the worm that is smoking and asks him "where should I go?" and he answers "well where do you want to get to?" and Alice says "I don't know". The worm answers "then whichever direction you go, you will get somewhere!". without asking the right question we are just randomly wandering and will get somewhere, sometime, but not where we want to go.
Transformers seem to be the latest big thing! Kevin Rose/Tim Ferriss were impressed by OpenAI GPT-3's automatic website generation (they talk about it on the Random Show).
In its head hahahaha
Your very convincing almost had "it" hahaha