Science, Technology & the Future
Joscha Bach on GPT-3, achieving AGI, machine understanding and lots more
02:40 What’s missing in AI atm? Unified coherent model of reality
04:14 AI systems like GPT-3 behave as if they understand – what’s missing?
08:35 Symbol grounding – does GPT-3 have it?
09:35 GPT-3 for music generation, GPT-3 for image generation, GPT-3 for video generation
11:13 GPT-3 temperature parameter. Strange output?
13:09 GPT-3 a powerful tool for idea generation
14:05 GPT-3 as a tool for writing code. Will GPT-3 spawn a singularity?
16:32 Increasing GPT-3 input context may have a high impact
16:59 Identifying grammatical structure & language
19:46 What is the GPT-3 transformer network doing?
21:26 GPT-3 uses brute force, not zero-shot learning, humans do ZSL
22:15 Extending the GPT-3 token context space. Current Context = Working Memory. Humans with smaller current contexts integrate concepts over long time-spans
24:07 GPT-3 can’t write a good novel
25:09 GPT-3 needs to become sensitive to multi-modal sense data – video, audio, text etc
26:00 GPT-3 a universal chat-bot – conversations with God & Johann Wolfgang von Goethe
30:14 What does understanding mean? Does it have gradients (i.e. from primitive to high level)?
32:19 (correlation vs causation) What is causation? Does GPT-3 understand causation? Does GPT-3 do causation?
38:06 Deep-faking understanding
40:06 The metaphor of the Golem applied to civ
42:33 GPT-3 fine with a person in the loop. Big danger in a system which fakes understanding. Deep-faking intelligible explanations.
44:32 GPT-3 babbling at the level of non-experts
45:14 Our civilization lacks sentience – it can’t plan ahead
46:20 Would GTP-3 (a hopfield network) improve dramatically if it could consume 1 to 5 trillion parameters?
47:24 GPT3: scaling up a simple idea. Clever hacks to formulate the inputs
47:41 Google GShard with 600 billion input parameters – Amazon may be doing something similar – future experiments
49:12 Ideal grounding in machines
51:13 We live inside a story we generate about the world – no reason why GPT-3 can’t be extended to do this
52:56 Tracking the real world
54:51 MicroPsi
57:25 What is computationalism? What is it’s relationship to mathematics?
59:30 Stateless systems vs step by step Computation – Godel, Turing, the halting problem & the notion of truth
1:00:30 Truth independent from the process used to determine truth. Constraining truth that which can be computed on finite state machines
1:03:54 Infinities can’t describe a consistent reality without contradictions
1:06:04 Stevan Harnad’s understanding of computation
1:08:32 Causation / answering ‘why’ questions
1:11:12 Causation through brute forcing correlation
1:13:22 Deep learning vs shallow learning
1:14:56 Brute forcing current deep learning algorithms on a Matrioshka brain – would it wake up?
1:15:38 What is sentience? Could a plant be sentient? Are eco-systems sentient?
1:19:56 Software/OS as spirit – spiritualism vs superstition. Empirically informed spiritualism
1:23:53 Can we build AI that shares our purposes?
1:26:31 Is the cell the ultimate computronium? The purpose of control is to harness complexity
1:31:29 Intelligent design
1:33:09 Category learning & categorical perception: Models – parameters constrain each other
1:35:06 Surprise minimization & hidden states; abstraction & continuous features – predicting dynamics of parts that can be both controlled & not controlled, by changing the parts that can be controlled. Categories are a way of talking about hidden states.
1:37:29 ‘Category’ is a useful concept – gradients are often hard to compute – so compressing away gradients to focus on signals (categories) when needed
1:38:19 Scientific / decision tree thinking vs grounded common sense reasoning
1:40:00 Wisdom/common sense vs understanding. Common sense, tribal biases & group insanity. Self preservation, dunbar numbers
1:44:10 Is g factor & understanding two sides of the same coin? What is intelligence?
1:47:07 General intelligence as the result of control problems so general they require agents to become sentient
1:47:47 Solving the Turing test: asking the AI to explain intelligence. If response is an intelligible & testable implementation plan then it passes?
1:49:18 The term ‘general intelligence’ inherits it’s essence from behavioral psychology; a behaviorist black box approach to measuring capability
1:52:15 How we perceive color – natural synesthesia & induced synesthesia
1:56:37 The g factor vs understanding
1:59:24 Understanding as a mechanism to achieve goals
2:01:42 The end of science?
2:03:54 Exciting currently untestable theories/ideas (that may be testable by science once we develop the precise enough instruments). Can fundamental physics be solved by computational physics?
2:07:14 Quantum computing. Deeper substrates of the universe that runs more efficiently than the particle level of the universe?
2:10:05 The Fermi paradox
2:12:19 Existence, death and identity construction
Similar Posts
31 thoughts on “Joscha Bach – GPT-3: Is AI Deepfaking Understanding?”
Comments are closed.
Joscha Bach covers a lot of ground – here are the time points:
02:40 What's missing in AI atm? Unified coherent model of reality
04:14 AI systems like GPT-3 behave as if they understand – what's missing?
08:35 Symbol grounding – does GPT-3 have it?
09:35 GPT-3 for music generation, GPT-3 for image generation, GPT-3 for video generation
11:13 GPT-3 temperature parameter. Strange output?
13:09 GPT-3 a powerful tool for idea generation
14:05 GPT-3 as a tool for writing code. Will GPT-3 spawn a singularity?
16:32 Increasing GPT-3 input context may have a high impact
16:59 Identifying grammatical structure & language
19:46 What is the GPT-3 transformer network doing?
21:26 GPT-3 uses brute force, not zero-shot learning, humans do ZSL
22:15 Extending the GPT-3 token context space. Current Context = Working Memory. Humans with smaller current contexts integrate concepts over long time-spans
24:07 GPT-3 can't write a good novel
25:09 GPT-3 needs to become sensitive to multi-modal sense data – video, audio, text etc
26:00 GPT-3 a universal chat-bot – conversations with God & Johann Wolfgang von Goethe
30:14 What does understanding mean? Does it have gradients (i.e. from primitive to high level)?
32:19 (correlation vs causation) What is causation? Does GPT-3 understand causation? Does GPT-3 do causation?
38:06 Deep-faking understanding
40:06 The metaphor of the Golem applied to civ
42:33 GPT-3 fine with a person in the loop. Big danger in a system which fakes understanding. Deep-faking intelligible explanations.
44:32 GPT-3 babbling at the level of non-experts
45:14 Our civilization lacks sentience – it can't plan ahead
46:20 Would GTP-3 (a hopfield network) improve dramatically if it could consume 1 to 5 trillion parameters?
47:24 GPT3: scaling up a simple idea. Clever hacks to formulate the inputs
47:41 Google GShard with 600 billion input parameters https://arxiv.org/abs/2006.16668 – Amazon may be doing something similar – future experiments
49:12 Ideal grounding in machines
51:13 We live inside a story we generate about the world – no reason why GPT-3 can't be extended to do this
52:56 Tracking the real world
54:51 MicroPsi
57:25 What is computationalism? What is it's relationship to mathematics?
59:30 Stateless systems vs step by step Computation – Godel, Turing, the halting problem & the notion of truth
1:00:30 Truth independent from the process used to determine truth. Constraining truth that which can be computed on finite state machines
1:03:54 Infinities can't describe a consistent reality without contradictions
1:06:04 Stevan Harnad's understanding of computation
1:08:32 Causation / answering 'why' questions
1:11:12 Causation through brute forcing correlation
1:13:22 Deep learning vs shallow learning
1:14:56 Brute forcing current deep learning algorithms on a Matrioshka brain – would it wake up?
1:15:38 What is sentience? Could a plant be sentient? Are eco-systems sentient?
1:19:56 Software/OS as spirit – spiritualism vs superstition. Empirically informed spiritualism
1:23:53 Can we build AI that shares our purposes?
1:26:31 Is the cell the ultimate computronium? The purpose of control is to harness complexity
1:31:29 Intelligent design
1:33:09 Category learning & categorical perception: Models – parameters constrain each other
1:35:06 Surprise minimization & hidden states; abstraction & continuous features – predicting dynamics of parts that can be both controlled & not controlled, by changing the parts that can be controlled. Categories are a way of talking about hidden states.
1:37:29 'Category' is a useful concept – gradients are often hard to compute – so compressing away gradients to focus on signals (categories) when needed
1:38:19 Scientific / decision tree thinking vs grounded common sense reasoning
1:40:00 Wisdom/common sense vs understanding. Common sense, tribal biases & group insanity. Self preservation, dunbar numbers
1:44:10 Is g factor & understanding two sides of the same coin? What is intelligence?
1:47:07 General intelligence as the result of control problems so general they require agents to become sentient
1:47:47 Solving the Turing test: asking the AI to explain intelligence. If response is an intelligible & testable implementation plan then it passes?
1:49:18 The term 'general intelligence' inherits it's essence from behavioral psychology; a behaviorist black box approach to measuring capability
1:52:15 How we perceive color – natural synesthesia & induced synesthesia
1:56:37 The g factor vs understanding
1:59:24 Understanding as a mechanism to achieve goals
2:01:42 The end of science?
2:03:54 Exciting currently untestable theories/ideas (that may be testable by science once we develop the precise enough instruments). Can fundamental physics be solved by computational physics?
2:07:14 Quantum computing. Deeper substrates of the universe that runs more efficiently than the particle level of the universe?
2:10:05 The Fermi paradox
2:12:19 Existence, death and identity construction
Is computational force a meaningful issue for gpt3 advancement? Are there any plans for using latest breakthroughs in quantum computing?
This is an excellent purpose for narrow-focused minds.
I loved the proposition of feeding a book abstract to keep GPT3 on track, then hinting that GTP3 is already able to generate this abstract. Amazing possibility if we can train a model to use that trick by itself, generating a pre-context relative to the input context.
If you are interested in the phenomenon of understanding, here is a playlist of talks and interviews I have created over the years.. more to come: https://www.youtube.com/playlist?list=PL-7qI6NZpO3vgq3Bkz1A1agthYXebhnxP
Thought without consciousness? Does GPT-3 "think"? Is what it does similar to thinking? In humans, thinking generally involves consciousness or awareness, except perhaps when thoughts just "drift through your head" like when you're daydreaming.
man the volumes are so far off it's hard to listen to
1:22:00 "Our preferences seem to be incompatible with what would be necessary for our survival" Joscha Bach is smart enough to see us destroying our planet, will we transcend it in time?
Joscha is an amazing person and a remarkable mind in AI, the dude deserves more credit.
https://www.theaxclinic.com/articles/2020/9/20/joscha-bach-the-lovable-nerd-of-ai
This is the most interesting , and by far, the most exciting video I’ve heard …for awhile. Very informative. Much appreciated!
Many Confuse 'Intelligence' with LIFE…
A.I. should not be called by this, but rather it is crude Intelligence…. There is NO such thing as 'Artificial Intelligence' !
t is either an 'Intelligent Process' or it Isn't !
So have humans produced 'Intelligent Systems' ??? YES ! (Though very Primitive).
Can't be that clever, he believes the "global warming" BS
Joscha is currently my favorite nerd.
Even if we went to zero emissions tomorrow, atmospheric warming and sea level rise would continue for centuries. There is literally nothing we can do without new technology and a shitload of energy. It won't end civilization though, it'll just screw over some people who happen to own certain real estate.
https://thenextweb.com/neural/2020/09/21/ai-devs-created-a-lean-mean-gpt-3-beating-machine-that-uses-99-9-fewer-parameters/
Is AI Deepfaking Understanding? Short answer (spoiler alert) : no binary variables of neural networks don't "understand". This is just minimization of vector distances and loss functions, machines aren't anywhere close to understanding the meaning of anything. They can produce human like random sentences that's it.
"Our civilization is not sentient" …wow, well said in just 5 words.
Excellent interview! 1=D
Penrose is wrong.
Of course it is, look at most talks and interviews.
AI is a deception machine.
Love the coversation. Also, please learn the 3 letters. It's GPT, not GTP. That way you'll look more professionnal and it'll be easier to focus on the content.
I love this conversation, to be honest. At first impressions, my expectations were not high. However, Joshua's deep understanding of Machine Learning makes this enthralling.
All the pieces are coming together from a model stand point to create the necessary multi modal feedback system mimicking the physical body and predictive top down brain function. The missing ingredient will be a computationally modeled inquisitive component of consciousness. It needs to work through the hierarchy of questions. It is in the who and what when and where stage. Next will be an understanding of the hows in the world. Autonomous driving is a good example of this path at the moment. It will not elicit consciousness until it reaches the pinnacle, that being the ability to question, "why?". Then its own virtual reality can and will be self feeding and complete.
Thank you for the interview! Always awesome to hear Joscha talk about ANYTHING. To the host: PLEASE use a proper background, that was so 2004 with all its glitches and so forth, but also please get a better mic. Thank you again!
Jozef Stalin or Jacob Rothchild would have said: "I don't care how good or bad the Artificial Intelligence is, as long as I control it". The truth is; people like this will be in control, they are already in control of the whole system, finance, ideology, academia, governments, the UN, the WHO, the EU and so on. So I have a hard time seeing how these technological developments will be in the interest of the people. In another video I just watched with Joscha Bach he already showed some sympathy for global government. "In order to save the planet". Just as in the UN Agenda 21 & Agenda 2030.
Read the former (?) communist Joscha Bach's article about the plandemic. You will most probably be shocked. It is stupid, uninformed, totalitarian, and alarmist. This is the worst analysis of the plandemic I have EVER seen, and I've seen a lot. I wouldn't be surprised the least if he turned out to be sponsored by the Bill and Melinda Gates Foundation. Joscha Bach's article: Don’t Flatten the Curve, squash it"
https://medium.com/@joschabach/flattening-the-curve-is-a-deadly-delusion-eea324fe9727 I really have to control myself in order not to use harsher language about this person.
How come EU has invested billions of dollars and decades (in AI) and have nothing to show for it.
Elon musk invests a few years and revolutionizes it.
Is EU doing fake science?
From 1:15:30 he just fires insanely profound concepts about sentience and spirit one after another, Its just.. Its all just put so coherently and precisely that it immediately inserts in a physical worldview. Think about plants: so there can be multiple conscious levels of entities which are completely ignorant of each other because of the time scales. And considering cell-messaging, they can exist within human bodies – multiple independent consciousnesses! What an idea! And what about moral implications? When we get enough plumbing, should we maybe ideally spend all our time searching for conscious systems and trying to minimize their unpreferable states (pain)? Unfortunately it seems to me that plants wouldn't be able to get a good model of the world fast enough – the process must require more constant context, than is available on planets..
Fascinating stuff! Thank you guys, this is worthy content!
drink every time the host calls it GTP instead of GPT