Techmeme Podcast
A lot of folk in Silicon Valley have decided that GPT-3 might herald a next big thing in tech. OpenAI is an AI research foundation started by Elon Musk, Sam Altman and a who’s who of big names in machine learning. GPT-3 is OpenAI’s text-generating AI that debuted about a month ago, and is a language model trained with 175 billion parameters, a successor to GPT-2 which had only 1.5 billion parameters. In short, remember how we’ve discussed on this show before how AI can’t read and it can’t speak. Yeah, well…
https://delian.substack.com/p/quick-thoughts-on-gpt3 Quick thoughts on GPT3 (Delian Asparouhov)
http://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html Giving GPT-3 a Turing Test (Kevin Lacker)
are you on spotify?
Will it predict my 3rd wife is gonna shaft me like the previous two cold hearted wenchs did??
Can it answer the question
"Who was the president when Obama was born" ?
Google can not.
LOL
… but can it disambiguate language?
…or it just converts some 'A' to some 'B'?
It is well overdue to remove the human from the process of software development.
Automate the software development
What is human learning but a slow approximation towards a mean, with a plateau when we master something?
It’s a mistake to assume that we don’t learn the same way.
Here's my pet peeve: when people say things like "the AI is just predicting text with statistics, it doesn't truly understand anything". I argue that HUMANS do the same thing, and it is inherently understanding to be able to do so. 90% of what we humans learn is through effectively an advanced predictive text. If you take away PE and science labs, nearly ALL of school learning is just learning from reading and hearing other people (usually teachers) say words. The ability to take in all those words and, later, to predict the answers to questions from that is understanding and comprehension.
It's true that humans have other methods of learning that GPT doesn't have, such as sight and various tactile inputs. But the lack of those doesn't imply it can't be intelligent. For instance, consider the following hypothetical: you ask GPT-3, "what color is the sky?" and it says, "blue". You claim it doesn't truly understand that the sky is blue, it's just repeating what it read about the sky. But then you ask a blind person, "what color is the sky?" and they say, "blue". Will you claim that the blind person is incapable of truly understanding that the sky is blue because they can't look up and see it? Or would you instead say that the blind person's ability to use knowledge they've heard and read from other people does, in fact, confer understanding to them? I'd argue the latter. But this is really just a modern formulation of the age-old philosophical P-Zombie problem. The problem goes like this: say you have a creature or machine that looks human in every way. No matter how you examine it, it matches with humans. It behaves exactly as you'd expect a person to behave: says "ow" when pricked with a needle, smiles when it sees a puppy, etc. How do you know whether this "philosophical zombie" (P-Zombie) actually has experiences? How do you know it has subjective qualia or understanding? How could you ever test or prove that it does or does not? Computer AI is exactly this. You can say "it's just mimicking humans", but you also can't really show that this is not exactly what humans already do. Everyone around you could just be "mimicking humans" and you'd never know. In fact, YOU could just be a P-Zombie and not know it, either, believing your statistically modeled behavior to be something more.
Honestly, other than scale, I think GPT-3's biggest obstacle is that its short-term memory is drastically restricted. By this I mean it has learned from a massive amount of text online — the pre-training — but after that, its weights are set. It is prevented from learning anything more. After it gives a response, it immediately forgets anything it just read or replied. You can work around this to some degree by passing the entire context, plus the previous output, back into GPT-3 as the new context for each request, but that's just forcing it to "relearn" the same conversation over and over, and the total context + output is limited to 2048 tokens maximum, so even that can't go for as long as a human's short-term memory could. If we truly want to see what GPT-<N> is capable of, it needs to be able to continue adjusting its weights with each request. I believe OpenAI has said they're working on and/or thinking of adding the ability to fine-tune the network, which is exactly what I'm describing; someone on Hacker News suggested adding a couple of layers onto the network for each application that can be trained as a sort of module. But however it's done, forcing GPT-3 to forget all new data it reads after its pre-training, by not allowing new requests/contexts to backpropagate through the network, will always limit its capabilities.