Ask anyone who has hired someone who says all the right things and seems intelligent, but has no experience or skills in what they actually talk about.
When I write code, I don’t just focus on solving the problem at hand, I think about things like, like: how will another human interpret this, how maintainable will this be, what are the pitfalls down the line, what are the consequences of this, any side effects, performance implications, costs, etc… things GPT does not know.
And your point about humans lying about knowledge only to be found inexperienced is quite the opposite of an LLM (albeit there is the hallucination problem, but GPT-4 is a massive improvement there):
These models do have “experience” aka their training data. And I would argue with most every one of your examples of things that GPT doesn’t know.
You can ask it about performance implications, side effects, costs. It’s quite good at all that right now even! Imagine the future just a few years out.
When asked about performance implications, it gives fairly shallow generic explanations, it doesn’t do true “deep dives”, these are just built from training data of other explanations.
There is no “getting better” from this. If you gave a monkey a type writer and it occasionally typed words randomly you wouldn’t say “Wow this is just what it can do now, imagine several years out!”
Continue asking it to provide details and it can. Or , prior to asking it about performance, ask it to respond with as much detail as it can and have it include details you specially want to see.
Comparing GPT-4 to a monkey with a typewriter , and claiming the absolute of “there’s no getting better from this” when we’ve literally seen dramatic progress in just months?
I think you’re missing out on some of the utility this stuff can actually provide .
No, you see it needs to do these things on its own, unprompted. It has to consider multiple solutions to problems it encounters and choose the best one, not just the most probable one. It’s not made to evaluate things that way, you can’t hand it multiple implementations and ask it to weigh the pros and cons of the different approaches and recommend the best one for what you’re trying to do. You can’t hand it your code for code review and ask what you could improve and expect to get a response that isn’t just fabricated from what other people have said in code reviews.
And it will never do those things, because it’s an LLM and there are limits to what LLMs can do. There is no “getting better”, it will only sound better.
If it’s going to replace programming, the prompts simply cannot be more laborious than writing the damn code yourself in the first place.
Think 50 LLMs with different personalities and focus points talking to each other, mixed with stuff like Wolfram. You can instruct them to “invoke” tools. An outside system parses their “tool use” and injects results. You can get quite crazy with this.
LLMs are just the part of a much larger looping system that can do these things you speak of. Be active and seek out stuff. Of course, it’s all illusory, but I’m sorry I think it’s no different with myself.
By the way, it actually gives ok reviews on novel code, so I’m not sure what you mean. At some point nothing is truly novel, even innovation is composing existing “patterns” (at whatever abstraction level).
> There is no “getting better” from this. If you gave a monkey a type writer and it occasionally typed words randomly you wouldn’t say “Wow this is just what it can do now, imagine several years out!”
So thinking that chatGTP could gain understanding is as crazy as the idea that primates could learn to use tools or type words?
When I write code, I don’t just focus on solving the problem at hand, I think about things like, like: how will another human interpret this, how maintainable will this be, what are the pitfalls down the line, what are the consequences of this, any side effects, performance implications, costs, etc… things GPT does not know.