> GPT-4 is ~200 Elo better than the next best semi-public Vicuna-13B in Chatbot Arena [2]. That is a non-zero moat
Its a non-zero advantage.
A moat is something that inhibits someone from closing an advantage.
(Also, its odd that the biggest models, outside of the big vendor centralized ones, they are testing are 13B-14B when 30B-ish and 65B-ish versions exist.)
However, if the advantage is due to things like inference infrastructure to support a massive model, that isn't easy to duplicate.
I would also say that the quality of these smaller models are good, but we also may not be measuring them correctly. Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1]. It's easy to have your inputs be inside a happy distribution for a paper but fail in the real world in ways that GPT-4 doesnt.
Lmsys would love to compare with bigger models but have limited resources. Contributions are welcome [2]
The moat (or the water/crocodiles in it) is the content that the company gathers in relation to the offering that is being defended. Microsoft has Github, which is a source of code that the model can operate upon, as well as the interactions/queries with the users. OpenAI is playing around with sharing content because of this. They want to build a moat and will use us to do it.
If someone just has an approach to solving the problem, i.e. code that does this that and the other, then there is no moat.
Its a non-zero advantage.
A moat is something that inhibits someone from closing an advantage.
(Also, its odd that the biggest models, outside of the big vendor centralized ones, they are testing are 13B-14B when 30B-ish and 65B-ish versions exist.)