There is a neat potential speedup here for the case where the bandwidth to your model weights is the limiting factor.
If you have a guess what the model will output, then you can verify that your guess is correct very cheaply, since you can do it in parallel.
That means there is the possibility to have a highly quantized small model in RAM, and then use the big model only from time to time. You might be able to get a 10x speedup this way if your small model agrees 90% of the time.
It looks like a description of Speculative Sampling. There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course.
The potential for speedup according to their paper is closer to 2x than 10x however.
If you have a guess what the model will output, then you can verify that your guess is correct very cheaply, since you can do it in parallel.
That means there is the possibility to have a highly quantized small model in RAM, and then use the big model only from time to time. You might be able to get a 10x speedup this way if your small model agrees 90% of the time.