Transformers still didn't hit their scaling limits wrt number of tokens trained on and the number of layers. Model size limits are now purely given by finances ($100M/training run still seems to be too excessive).