Architecture improvements continue, but if you look at the single-core performance curve, it has been much flatter since 2004.
The "won't ever get faster" part should be qualified, though, with "at least not on silicon". There is still the possibility of a breakthrough using some other semiconductor -- silicon germanium, buckytubes, or something. Any such thing is years away, admittedly.
Unless the breakthrough involves making arbitrary-thickness chips, it's only going to be a modest improvement; the best possible density/frequency for any realistic material isn't that many half-nodes better than the best for silicon.
The "won't ever get faster" part should be qualified, though, with "at least not on silicon". There is still the possibility of a breakthrough using some other semiconductor -- silicon germanium, buckytubes, or something. Any such thing is years away, admittedly.