I think the hold-out paradigm is woefully overrated in part for similar reasons. It's always seemed odd to me to emphasize hold out samples when you know their asymptotic performance in the form of fit statistics.

I'm always for examining replicability, but the current paradigm to me seems misguided and this articulates some of the reasons very well.