I think your confusion is based on the assumption of microservices. If you're deploying something which shuffles packets, yes, containers are great for quickly spawning instances and getting more utilization on a given host. Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant. (There's a similar point in overhead: unless you're doing privileged calls in a loop, hypervisor overhead hasn't been significant for at least a decade unless you're talking about something like VirtualBox)
Note that I'm not saying containers are bad, just that the people who disagree are almost certainly speaking from experience in different environments where other trade offs make sense compared to where you're working.
> Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant
If you spin up the VM once and it runs for days/weeks that is true. In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy. That's where VM startup time becomes an issue.
> In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy.
That's kind of my point: that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments. If you're doing anything which copies much data or does non-trivial initialization work, saving a minute by launching a container vs. a new VM is still a healthy improvement but it's not the expensive part of a new deployment and unless you're churning many jobs per hour it's a rounding error in the total execution time.
(I'd still prefer a container for that kind of workload but it'd be for the consistency & simplicity aspects)
> that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments.
That might be true for enterprise (I simply don't know) but in the startup world these are these are actually the most common workloads: Stateless apps that scale vertically.
I agree that startup time doesn't matter much if you need to do lot of initializations. But in my experience in most infrastructures these are a few databases of which there are orders of magnitude less than other workloads.
Note that I'm not saying containers are bad, just that the people who disagree are almost certainly speaking from experience in different environments where other trade offs make sense compared to where you're working.