People like the idea of isolation that containers (LXC) provide. We don't have to worry about stepping on toes, my container works with any number of other containers running on the same host, not sometimes, all times.
The env is moving fast, Kubernetes and Helm are on a tear. It's also worth taking another look at cluster standup CoreOS+Vagrant/Tectonic/Minikube/kubeadm are filling in gaps.
The VM overhead is ludicrous compared to LXC containers though, even with all that pass-through that modern hypervisors do. I don't get all the hate for containers from the VM folk, those technologies are by no means exclusive, it's just that spinning up an entire virtual machine to run some microservice strikes me as wasteful.
I think your confusion is based on the assumption of microservices. If you're deploying something which shuffles packets, yes, containers are great for quickly spawning instances and getting more utilization on a given host. Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant. (There's a similar point in overhead: unless you're doing privileged calls in a loop, hypervisor overhead hasn't been significant for at least a decade unless you're talking about something like VirtualBox)
Note that I'm not saying containers are bad, just that the people who disagree are almost certainly speaking from experience in different environments where other trade offs make sense compared to where you're working.
> Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant
If you spin up the VM once and it runs for days/weeks that is true. In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy. That's where VM startup time becomes an issue.
> In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy.
That's kind of my point: that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments. If you're doing anything which copies much data or does non-trivial initialization work, saving a minute by launching a container vs. a new VM is still a healthy improvement but it's not the expensive part of a new deployment and unless you're churning many jobs per hour it's a rounding error in the total execution time.
(I'd still prefer a container for that kind of workload but it'd be for the consistency & simplicity aspects)
> that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments.
That might be true for enterprise (I simply don't know) but in the startup world these are these are actually the most common workloads: Stateless apps that scale vertically.
I agree that startup time doesn't matter much if you need to do lot of initializations. But in my experience in most infrastructures these are a few databases of which there are orders of magnitude less than other workloads.
minikube was not quite primetime when I tried it, but I saw release notes a few weeks later that fixed everything I ran into. (IIRC it was ingress or DNS problems).
Your impressions about Kubernetes sounded quite out of date when I read your post. I strongly suggest you give things another try. Things move fairly fast in this area.
Yep with today's minikube, you can do ingress. For example, I'll point myproject.minikube.local to 192.168.99.100 and use the stable/nginx-ingress helm chart.
When I'm running my application stack on minikube, I can set a helm template variable to myproject.minikube.local.
The nginx ingress controller will route the myproject traffic to the proper backing service.
On prod, I'll set the template variable to super-social-money-maker.com and turn up the stack with a helm install. Here, ELB does the ingress for me, but as an app developer, I don't have to be concerned with the particulars of the cluster's ingress implementation.
The env is moving fast, Kubernetes and Helm are on a tear. It's also worth taking another look at cluster standup CoreOS+Vagrant/Tectonic/Minikube/kubeadm are filling in gaps.