With Kubernetes v1.24, non-expiring service account tokens are no longer auto-generated. This blog post highlights what this means in practice, and what to do if you rely on non-expiring service account tokens.
Mesos started as a research project at Berkeley in 2009 and was originally focused on cluster computing frameworks like Hadoop. From the paper: "We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI."
It actually predates YARN by a few years.
But, it very quickly (in 2010) saw production use at Twitter as the foundation for Twitter's custom PaaS which was later open sourced as Apache Aurora.
Marathon's main use case was actually for running microservice application in containers, which is why it has some advanced features around managing groups of containerized apps and their dependencies. The "meta-framework" use case for launching custom frameworks was also important but basically just needs Marathon to keep a container alive. Mesosphere never made Marathon proprietary. The full code is still OSS here: https://github.com/mesosphere/marathon/
Our commercial product DC/OS just added advanced workflows through a UI on top, and better integration with the rest of the components around Mesos.
ML is a relatively young field, and decades behind Software Engineering in terms of best practices for running production systems. CI/CD massively improved the innovation cycle time and quality of production software, and I believe it is key for building robust production ML systems as well. CML looks like a really easy to use product for bringing CI/CD to ML projects.
Thanks for the kind words! DevOps has had profound results and we think culturally and technically, it's the right moment to push for those practices in ML.
You're correct in that GCP runs k8s in VMs, DC/OS doesn't. What's similar is that there's a resource manager underneath -
Borg for GCP, Mesos for DC/OS. They serve similar purposes like resource management, isolation, and operating the services on top.
>"What's similar is that there's a resource manager underneath - Borg for GCP, Mesos for DC/OS."
Maybe I don't fully understand DC/OS then. I was under the impression that DC/OS was simply a distro for Mesosphere. But your comment make me think that either my understanding is incorrect or else DC/OS has become something more than a Mesos distro. Could you elaborate? Thanks.
This is correct: "DC/OS is better for running stateful services and then you can use K8 to run your stateless services"
Data services run directly on DC/OS via application-aware schedulers. They have the operational logic for how to bring up say a Cassandra cluster correctly, how to upgrade it to a new version without breaking it, change config, scale up, etc. All things you usually have to figure out yourself. When you run k8s on DC/OS you get these same benefits.
For most people k8s (like any other distributed system) is pretty hard to operate and wrap their head around. The goal of DC/OS is to make building and operating distributed systems easy. Mesos is pretty different from k8s. It doesn't have an API for microservice developers for example, instead it has low level primitives similar to an operating system kernel, on which you can build any software. We recently launched an SDK to make that even easier, along with a bunch of open source and commercial software that uses it, for example Cassandra/Datastax, Kafka/Confluent, Redis, Elastic, Couchbase, Alluxio.
So DC/OS ends up looking a lot more like an entire cloud platform like GCP, which offers managed k8s, a managed database (BigTable), machine learning and so on. Under the hood it all runs on Google's proprietary Borg, which has some similarities with Mesos (Google is a sponsor of the lab where Mesos was created).
Could you please disclose in what time frame would be available "managed k8s" through Universe or in some other way. Cause currently there is an issue with using kube-mesos-framework and DC/OS as I got.
Running workload-specific schedulers like Kubernetes on Mesos is one of its fundamental ideas. From the paper:
"It seems clear that new cluster computing frameworks
will continue to emerge, and that no framework will be
optimal for all applications. Therefore, organizations
will want to run multiple frameworks in the same cluster,
picking the best one for each application."
My favorite part is that moving Siri to Mesos actually had a noticeable impact on Siri's response time. It didn't just make the lives of Apple engineers easier but improved the product as well. For every one of the more than 700 million iPhone users worldwide.
