If one isn't using the tools available from the OS or even the OS regularly, it's tough to know that they are there. I picked on cron/syslog/apt as there aren't too many times I see them used on server fleets these days. That's coming from 10 years working in both SRE/systems and Software Eng roles on fleets of 1k+ servers both in and out of clouds. It's really diffiult to convince new engineers that even having a persistent process is a good idea vs. using an orchestration service.
As for the syslog/cron/apt examples I'll share some more detail below.
- Syslog Isn't used as it's pretty straightforward to pull in ones favorite language specific log library, combined with ones favorite log tool of choice and be off to the races.
- Cron doesn't get used as it can only easily handle scheduling on a specific host. Whereas there are a plethora of distributed schedulers which will happily run commands on any number of hosts.
- Apt/yum don't get used as folks want to have their builds to be repeatable, and rely on software built in via their dependency management system of choice and deployed via fat statically linked binary.
> Whereas there are a plethora of distributed schedulers which will happily run commands on any number of hosts.
If you have any specific suggestions, I am on the lookout. I checked out tools like airflow and dask but did not see what they added. Trying not to go full blown HPC cluster.
HashiCorp Nomad[0] can do scheduled periodic tasks across a cluster of machines. Much lighter weight than Kubernetes, but a shotgun flyswatter when compared to cron.
If you're on a PaaS solution, I'd take a look at the built-in scheduler e.g. scheduled tasks in Kubernetes, fargate or scheduled tasks in Mesos via Singularity or other frontend.
If you aren't using a PaaS, take a look at hashicorp's nomad tool. It's worth noting that if your team uses microservices or just likes making lots of services, a self-hosted or cloud PaaS solution will save you inordinate amounts of time when it's used by more than 20 engineers.
Cognitive overhead perhaps. I hope the new simplified platform is actually good and not an ad hoc, informally-specified, bug-ridden, slow implementation of half of Unix, though.
