Cylc is a GPL system written in Python, and used in some countries (Au/NZ, UK, South Africa, Korea, other institutions in Europe and US) mainly to run cyclic workflows, e.g. forecast models.
It supports the same syntax as Cron for scheduling jobs (called suites in Cylc) but with the concept of time points. So you are able to configure a workflow with some tasks that would run every midnight, other tasks that would run every 5 minutes. And in case any of these tasks fail, Cylc has a mechanism to restart the workflow and catch-up with these tasks.
There are a couple of commercial systems that provide similar features, and also Luigi, Airflow, but with limitations (mainly around cyclic tasks, and the catch-up mechanism).
There is a new web GUI under development, and questions, suggestions, or issues/pull requests welcome!
Its web site is https://cylc.github.io/cylc/ but we also just (yesterday) acquired cylc.org and will start updating the site and documentation soon.
Cylc is a GPL system written in Python, and used in some countries (Au/NZ, UK, South Africa, Korea, other institutions in Europe and US) mainly to run cyclic workflows, e.g. forecast models.
It supports the same syntax as Cron for scheduling jobs (called suites in Cylc) but with the concept of time points. So you are able to configure a workflow with some tasks that would run every midnight, other tasks that would run every 5 minutes. And in case any of these tasks fail, Cylc has a mechanism to restart the workflow and catch-up with these tasks.
There are a couple of commercial systems that provide similar features, and also Luigi, Airflow, but with limitations (mainly around cyclic tasks, and the catch-up mechanism).
There is a new web GUI under development, and questions, suggestions, or issues/pull requests welcome!
Its web site is https://cylc.github.io/cylc/ but we also just (yesterday) acquired cylc.org and will start updating the site and documentation soon.
Cheers