Sorry for the plug, but I can't help myself everytime I read about task managers to notice you have to learn yet another DSL.
DSL are the plague of our field, for one that is useful like SQL, you have a hundred that turn your life into hell. Make's DSL is already full of gotchas, but templated YAML based DSL are just insanity. They have so many footguns, and so little flexibility and ability to be debugged.
After trying a lot of task runners and builders, I finally settled on doit:
- It makes the simple things simple, and the hard things possible.
- It scales up (you get targets, dependencies, manual scripting, etc), but it also scales down (you can define a simple list of shell commands to run and that's it).
- Instead of a DSL, you have regular Python. Which for all it's faults, is an excellent scripting language. It has great tooling support, and comes with a debugger.
- To avoid people abusing the expressiveness of Python, the default syntax to create tasks encourages a declarative style.
I've been using doit for years, and every single project I introduced it to was improved by it.
Of course, it comes with it's own problems (non python dev can dislike having to install that, and it doesn't yet provide a stand alone executable), but the ROI is way better than the alternatives I tried.
Python is such a massive dependency though, people who code in python rarely realize how big of a barrier it is for people not in that ecosystem. Package management is insane and every project has its own way of dealing with it. Installing python by itself is a chore on many operating systems.
I wish more projects would compile their Python tool into a stand alone executable so that the users wouldn't have to deal with the installation process.
I will at some point write about how to do that with Python, so that the knowledge will more broadly available and maybe the community will grow the habit of doing it.
Excellent, one of the problems with this is that you can't google it. Well, of course you can but you'll get a million contradicting resources and all that float to the top are too basic anyway. I really enjoyed your post, saved and will pass it along.
Thanks a lot, I planned to do it and I forgot. I've just added the link and it's indeed way better like this.
Substack doesn't really have any way to make a series of articles, or to bundle stuff together, so linking manually is the only thing you can do that will give a little structure.
One more reason to move off it when I find the time.
And it breaks over time. Absolutely terrible choice for makefile. It would be far better to just use bash, even though it would mean coding your own dependecy and update-detecting logic.
I wouldn't write anything in python that I still wanted to work a month later or on even one other system.
Python has no standard and the language implemented by the reference compiler is a moving target. (E.g. every script in Python 2 that used print without () broke in Python 3. There were other breaking changes. Some software still depends on installing Python 2.). The ecosystem in general is similarly dynamic. It's common to install a set of specific versioned packages and Python interpreter to get a package running.
It's not really a language where you can write something and be confident it'll still work in ten years with the latest release of Python that'll be shipping with distros.
> every script in Python 2 that used print without () broke in Python 3. There were other breaking changes. Some software still depends on installing Python 2
People had 15 years to move from 2 to 3.
I had to port dodo files from 2 to 3, it's a 10 minutes job. Most of the time, running 2to3 on it does the job automatically.
After all, you only use the stlib for task runners, delegating a lot of work to bash, and they are quite short.
> It's not really a language where you can write something and be confident it'll still work in ten years
I still have Python 2.7 projects running fine, and reinstalled one serving half million users a few month ago.
But even without this, Python was created in 1991, and Python 3 was released in 2008, that's 17 years between first release and the first big migration. To compare, go is 14 years old. In total.
Plus your build file will have been edited so many times by the end 10 years anyway.
Make has a lot going for it, but this is a terrible argument.
And yet, almost every random python script I come across does not work on my system, either because of modules or installation layout differences between the authors system and mine, or the passage of a mere few months or a year of time.
They are usually fixable of course. Assuming the thing worked on the authors system at all, it's possible to diagnose and fix it up for the current environment. But that has to be done, and sometimes takes more than a few minutes to hunt everything down.
And then one day I started seeing these pyc directories appearing out of the blue, right in whatever directory any script happened to be in. The same scripts that didn't do that yesterday. What an awesome thoughtfully engineered system!
The argument was not terrible, it was plain lived experience fact.
I don’t disagree with you, but I also don’t think that’s a bad thing. Things certainly don’t break quickly, there are years of warnings for deprecated functionality. I also think it’s not a bad thing to drop old functionality. It may not be warranted that code that worked 10/20 years ago should still work today. Languages evolve, and by pruning old features, footguns, or bad patterns, the language can evolve without carrying the weight of every mistake or legacy feature.
And most people will never ever encounter this article, so they will hit a missing command, a missing package, a PATH error, a PYHONPATH error, a version error or something like that. They may even try to solve it by copy/pasting many commands from the web, some of them with admin right, and make a mess of their setup.
That's why, for end user tools like doit, providing a stand alone executable is still the best solution.
Ruby package management is lacking in many ways. E.G: there is not good build in solution to manage projects and multiple versions of ruby on windows, it's all 3rd party. And C-extensions are still terrible to install
The apparent simplicity comes from the fact the ruby ecosystem is much smaller than the Python one, with 95% of it revolving around ror, so you are more often on the happy path and see less errors.
It started off with configuration files, but then we needed logic, control flow, or variables; and for some reason nobody goes "ok, maybe we need an actual progamming language now" and instead we've created horrors beyond our comprehension.
Azure Pipelines is the worst I've used: three types of variables, conditional statements return string "True" and "False", and of course it's mostly undebuggable.
I have hopes that one day something like CUElang (https://cuelang.org) will become popular to be the configuration language of everything. Just enough logic to get by, but not turing complete, plus strong typing and embedded schemas.
However, for now there is just a single implementation in GO.
Templated YAML just means that something somebody thought was configuration either wasn't or wasnt under every circumstance. It's not really a problem with YAML.
In most cases I think it's the fault of the creators of the YAML DSL (e.g. ansible) but with CI pipelines people really just should be writing scripts 100% in other languages and wiring them up to events with YAML. I dont have any sympathy for people using them as programming languages.
I'm sure Microsoft is happy to indulge it though because hey, vendor lock-in.
And yet you are here complaining about Taskfile and YAML overall. Nobody forces you to write full scripts in the YAML file. Just call scripts places somewhere next to it from it
But I'd do exactly the same with .phony targets in a Makefile. Keep the targets short and move larger chunks of code to individual script files that are just called from the Makefile
I just prefer Python to make, it's easier for me and most colleagues I met to write and to debug. And making a custom Python function when I need more than bash inside the dodo file is great.
This is the exact reason why I moved to magefiles[0] in my Go projects. I had a semi-working Makefile base I spent days honing years ago, but then I ran into a few walls (can't remember which).
Enter magefiles, I could use the same language in them as I could with my other code and stuff just worked.
No need to apologise, I don't like YAML, I like Python, and I agree: I'd rather have a domain specific library than a DSL anyday.
The best of both worlds is a good programming language that allows expressing domain specific libraries almost as if they were domain specific languages, without going so declarative that people don't know how to script anything anymore.
I don't know if Python/doit really achieved that, but its task definitions seem almost as readable as the YAML, in a real language where I can do anything I want without asking permission. So I would always choose doit over Taskfile.
But a better comparison is between doit and make. I much prefer make's basic syntax, although I dislike newer features which inevitably have stretched the syntax beyond its original simplicity, but conceivably you can achieve something similar to make in a language but as a library rather than a DSL, and it would be immediately friendlier and more powerful than make. Depending on what I was doing I would use doit, but for something simple I've still got make (and it would be easy to convert if desired, in fact I have some projects in mind that could do with this!).
Yes. I said it in another comment as well, but I'm rooting for a solid non turing complete conf language like CUElang (https://cuelang.org) to become popular. For now we can already use it to check and generate regular JSON or YAML, which is a good compat story.
But imagine having this in CI instead of YAML files.
I've tried the 3 of them, eventually I found CUElang was the best balance between expressiveness, ability to enforce and check for constraints, and easy of use.
But I would welcome any of them replacing the YAML mess we currently have in sysadmin.
For Pythonistas or people interested in Python, I can't plug the commenter's newsletter enough: it is an absolute treasure trove of good advice for new and old users of Python. They did a whole series on the right way to install and use Python [1]. If you follow their advice you'll never struggle with any of the problems people complain of encountering with Python.
> DSL are the plague of our field, for one that is useful like SQL, you have a hundred that turn your life into hell.
The same can be said about using general purpose programming languages to specify stuff declaratively, but instead of implementation the problems happen in each and every single use of it. It only takes a single user that either does not comply with standards or feels they are particularly clever to mess everything for everyone.
I guess that could work for a Python project, where you already have Python installed. But still you would need to set up a venv or similar to keep your system Python clean. Then you cannot run the tasks without creating that venv or similar. How to create the venv and install all the correct dev dependencies into it? You can't use the task runner yet, because you still need to install that ... Maybe a Makefile?
DSL are the plague of our field, for one that is useful like SQL, you have a hundred that turn your life into hell. Make's DSL is already full of gotchas, but templated YAML based DSL are just insanity. They have so many footguns, and so little flexibility and ability to be debugged.
After trying a lot of task runners and builders, I finally settled on doit:
https://www.bitecode.dev/p/doit-the-goodest-python-task-runn...
- It makes the simple things simple, and the hard things possible.
- It scales up (you get targets, dependencies, manual scripting, etc), but it also scales down (you can define a simple list of shell commands to run and that's it).
- Instead of a DSL, you have regular Python. Which for all it's faults, is an excellent scripting language. It has great tooling support, and comes with a debugger.
- To avoid people abusing the expressiveness of Python, the default syntax to create tasks encourages a declarative style.
I've been using doit for years, and every single project I introduced it to was improved by it.
Of course, it comes with it's own problems (non python dev can dislike having to install that, and it doesn't yet provide a stand alone executable), but the ROI is way better than the alternatives I tried.