On practice, my mother pays 30€ a month of electricity in France, where nuclear energy is everywhere. She asked for a quote to get panels on her house, and they got back to her with 20000€.
Sure you can move the needle by noting I drain way more power than her, than she heats her house using fossil fuel and that the quotes could be have been reduced in many ways.
You can also note that solar panels have to be replaced several times, take much more space, don't have to including wiring in their ROI calculation (while nuclear does for some weird reason) and are created by electricity generated by fossil or nuclear fuel, so their building price is already cheap.
I mean that in France, the reports assessing the cost of the energy produced by wind turbines and solar panels don't include the wiring from the source of energy to where the energy is distributed.
Which is logical.
For some reason I don't know, nuclear reactors cost evaluation must include the wiring.
99% of everything ever written is crap, so you need filtering the huge quantity of brain manure the humanity keeps producing.
And hand written blogs by somebody for several years is very high in my filter list to check out something.
So I care very much about your blog.
Besides, the best way to make sure there is more quality content in the world is to produce it. And that's why I write https://bitecode.dev. Because I want others to have what I wish I had when I needed it.
Sorry for the plug, but I can't help myself everytime I read about task managers to notice you have to learn yet another DSL.
DSL are the plague of our field, for one that is useful like SQL, you have a hundred that turn your life into hell. Make's DSL is already full of gotchas, but templated YAML based DSL are just insanity. They have so many footguns, and so little flexibility and ability to be debugged.
After trying a lot of task runners and builders, I finally settled on doit:
- It makes the simple things simple, and the hard things possible.
- It scales up (you get targets, dependencies, manual scripting, etc), but it also scales down (you can define a simple list of shell commands to run and that's it).
- Instead of a DSL, you have regular Python. Which for all it's faults, is an excellent scripting language. It has great tooling support, and comes with a debugger.
- To avoid people abusing the expressiveness of Python, the default syntax to create tasks encourages a declarative style.
I've been using doit for years, and every single project I introduced it to was improved by it.
Of course, it comes with it's own problems (non python dev can dislike having to install that, and it doesn't yet provide a stand alone executable), but the ROI is way better than the alternatives I tried.
Python is such a massive dependency though, people who code in python rarely realize how big of a barrier it is for people not in that ecosystem. Package management is insane and every project has its own way of dealing with it. Installing python by itself is a chore on many operating systems.
I wish more projects would compile their Python tool into a stand alone executable so that the users wouldn't have to deal with the installation process.
I will at some point write about how to do that with Python, so that the knowledge will more broadly available and maybe the community will grow the habit of doing it.
Excellent, one of the problems with this is that you can't google it. Well, of course you can but you'll get a million contradicting resources and all that float to the top are too basic anyway. I really enjoyed your post, saved and will pass it along.
Thanks a lot, I planned to do it and I forgot. I've just added the link and it's indeed way better like this.
Substack doesn't really have any way to make a series of articles, or to bundle stuff together, so linking manually is the only thing you can do that will give a little structure.
One more reason to move off it when I find the time.
And it breaks over time. Absolutely terrible choice for makefile. It would be far better to just use bash, even though it would mean coding your own dependecy and update-detecting logic.
I wouldn't write anything in python that I still wanted to work a month later or on even one other system.
Python has no standard and the language implemented by the reference compiler is a moving target. (E.g. every script in Python 2 that used print without () broke in Python 3. There were other breaking changes. Some software still depends on installing Python 2.). The ecosystem in general is similarly dynamic. It's common to install a set of specific versioned packages and Python interpreter to get a package running.
It's not really a language where you can write something and be confident it'll still work in ten years with the latest release of Python that'll be shipping with distros.
> every script in Python 2 that used print without () broke in Python 3. There were other breaking changes. Some software still depends on installing Python 2
People had 15 years to move from 2 to 3.
I had to port dodo files from 2 to 3, it's a 10 minutes job. Most of the time, running 2to3 on it does the job automatically.
After all, you only use the stlib for task runners, delegating a lot of work to bash, and they are quite short.
> It's not really a language where you can write something and be confident it'll still work in ten years
I still have Python 2.7 projects running fine, and reinstalled one serving half million users a few month ago.
But even without this, Python was created in 1991, and Python 3 was released in 2008, that's 17 years between first release and the first big migration. To compare, go is 14 years old. In total.
Plus your build file will have been edited so many times by the end 10 years anyway.
Make has a lot going for it, but this is a terrible argument.
And yet, almost every random python script I come across does not work on my system, either because of modules or installation layout differences between the authors system and mine, or the passage of a mere few months or a year of time.
They are usually fixable of course. Assuming the thing worked on the authors system at all, it's possible to diagnose and fix it up for the current environment. But that has to be done, and sometimes takes more than a few minutes to hunt everything down.
And then one day I started seeing these pyc directories appearing out of the blue, right in whatever directory any script happened to be in. The same scripts that didn't do that yesterday. What an awesome thoughtfully engineered system!
The argument was not terrible, it was plain lived experience fact.
I don’t disagree with you, but I also don’t think that’s a bad thing. Things certainly don’t break quickly, there are years of warnings for deprecated functionality. I also think it’s not a bad thing to drop old functionality. It may not be warranted that code that worked 10/20 years ago should still work today. Languages evolve, and by pruning old features, footguns, or bad patterns, the language can evolve without carrying the weight of every mistake or legacy feature.
And most people will never ever encounter this article, so they will hit a missing command, a missing package, a PATH error, a PYHONPATH error, a version error or something like that. They may even try to solve it by copy/pasting many commands from the web, some of them with admin right, and make a mess of their setup.
That's why, for end user tools like doit, providing a stand alone executable is still the best solution.
Ruby package management is lacking in many ways. E.G: there is not good build in solution to manage projects and multiple versions of ruby on windows, it's all 3rd party. And C-extensions are still terrible to install
The apparent simplicity comes from the fact the ruby ecosystem is much smaller than the Python one, with 95% of it revolving around ror, so you are more often on the happy path and see less errors.
It started off with configuration files, but then we needed logic, control flow, or variables; and for some reason nobody goes "ok, maybe we need an actual progamming language now" and instead we've created horrors beyond our comprehension.
Azure Pipelines is the worst I've used: three types of variables, conditional statements return string "True" and "False", and of course it's mostly undebuggable.
I have hopes that one day something like CUElang (https://cuelang.org) will become popular to be the configuration language of everything. Just enough logic to get by, but not turing complete, plus strong typing and embedded schemas.
However, for now there is just a single implementation in GO.
Templated YAML just means that something somebody thought was configuration either wasn't or wasnt under every circumstance. It's not really a problem with YAML.
In most cases I think it's the fault of the creators of the YAML DSL (e.g. ansible) but with CI pipelines people really just should be writing scripts 100% in other languages and wiring them up to events with YAML. I dont have any sympathy for people using them as programming languages.
I'm sure Microsoft is happy to indulge it though because hey, vendor lock-in.
And yet you are here complaining about Taskfile and YAML overall. Nobody forces you to write full scripts in the YAML file. Just call scripts places somewhere next to it from it
But I'd do exactly the same with .phony targets in a Makefile. Keep the targets short and move larger chunks of code to individual script files that are just called from the Makefile
I just prefer Python to make, it's easier for me and most colleagues I met to write and to debug. And making a custom Python function when I need more than bash inside the dodo file is great.
This is the exact reason why I moved to magefiles[0] in my Go projects. I had a semi-working Makefile base I spent days honing years ago, but then I ran into a few walls (can't remember which).
Enter magefiles, I could use the same language in them as I could with my other code and stuff just worked.
No need to apologise, I don't like YAML, I like Python, and I agree: I'd rather have a domain specific library than a DSL anyday.
The best of both worlds is a good programming language that allows expressing domain specific libraries almost as if they were domain specific languages, without going so declarative that people don't know how to script anything anymore.
I don't know if Python/doit really achieved that, but its task definitions seem almost as readable as the YAML, in a real language where I can do anything I want without asking permission. So I would always choose doit over Taskfile.
But a better comparison is between doit and make. I much prefer make's basic syntax, although I dislike newer features which inevitably have stretched the syntax beyond its original simplicity, but conceivably you can achieve something similar to make in a language but as a library rather than a DSL, and it would be immediately friendlier and more powerful than make. Depending on what I was doing I would use doit, but for something simple I've still got make (and it would be easy to convert if desired, in fact I have some projects in mind that could do with this!).
Yes. I said it in another comment as well, but I'm rooting for a solid non turing complete conf language like CUElang (https://cuelang.org) to become popular. For now we can already use it to check and generate regular JSON or YAML, which is a good compat story.
But imagine having this in CI instead of YAML files.
I've tried the 3 of them, eventually I found CUElang was the best balance between expressiveness, ability to enforce and check for constraints, and easy of use.
But I would welcome any of them replacing the YAML mess we currently have in sysadmin.
For Pythonistas or people interested in Python, I can't plug the commenter's newsletter enough: it is an absolute treasure trove of good advice for new and old users of Python. They did a whole series on the right way to install and use Python [1]. If you follow their advice you'll never struggle with any of the problems people complain of encountering with Python.
> DSL are the plague of our field, for one that is useful like SQL, you have a hundred that turn your life into hell.
The same can be said about using general purpose programming languages to specify stuff declaratively, but instead of implementation the problems happen in each and every single use of it. It only takes a single user that either does not comply with standards or feels they are particularly clever to mess everything for everyone.
I guess that could work for a Python project, where you already have Python installed. But still you would need to set up a venv or similar to keep your system Python clean. Then you cannot run the tasks without creating that venv or similar. How to create the venv and install all the correct dev dependencies into it? You can't use the task runner yet, because you still need to install that ... Maybe a Makefile?
I would agree with the other comment that the fact we can measure gravitational waves at all using pulsar timing with data collected over decades is what is mind-blowing, more than the actual waves being detected.
However, depending on their nature, there is a possibility that this is still a pretty significant event. When we observed the cosmic microwave background, because this is light emanating from everywhere at the very first time photons decoupled from ionized matter and the mean free path exceeded the particle horizon (the universe "became transparent" as they say), this is the furthest back you ever see using light. A much earlier event can possibly be observed via the cosmic neutrino background. The universe would have been transparent to neutrinos much earlier, when the electroweak force first decayed into the weak and electromagnetic forces. This is a big difference, fractional seconds after the Big Bang compared to I think 379,000 years or so for light.
However, the very earliest event that could ever be observed by any means at all would be the cosmic gravitational background expected to exist when gravity first decoupled from strong and electroweak forces. If that's what this is, we're not seeing the literal beginning, but it's the closest we can ever get by any sensor we currently have the ability to conceive of.
One lesson from the history of science is that new developments in detection and sensing capabilities often herald tremendous increases in knowledge.
Lenses, and the telescope and microscope, both advanced astronomy and biology tremendously. Interferometry grows out of optics, spectroscopy, and the ability to detect interference at scales. Chemical tests, electromagnetic principles, radiation, and the ability to detect electromagnetic radiation outside the visible spectrum (infrared, radio, ultraviolet, x-ray, gamma ray) had similarly large influences. Another area of progress is increased precision in measuring both space and time, which this discovery addresses.
I've also done some fairly casual eyeball-assessments of trends in Nobel-awarded research in physics and chemistry since roughly 1960, which is a convenient demarcation between the period of profound discovery through the earlier part of the 20th century, with fundamental laws and particles making frequent mention, to the past 60 or so years in which it's often been measurement or emission capabilities (think LIGO and blue LEDs) which have been the subject of awards. In one sense, these seem less significant than the earlier work, and that's been my own tendency in ascribing heft. But if this does in fact represent a gestation period in which new observational tools and techniques are being cultivated, there might well be more impact than appears initially.
We're in what may well be the early stages of gravitational wave measurement and detection, and we're figuring out how to "build" galactic-- and intergalactic-scale measurement "devices" which are necessary to detect this pervasive but very low-energy effect. It is exciting, and I'm thinking over the past record of sensing technologies to think how these might inform us. My initial read is that gravitational waves are very low frequency, which is to say, low resolution, but because they have extraordinarily high upper bounds on magnitude (unlike, say, magnetic or electromagnetic events associated with stars, galaxies, and even black-hole and neutron-star interactions, though the latter play a strong role in gravitational wave phenomena), what I suspect we'll find is a general sensing capability which can "see through" cosmic structures that are otherwise opaque to us (gravity isn't bothered by dust and other matter), and also provides a mechanism for directly observing dark matter, which is defined by the fact that it does interact gravitationally but does not with the electromagnetic spectrum.
That's a lay view, no specific expertise, but informed by a model of technologies and capabilities afforded, one element of which is information, itself comprised of sensing, processing, storage and recall, and transmission.
No, this is about the nanohertz gravitational wave "background", which comes from massive binaries with orbital periods of on the order of months to years. Sources are expected to be be randomly arranged in spherical coordinates (up to an unfortunately low radial distance cutoff), like the (low-redshift) galaxies they likely live in the middle of.
Primordially, most still-viable types of inflation produce gravitational waves on all length scales, so some very short wavelength primordial GWs are in principle accessible to us despite the large cosmological redshift (z >> 10^10; 10^10 is about the redshift at neutrino decoupling, a good guess is that z ~ 10^25 is the ballpark for the latest high-amplitude-peak (~ 10^-17 Hz) inflation-produced gravitational radiation[1]). If multimessenger and other techniques can help subtract the "foreground" of binary black holes and the like, then perhaps we will see them as redshifted-to-light-year-wavelengths not traceable to a hard SMBH binary or foreground mimicers.
The CMB, particularly in its B-mode polarization, is already a useful probe of primordial gravitational radiation as was anticipated at least as early as BOOMERanG's 2003 flight. A better measurement of (dust and gas, and also some gravitational lensing) foregrounds is important there too.
PS: Bécsy, Cornish & Kelley 2022 (open access at The Astrophysical Journal, but easiest to grab the arxiv version which matches the published paper) https://arxiv.org/abs/2207.01607 is a very good overview of the nanohertz GW background(s) and how they're accessible to pulsar timing arrays. A couple of the recent coordinated-release papers by the various intl PTA members are also good overviews https:///www.ipta4gw.org/
I was referring more to the measurement of slight variations in pulsar frequencies across decades used to detect background gravitation waves - I'm afraid I have no context besides this article to ELY5 the science they're trying to prove.
You'd have to go further than that. The actual law says:
>specifically designed to operate primarily in an environment that is
regularly subject to splashing water, water streams or water immersion, and that are
intended to be washable or rinseable
An iPhone is specifically designed to cope with primarily outdoor use. It rains outdoors. Outdoors is "an environment that is regularly subject to splashing water".
This is a lesson I keep repeating to our juniors: when you do something notable, celebrate it, and make sure to include the whole team, the users and the boss in the celebration.
It can be as simple as an email, but sometimes, making a few chat messages or even calls. It takes times, it seems silly, but you get a fantastic ROI on this.
Because now people see your work as valuable. It's concrete to them. They understand its value. And this has many positive consequences on your life.
This, of course, works only if the celebration avoids technical terms, is short, straight to the point, and explain what's the benefit to them and the org and how much it took to to it.
People must know that you are doing a good job, and you are the only one that is in the position to let them know. They will not get out of their way to learn about this.
E.G of a chat message I would send to my client's team boss:
Hello <first name>, it took 3 weeks, but we finally finished improving <system x>. This should save us <y> of <z> in the future. We tried hard to make sure your work is not disturbed by the change so it should be transparent for you. But let me know if you notice anything out of the ordinary. Cheers
Believe it or not, people actually like it. It keeps them in the loop, give them context and perspective, and help them take better decisions. So don't hesitate to do it.
> It keeps them in the loop, give them context and perspective, and help them take better decisions
Other side of the coin:
"I just spent 2 weeks optimising a daily process to be twice as fast" might well end up in me wondering why anyone would care about the 1 minute reduction in delivery time for daily cadence data. It might also result in me being stroppy about work I cannot get resourced that has obvious $ value.
But that is also a win for the business. Either I am wrong and am persuaded of the value, or I am not and I become more engaged with stakeholder prioritisation and we set up something more effective.
That's good feedback to have. If your boss/client step in and says "we don't need that", this allows you to re-calibrate the team priorities and shows you need to get better at extracting the real need in the future.
I completely agree with your initial comment and your response here. Transparency is a great feedback loop. And it's always good to celebrate what you do.
1) Break something on purpose. 2) Make sure the business notice it (usually a spectacular cost increase does the job).
3) Come in and fix the issue
4) Make a cost comparison quick chart before/after
5) Become famous
If you look long enough in most code bases, you will eventually find a spectacular problem.
Case in point, I was working for several months for a client, and I had a slow afternoon, so I decided to convert some calculation to numpy, see if we gained any free perf.
We got a x100 local speed up, which was very fishy. Gaining speed is common with vectorization, but two orders of magnitude is a lot.
So I looked at the original algo. There was a glaring mistake in it, that I fixed by my numpy code without noticing, and just changing that in the pure Python algo, without using numpy at all, made it X50 faster.
I could then call my client, and celebrate the good news. Not "there was a mistake", no. But "we found margin for progress".
On paper, it looks like it.
On practice, my mother pays 30€ a month of electricity in France, where nuclear energy is everywhere. She asked for a quote to get panels on her house, and they got back to her with 20000€.
Sure you can move the needle by noting I drain way more power than her, than she heats her house using fossil fuel and that the quotes could be have been reduced in many ways.
You can also note that solar panels have to be replaced several times, take much more space, don't have to including wiring in their ROI calculation (while nuclear does for some weird reason) and are created by electricity generated by fossil or nuclear fuel, so their building price is already cheap.
I wouldn't say the answer is that clear cut.