Also, critique starts with "We like controlling the startup of the system with shell scripts that are readable".. How on earth is a systemd service file less readable then a hundreds of lines bash script?
Being a developer and being an admin are completely different jobs.
A admin needs to know a dozen languages and codes in bash most of the time, they have a hand written folder with glued in scraps next to their terminal which explains why they did what, how and why. This folder has entries like "reboot three times, do an rain dance and run godHelpYou.bash as root" because that's the only way that you've found out how the libraries actually talk to each other.
For people like this "do one thing and do it well" is the only way that you can beat the impossible complexity of a modern machine back into a running computer even some of the time. When you have a single point of failure which is a black box you're just back to windows land: restart the service, reboot the computer, reinstall the system.
From this point of view devs look like a tribe of monkeys building a shit pyramid over your village. They just keep flinging more shit at the top thinking they are the greatest ever monkeys. When the shit pyramid collapses and buries everything you know and love under a few meters of shit, probably killing someone in the process, these monkeys just shrug say a few words about how that was the best shit pyramid anyone had ever build and start making a new one one or two valleys over.
For people like this "do one thing and do it well" is the only way that you can beat the impossible complexity of a modern machine back into a running computer even some of the time. When you have a single point of failure which is a black box you're just back to windows land: restart the service, reboot the computer, reinstall the system.
For the developers, on the other hand, it might mean that some use case that is important for the admin is simply not yet implemented. The solution, as so often, is... drumroll to report a bug! Developers care about their users. Just talk to them.
Also, admins should realize that "reducing complexity" is a goal that they share with the developers. systemd is all about reducing complexity by implementing functionality the right way, on the right layer. For example, with systemd it's way easier to write a daemon, because it takes care of
* daemonizing: just stay in foreground
* logging: just output to stdout/stderr. journald will collect everything, and if you want, forward it to a classic syslog service
* startup: no more shell scripts required, just a simple service file
* supervision: just add "Restart=on-failure" to your service file. It also supports software watchdogs.
And what happens when that single file doesn't work? The one time I had to deploy a custom service using systemd, there was no feedback as to why it wasn't working with service start. No error message, no guidance on how to debug online, no indication of where to look. By contrast adding it to a shell script just worked.
An init system should have a verbose mode that tells you exactly what steps are being taken to start a service and what the result of each step is. Maybe this will come later, but -given the guy's history- it won't be written by Poettering.
you're seriously suggesting strace as an acceptable form of troubleshooting?
Don't get me wrong, I love to bring out strace and will use it to solve all sorts of problems, but it really shouldn't be necessary use it near the start of troubleshooting.
Different people have different definitions of complexity. For me having a single point of failure is complexity, for most people a centralized single process handling as much as possible is simplicity. Neither is right or wrong.
Whether a system is tightly coupled is independent of whether it's implemented as a single process. Objections to systemd as monolithic, as constituting a single point of failure, etc. are based on its being a tightly coupled set of components.
That systemd is tightly coupled is a valid objection, but "single process" has a specific meaning in this context and it implies a much worse design, with every component potentially taking down PID1. As far as I know, this is not the case in systemd.
Just curious, have you ever used systemd and ran into a problem where your service file screwed up and you had yo read systemd's source code or are you making this up?
I actually had to try to figure out why, turns out systemd supercedes lsmod/modprobe etc; causing those programs to return 1 when invoked, there's no debug or anything so I was seriously weirded out.
took me some time to figure out it was systemd.
the firewall wrapper for iptables gives you an insecure default config and is hard to fix via config management "add a service that means port 3128 TCP, now allow incoming connections"
My config management system creates files and pushes them out, I'm not in the habit of running idempotent commands repeatedly.. I'd rather check if something is correct before correcting it.
> I actually had to try to figure out why, turns out systemd supercedes lsmod/modprobe etc; causing those programs to return 1 when invoked, there's no debug or anything so I was seriously weirded out.
> My config management system creates files and pushes them out, I'm not in the habit of running idempotent commands repeatedly.. I'd rather check if something is correct before correcting it.
You should be laying down a base initial configuration at the time that the machine was provisioned. The basic example of this is the sudoers file. The default setup for Ubuntu/RHEL is different enough that it was causing problems and the solution was simple - force defaults.
> And if that service file screws up how do you troubleshoot it? You end up diving in to the source code of systemd, versus fixing a bug in your script.
Surely the systemd equivalent would be delving into the source code for dash, or god help you dash... I know which one I'd prefer--not that I've ever hard to do either, fortunately!
> And if that service file screws up how do you troubleshoot it? You end up diving in to the source code of systemd, versus fixing a bug in your script.
? You end up diving in to the service file vs. the script.
> And don't get me started on binary log files.
Having worked with binary and plaintext log data for the better part of a decade now, I have to say that pretty much the entire set of claimed advantages of plaintext log data is very wrong, and we've wasted a ton of productivity & efficiency because of it.
If you really are sure of its superiority though, just put a little stub in front that converts the binary log to a human readable format that still needs a program in order to present to the user...
> SysVInit has served me thus far with no issues.
Speaking of attitude... there are two ways to look at that datapoint...
And if that service file screws up how do you troubleshoot it? You end up diving in to the source code of systemd, versus fixing a bug in your script.
That's a good thing, because if you find a bug, it will be fixed for everybody, not just for your one-off init script.
your attitude is the problem
The attitude here is to fix problems at their source. Instead of using the hammer for everything, sometimes coming up with a new tool is a better idea.
That's a good thing, because if you find a bug, it will be fixed for everybody, not just for your one-off init script.
Generally I agree with this at a broad level. But, there is still the problem of: I need this running now. To do that, I need to compile and run a custom systemd, and hope that my patch gets accepted upstream, and that in a few months my distro finally adopts it and puts it in the standard package.
Generally speaking I really like systemd and am excited to get some time to update my installed stuff to use it, but the above is a real concern.
We have SysV shim layers for that though. We had them in Upstart, and we have them for SystemD. Boiled down, it can still run shell scripts that look like SysV scripts, and will be able to do so indefinitely because that's basic functionality.
At the worst case though, this reduces systemd to a shim-launcher for SysV init, which is the situation you find yourself in now or when using Upstart, which tons of Ubuntu users already have been.
I dare you to tell my why lxc-docker silently fails to start at boot in jessie. Without a single shred of clue in any log, no matter what logging options are turned up - at least with a shell script I can add some echo/exit/touch etc. statements to convince myself it's at least being noticed at startup: https://groups.google.com/forum/#!topic/docker-user/bg5-hlmR...
> I dare you to tell my why lxc-docker silently fails to start at boot in jessie. Without a single shred of clue in any log, no matter what logging options are turned up - at least with a shell script I can add some echo/exit/touch etc. statements to convince myself it's at least being noticed at startup: https://groups.google.com/forum/#!topic/docker-user/bg5-hlmR....
Have you tried adding "touch" commands in ExecStartPre/ExecStartPost? Systemd also gives you a hint of why something didn't start via "systemctl status $servicename".
No offence, but systemctl status isn't exactly new to me. The issue is that network.target apparently never fires any service which requires it (in Jessie at least). For what it's worth systemctl status docker shows it's just inactive (dead) - journalctl similarly shows no clues at all. I have a Debian pressed.cfg & virt-install script to reproduce if you're interested, https://gist.github.com/csirac2/f6e4ac6e7c715243d4b8 and https://gist.github.com/csirac2/57a1ddf7931fa3e9d17a although these aren't up to date with what I've been preparing for a bug report
The next step in troubleshooting that one is to run strace and find out what's happening. I agree it's frustrating but if no one debugs these things, problems won't get solved. (and yes, scorching the earth and installing a different distro is a valid solution to the problem at hand)
I'm not sure how to make strace pid 1 :-) And I'm not quite ready to abandon Debian yet. I've created a virt-install script which reproduces the problem reliably (honestly all it does is install jessie, then lxc-docker). Hopefully my workload eases and I can put this trivially reproducible thing into a bug report somewhere (debian? systemd? docker) for somebody to see what's going on.
FWIW systemctl start docker works fine after the system has booted. It's configured to require network.target, and even though I can see network target scroll by during boot, it's still not being noticed.
The work-around is to make multiuser want docker, then I've got docker starting at bootup.
Sounds like lxc-docker.service simply wasn't enabled. "enabling" is basically creating a symlink from multi-user.target.wants to lxc-docker.service. This is done by 'systemctl enable lxc-docker.service' once that service declares WantedBy=multi-user.target in its [Install] section.
Other symptoms that would indicate this was the problem would have been that 'systemctl status lxc-docker.serivce' would have said "Loaded: loaded (...; disabled)" and "Active: inactive (dead)". i.e., systemd never started the service because nothing ever wanted it.
strace -p1 should do the trick, you can attach after start, but you will have to get it started early, probably as a systemd service. Or if its in a container, strace form outside the container.
Without doing any further investigation: does the type of service you're setting in your user file match the way you're starting up the daemon? For a "normal" service, you need to start the service as "normal" and have it run without forking / exiting (as systemd handles daemonization).
If you start a "normal" service using a command that double-forks and immediately exits (like a `--daemon` flag), systemd will see that the process has exited to mean that the service has finished, and will terminate any of its child processes (as you want runaway dependencies culled when a service crashes). This sounds like the behavior you describe.
I read the 'shut up' line in the post above yours as more along the lines of 'do something useful, post bugs, show some commitment' rather than 'shut up systemd is the Only Way'.
Perhaps the sponsors of the linked site are doing bug reports on packages like systemd-shim and others. Perhaps the upstream project members involved are ensuring that they have init scripts for systemv available. They could add a question with links along those lines easily.
This sentence (if it's not obvious) is a hint to invest energies into making debian work fine without systemd instead of countless whining and crying but in the end doing nothing.
There may be some people opposed to systemd, but there have been votes that saw systemd as the clear winner for the default init system. Why should the majority of users who are FOR systemd bending to some users who don't like systemd out of some philosophical view? Instead of behaving like a little child, people can work on letting Debian run fine without systemd (which IS the fact today, i can simply switch between sysvinit and systemd just by installing a package).
And in the end, you should remind yourself that this is only about the default for one linux distribution. You will still be able to use sysvinit if you like. There is choice. The majority wants systemd. The minority wants sysvinit. Only because the opposers are loud and make a lot of noise about this doesn't change the fact that they are a relatively small number. Changing the default is the only sane thing to do.
> people can work on letting Debian run fine without systemd
Arch switched to systemd about a year ago. When official distro packages are shipping with systemd service files by default, it means there is drastically less effort required on your part to maintain a systemd-based environment. It would take an experienced and dedicated crew to translate the systemd service files of dozens of upstream packages to Runit for instance. The enterprise level support just isn't there, whereas with systemd it is.
That being said, I like writing systemd service files. I just wonder what would happen if more distros pushed for, e.g. uselessd or Runit as the standard init system.
> uselessd (the useless daemon, or the daemon that uses less... depending on your viewpoint) is a project to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it’s systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design.
> Also, critique starts with "We like controlling the startup of the system with shell scripts that are readable".. How on earth is a systemd service file less readable then a hundreds of lines bash script?
Bash scripts? Unless I'm really mistaken, startup scripts need to be written towards the POSIX shell. It's kind of like bash scripts except you pretend the year is 1985 and no bash versions released after that exists.
It would be nice for those of us not in the know to see an example of more-or-less similar configuration expressed in the respective systems for comparison. Although I tend philosophically towards the declarative route, I don't really have a good understanding of the issues involved in this specific instance - but I'd like to.
Also, critique starts with "We like controlling the startup of the system with shell scripts that are readable".. How on earth is a systemd service file less readable then a hundreds of lines bash script?
Also relevant: http://www.itwire.com/business-it-news/open-source/65684-deb...