The trick is the same: use a popular linux distribution and don't fight the kinks.
The people who had no issues with Pulseaudio; used a mainstream distribution. Those distributions did the heavy lifting of making sure stuff fit together in a cohesive way.
SystemD is very opinionated, so you'd assume it wouldn't have the same results, but it does.. if you use a popular distro then they've done a lot of the hard work that makes systemd function smooth.
I was today years old when I realised this is true for both bits of poetter-ware. Weird.
pulseaudio I had to fight every single day, with my "exotic" setup of one set of speakers and a headset
with pipewire, I've never had to even touch it
systemd: yesterday I had a network service on one machine not start up because the IP it was trying to bind to wasn't available yet
the dependencies for the .service file didn't/can't express the networking semantics correctly
this isn't some hacked up .service file I made, it's that from an extremely popular package from a very popular distro
(yeah I know, use a socket activated service......... more tight coupling to the garbage software)
the day before that I had a service fail to start because the wall clock was shifted by systemd-timesyncd during startup, and then the startup timeout fired because the clock advanced more than the timeout
then the week before that I had a load of stuff start before the time was synced, because chrony has some weird interactions with time-sync.target
it's literally a new random problem every other boot because of this non-deterministic startup, which was never a problem with traditional init or /etc/rc
for what? to save maybe a second of boot time
if the distro maintainers don't understand the systemd dependency model after a decade then it's unfit for purpose
> it's literally a new random problem every other boot because of this non-deterministic startup, which was never a problem with traditional init or /etc/rc
This gave me a good chuckle. Systemd literally was created to solve the awful race conditions and non-determinism in other init systems. And it has done a tremendous job at it. Hence the litany of options to ensure correct order and execution: https://www.freedesktop.org/software/systemd/man/latest/syst...
And outside of esoteric setups I haven't ever encountered the problems you mentioned with service files.
systemd was created to solve the problems of a directory full of shell scripts. A single shell script has completely different problems. And traditional init uses inittab, which is not /etc/init.d, and works more like runit.
runit's approach is to just keep trying to start the shell script every 2 seconds until it works. One of those worse–is–better ideas, it's really dumb, and effective. You can check for arbitrary conditions and error–exit, and it will keep trying. If you need the time synced you can just make your script fail if the time is not synced.
traditional inittab is older than that and there's not any reason to use it when you could be using runit, really.
yeah, many options that are complicated beyond the understanding of the distro maintainers, and yet still don't allow expression of common semantics required to support network services reliably
like "at least one real IP address is available" or "time has been synced"
and it's not esoteric, even ListenAddress with sshd doesn't even work reliably
the ONLY piece of systemd I've not had problems with is systemd-boot, and then it turned out they didn't write that
> like "at least one real IP address is available" or "time has been synced"
"network-online.target is a target that actively waits until the network is “up”, where the definition of “up” is defined by the network management software. Usually it indicates a configured, routable IP address of some kind. Its primary purpose is to actively delay activation of services until the network has been set up."
For time sync checks, I assume one of the targets available will effectively mean a time sync has happened. Or you can do something with ExecStartPre. You could run a shell command that checks for the most recent time sync or forces one.
Are you running this particular unit file as a user unit or a system unit? Some targets like network-online.target don't work from user unit files.
You could also try targeting NetworkManager or networkd's "wait-online" services. Or if that doesn't work, something is telling systemd that you have an IP when you don't. NetworkManager has "ipv4.may-fail" and "ipv6.may-fail" that might be errenously true.
> at that point I might as well go back to init=/etc/rc
The difference is that systemd is much better at ensuring correctness. If you write the invoked shell command properly, it'll communicate failure or success correct and systemd will then communicate that state to the unit. It's still a lot more robust than before.
Seems like you have an axe to grind with systemd because it replaced your familiar (but extremely cruddy) init system and now you refuse to debug the problem because you prefer being able to blame systemd.
There is so much granularity and flexibility in what you can do it seems rather unlikely you cannot make it happen correctly. And if it is truly a bug... open an issue? They're rather responsive to it. And it isn't like the legacy init systems were bug free from inception (well, lord knows they were still chock full of bugs even when they were replaced).
Edit: sitting here with a grin
.. HN downvoting the advice of checking logs, debugging and opening an issue. I wish the companies y'all work at good luck.. they'll need it.
> Seems like you have an axe to grind with systemd because it replaced your familiar (but extremely cruddy) init system and now you refuse to debug the problem because you prefer being able to blame systemd.
I'm a pragmatist: I just want it to work
my solution to MULTIPLE different services failing to IP bind is to turn on the non-local ip binding sysctl, bypassing systemd's brokenness entirely
> There is so much granularity and flexibility in what you can do it seems rather unlikely you cannot make it happen correctly.
I've written an init before (in C), I know how the netlink interface to set an IP address and add routing table entries works
I understand the difference between monotonic and wall clocks
I understand the difference between Wants and Require
I know what's going on at every, single, level
and I can't stand how unreliable systemd makes nearly every single one of my, bluntly, completely vanilla systems
> And if it is truly a bug... open an issue?
did you read the link I pasted earlier?
I'm not wasting my time with that level of idiocy (from LP himself)
I'm not a systemd hater or anything, but I continue to read stuff from Poettering which to me is deeply disturbing given the programs he works on.
Saying it's not a bug that service is launched despite a stated required prerequisite dependency failed... WTF?
Sure, I agree with him that most computers should probably boot despite NTP being unable to sync. But proposing that the solution to that is breaking Requires is just wild to me.
I'm not sure I understand why you think the solution proposed there is so bad.
The question in that issue is around the semantics of time-sync.target. Targets are synchronization points for the system and don't (afaik) generally make promises about the units that are ordered before them (in this case chrony-wait.service.
Does that answer your specific objection of "proposing that the solution to that is breaking Requires is just wild to me"? Basically, what is proposed in that issue is not breaking Requires=. The proposition is that the user add their own, specific Requires= as a drop-in configuration since that's not a generally-applicable default.
No, that does not make sense, because it goes against the systemd documentation.
Targets[1]: Target units do not offer any additional functionality on top of the generic functionality provided by units. They merely group units, allowing a single target name to be used in Wants= and Requires= settings to establish a dependency on a set of units defined by the target, and in Before= and After= settings to establish ordering.
boot-complete.target[2]: Order units that shall only run when the boot process is considered successful after the target unit and pull in the target from it, also with Requires=.
Note use of "only run" with a reference to Requires=.
time-sync.target[3]: This target provides stricter clock accuracy guarantees than time-set.target (see above), but likely requires network communication and thus introduces unpredictable delays. Services that require clock accuracy and where network communication delays are acceptable should use this target.
Especially note the last sentence there.
The documentation clearly indicates that targets can fail, and that services that needs the target to be successful, should use Requires= to specify that.
If the above is not true, the Requires= and Targets documentation should be rewritten to explicitly say that targets might fulfill Requires= regardless of state. Also, the documentation for time-sync.target should explicitly remove the last sentence and instead state there is no functional difference between Requires=time-sync.target and Wants=time-sync.target, it is best-effort only.
If he really don't want targets to deliver failed/success guarantees, then they've massively miscommunicated in their documentation. That in my book is a huge deal.
In either case the issue should in no circumstance be casually dismissed as not-a-bug without further action.
I don't personally find it as disturbing as you do, I think. Which isn't to say that I don't think it should be fixed, etc. etc.
I'm sure the project would accept a documentation patch to amend this discrepancy. At the end of the day (despite what some people on the internet might like to allege), systemd is a free software project that, despite having (more or less) a BIFL, is ultimately a relatively bazaar-like project.
Though since these targets and unit properties are very core to systemd-the-service manager, I do think that this is a bigger documentation oversight than most.
The disturbing part isn't the bug in time-sync.target or documentation, the disturbing part is how casually he brushes the issue away.
To me this is a huge red flag for a senior contributer to a core systems component, signalling some fundamental lack of understanding or imagination.
I very much disagree with not fixing time-sync.target, but if he had instead written a well-reasoned explanation for why time-sync.target should not propagate failed states and flagging it as a documentation bug, then that's something I'd respect and would be fine with. Or, even better IMHO, he'd fix time-sync.target and state that users who wants to boot regardless should use Wants instead.
That's entirely defined by whatever units order themselves before network-online.target (normally a network management daemon like NetworkManager or systemd-networkd). systemd itself doesn't define the details; that's left up to how that distro and sysadmin have configured the network manager/system.
Same. I run a server with a ton of services running on it which all have what I think are pretty complex dependency chains. And I also have used Linux with systemd on my laptop. Systemd has never, once, caused me issues.
I can totally relate to this, it's gotten to the point that I'm just as scared of rebooting my Linux boxes as I was of rebooting my windows machine a couple of decades ago. And quite probably more scared.
The only parties that really cared about boot time were the big hosting providers and container schleppers. For desktop linux it never mattered as much.
PipeWire was also made by a guy with a lot of multimedia experience (GStreamer).
ALSA was kind of OK after mixing was enabled by default and if you didn't need to switch outputs of a running application between anything but internal speakers and headphones (which worked basically in hardware). With any additional devices that you could add and remove, ALSA became a more serious limitation, depending. You could usually choose your audio devices (including microphones) at least at the beginning of a video conference / playing a movie etc, but it was janky (unreliable, list of 20 devices for one multi-channel sound card) and needed explicit support from all applications. Not sure if it ever worked with Bluetooth.
I still need to use alsamixer to unmute my headphones after accidentally unplugging them and plugging them in again fails to do so. That's with PipeWire - never had that problem with just ALSA.
Eh, I had to do that with pulseaudio too, but constantly, across all distros and headphones. Pipewire is shonky, I have to restart now and then on my steam deck (I'm using it as a desktop), but it's still much better than pulseaudio. Even ALSA was better than pulseaudio lol
For most of the (sadly not shorter) life of PulseAudio, ALSA was more reliable, but at some point, Firefox got a new audio backend that straight up dropped support for ALSA, and a few games started crashing with backtraces indicating audio trouble when not run with PulseAudio. I've had to deal with PulseAudio's dropouts under load, latencies and lockups for 2-3 years before PipeWire became a viable replacement.
I got stuck for two weeks installing the kernel because I forgot to mount /boot. Perhaps I disabled it by accident when goofing around in alsamixer? Or my card did or didn't have hardware mixing?
I didn't actually know anything about Linux at the time and started with Gentoo because I saw a meme saying "install Gentoo" and people told me not to start with that distro. So it's possible I messed up the default config by accident.
That's because systemd originated at RedHat. If it had been designed distribution agnostic it would have worked a lot better on other distros besides RH.
What are the non-distribution agnostic parts of systemd? Considering it runs as PID1 (usually) it kinda is the base of distros and not really built on top of any distro other than "the linux kernel".
"The trick is the same: use a popular linux distribution and don't fight the kinks."
I believe that you are genuinely being sincere here, thinking this is good advice.
But this is an absolutely terrible philosophy. This statement is ignorant as well as inconsiderate. (again, I do believbe you don't intend to be inconsiderate consciously, that is just the result.)
It's ignorant of history and inconsiderate of everyone else but yourself.
Go back a few years and this same logic says "The trick is, just use Windows and do whatever it wants and don't fight."
So why in the world are you even using Linux at all in the first place with that attitude? For dishonest reasons (when unpacked to show the double standard).
Since you are using Linux instead of Windows, then you actually are fine with fighting the tide. You want the particular bits of control you want, and as long as you are lucky enough to get whatever you happen to care about without fighting too much, then you have no sympathy for anyone else who cares aboiut anything else.
You don't see yourself as fighting any tides because you are benefitting from being able to use a mainstream distro without customizing it. But the only reason you get to enjoy any such thing at all in the first place is because a lot of other people before you fought the tide to bring some mainstream distros into existence, and actually use them for ordinary activities enough despite all the difficulties, to force at least some companies and government agencies to acknowledge them. So now you can say things like "just use a mainstream distro as it comes and don't try to do what you actually want".
> Go back a few years and this same logic says "The trick is, just use Windows and do whatever it wants and don't fight."
This is basically exactly what I saw people saying in Windows subreddits. There's one post that particularly sticks out in my memory[0] that basically had everybody telling the OP to just not make any of the changes that they wanted to make. The advice seemed to revolve around adapting to the OS rather than adapting the OS to you, and it made me sad at the time.
> The people who had no issues with Pulseaudio; used a mainstream distribution. Those distributions did the heavy lifting of making sure stuff fit together in a cohesive way.
Incorrect. I used mainstream distro, still had issues, that just solved itself moving to pipewire. Issues like it literally crashing or emitting spur of max volume noise once every few months for no discernable reason.
Pulseaudio also completely denies existence of people trying to do music on Linux, there is no real way to make latency on it be good.
> SystemD is very opinionated, so you'd assume it wouldn't have the same results, but it does.. if you use a popular distro then they've done a lot of the hard work that makes systemd function smooth.
Over the years of using the "opinion" of SystemD seems to be "if it is not problem on Lennart's laptop, it's not a real problem and it can be closed or ignored completely".
For example systemd have no real method to tell it "turn off all apps after 5 minutes regardless of what silly package maintainers think". Now what happens if you have a server on UPS that have say 5 minutes of battery and one of the apps have some problem and doesn't want to close?
In SysV, it gets killed, and system gets remounted read only. You have app crash recovery but at least your filesystem is clean
In systemd ? No option to do that. You can set default timeout but it can be override in each service so you'd have to audit every single package and tune it to achieve that. That was one bug that was closed.
Same problem also surfaced if you have say app with a bug that prevented it from closing from sigterm and you wanted to reboot a machine. Completely stuck
But wait, there is another method, systemd have an override, you can press (IIRC) ctrl+alt+delete 7 times within 2 seconds to force it to restart ( which already confuses some people that expect it to just restart machine clean(ish) regardless https://github.com/systemd/systemd/issues/11285 ).
...which is also impossible if your only method of access is software KVM where you need to navigate to menu to send ctrl+alt+del. So I made ticket with proposal to just make it configurable timeout for the CAD ( https://github.com/systemd/systemd/issues/29616 ), the ticket wasn't even read completely because Mr. Poettering said "this is not actionable, give a proposal", so I pasted the thing he decided to ignore in original ticket, and got ignored. Not even "pull requests welcome" (which I'd be fine with, I just wanted confirmation that the feature like that won't be rejected if I start writing it).
There is also issue of journald disk format being utter piece of garbage ("go thru entire journal just to get app's last few lines bad", hundreds of disk reads on simple systemctl status <appname> bad) that is consistently ignored thru many tickets from different people.
Or the issue that resolvconf replacement in systemd will just roll a dice on DNS ordering, but hey, Mr. Lennart doesn't use openvpn so it's not real issue ( https://github.com/systemd/systemd/issues/27543 )
I'm not writing it to shit on systemd and praise what was before, as a piece of software it's very useful for my job as sysadmin (we literally took tens of thousands lines of fixed init scripts out because all of the features could be achieved in unit files) and I mean "saved tons of time and few demons running" in some cases, but Mr. Poettering is showing same ignorant "I know better" attitude he got scolded at by kernel maintainers.
> Pulseaudio also completely denies existence of people trying to do music on Linux, there is no real way to make latency on it be good.
I don't care much about PA at this point tbh and don't know much about the inner workings; it always worked just fine for me. But from what I read from people more "in the know" at the time, I'd heard that a lot of the (very real) user-facing problems with PA were ultimately caused by driver and other low-level problems. Those were hacky, had poor assumptions, etc. PA ultimately exposed those failures, and largely got better over time because those problems got fixed upstream of PA.
My takeaway from what I read was basically that PA had to stumble and walk so that pipewire could run.
> For example systemd have no real method to tell it "turn off all apps after 5 minutes regardless of what silly package maintainers think". Now what happens if you have a server on UPS that have say 5 minutes of battery and one of the apps have some problem and doesn't want to close?
Add a TimeoutStopSec= to /etc/systemd/system/service.d/my-killing-dropin.conf more or less, I think? These are documented in the systemd.service and systemd.unit manpages respectively.
> Same problem also surfaced if you have say app with a bug that prevented it from closing from sigterm and you wanted to reboot a machine. Completely stuck
See the --force option on the halt, poweroff, and reboot subcommands of systemctl. The kill subcommand if you want to target that specific service.
> so I pasted the thing he decided to ignore in original ticket, and got ignored. Not even "pull requests welcome" (which I'd be fine with, I just wanted confirmation that the feature like that won't be rejected if I start writing it).
I'm certainly sympathetic to this pain point. I'd take Lennart at his word that he's not opposed. Generally speaking, from following the systemd project somewhat, it's a very busy project and it's hard for all issues to get serviced. But they're very open to PRs, generally speaking.
> Or the issue that resolvconf replacement in systemd will just roll a dice on DNS ordering, but hey, Mr. Lennart doesn't use openvpn so it's not real issue ( https://github.com/systemd/systemd/issues/27543 )
Quickly taking a peek here (and speaking as a relatively superficial user of resolved myself), isn't the proposed solution to define interface ordering?
> it will ask on all links in parallel if there's no better routing info available. In your case there is none (i.e. no ~. listed among your network interfaces), hence it will be asked on all interfaces at the same time.
The people who had no issues with Pulseaudio; used a mainstream distribution. Those distributions did the heavy lifting of making sure stuff fit together in a cohesive way.
SystemD is very opinionated, so you'd assume it wouldn't have the same results, but it does.. if you use a popular distro then they've done a lot of the hard work that makes systemd function smooth.
I was today years old when I realised this is true for both bits of poetter-ware. Weird.