Take things as they are. Punch when you have to punch. Kick when you have to kick.
You might want to first check out Surviving Systemd where my initial simpler set of tips and rants appear.
The entire linux universe has been hornswoggled into slavishly adpoting systemd, no matter how insane it becomes. It is, in fact, the very first computer fungus. It keeps growing over and absorbing more and more pieces of linux (udev, xinetd, d-bus, they seem to have their eyes on cron next since timer units were added).
You can actually find almost everything wrong with systemd by reading systemd's own web page declaring all the problems as myths. If they call facts myths enough, they figure everyone will start believing they are myths. Basically the only argument presented on that web page is "That's not true because we say so!".
My favorite myth debunking: "We are pretty good in keeping systemd out of the list of blocker bugs of the distribution." Well, yes, since only systemd fanboys at redhat get to decide what is a blocker, then surprise! There aren't any :-). Of course there are minor things like corrupting the furshlugginer binary journal files causing infinite boot loops, but binary journal files are so much better than ones you can read that a little problem like not being able to boot is no big deal. Then there are the incredibly pointless systemd user daemons that never go away, but which systemd waits for on shutdown, causing shutdown to take at least five minutes. Or if you try to fix that by using the systemctl --force reboot command, systemd brilliantly shuts down the network devices first then tries to unmount all the NFS filesystems, which takes several hours for them to individually timeout. Yes, the force option is a great solution to the 5 minute shutdown times :-).
Of course, since systemd is fully scriptable (at least according to the myth debunking), then you ought to be able to just make systemd ignore all the NFS mounts and handle them yourself. Not really. Even if the mount isn't in the fstab, systemd (not bloated at all) has code hard wired to watch for new mounts and sieze control of them (even if you know it is going to fuck it up and would like to make it stop).
Rebooting
So, how the heck do you get a systemd controlled system to reboot before you die of old age? This program is one piece of the puzzle:
// Program to find all process trees with /usr/lib/systemd/systemd --user as
// the root process.
//
// Prints kill -9 commands ordered from the bottom of the tree up to the
// systemd --user command.
#include <sys/types.h>
#include <dirent.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include <string>
#include <set>
#include <vector>
#include <iostream>
#include <fstream>
#include <cstring>
// Make a multiset of these guys to hold the process tree
struct pidtree {
unsigned long parent_pid;
unsigned long child_pid;
};
// Compare multiset elements first on parent then on child (actually
// if I include the child in the key, I guess I don't really need a
// multiset, but what the heck...)
//
struct pidcomp {
bool operator() (const struct pidtree & a,
const struct pidtree & b) const {
if (a.parent_pid < b.parent_pid) {
return true;
} else if (a.parent_pid == b.parent_pid) {
return a.child_pid < b.child_pid;
}
return false;
}
};
typedef std::multiset<struct pidtree, pidcomp> treetype;
treetype tree;
// Recursively descend the process tree, printing kill -9 commands
// on the way back up.
//
void
print_tree(unsigned long parent)
{
struct pidtree p;
p.parent_pid = parent;
p.child_pid = 0;
std::pair<treetype::iterator, treetype::iterator> start;
start = tree.equal_range(p);
treetype::iterator it;
for (it = start.first; it!=tree.end(); ++it) {
struct pidtree elt = *it;
if (elt.parent_pid == parent) {
print_tree(elt.child_pid);
} else {
break;
}
}
std::cout << "kill -9 " << parent << std::endl;
}
int
main(int argc, char ** argv) {
std::vector<unsigned long> daemons;
// Read the /proc directory to find all the systemd user daemons
struct dirent * dirp;
DIR * procdir = opendir("/proc");
if (procdir == NULL) {
const char * msg = std::strerror(errno);
std::cerr << "Unable to read /proc : " << msg << std::endl;
exit(2);
}
while ((dirp = readdir(procdir)) != NULL) {
char * endp;
unsigned long int onepid = strtoul(dirp->d_name, &endp, 10);
if (*endp == '\0') {
// OK, this is an all digit directory and therefore a pid
// and not one of the other zillion things under /proc
// Read each process' status file to find the parent pid
// and build the process tree.
std::string statname("/proc/");
statname.append(dirp->d_name);
statname.append("/status");
{
std::ifstream sf(statname.c_str());
if (sf.is_open()) {
std::string buf;
unsigned long int oneparent = 0;
while (getline(sf, buf)) {
const char * bufp = buf.c_str();
if (strncmp(bufp, "PPid:", 5) == 0) {
oneparent = strtoul(bufp+5, NULL, 10);
break;
}
}
struct pidtree p;
p.parent_pid = oneparent;
p.child_pid = onepid;
tree.insert(p);
} else {
const char * msg = std::strerror(errno);
std::cerr << "Unable to open " << statname << " for reading : "
<< msg << std::endl;
}
}
// Check each process to see if it is one of the systemd
// user daemons and record the pid in the daemons array if it
// matches
std::string cmdname("/proc/");
cmdname.append(dirp->d_name);
cmdname.append("/cmdline");
{
std::ifstream cf(cmdname.c_str());
std::string arg;
std::string systemd("/usr/lib/systemd/systemd");
std::string user("--user");
bool arg1issystemd = false;
bool arg2isuser = false;
char c;
int argnum = 0;
while ((cf.get(c)) && (argnum <= 2)) {
if (c == '\0') {
++argnum;
switch(argnum) {
case 1:
if (arg == systemd) {
arg1issystemd = true;
} else {
argnum = 99;
}
break;
case 2:
if (arg == user) {
arg2isuser = true;
} else {
argnum = 99;
}
}
arg.clear();
} else {
if (argnum == 2) {
argnum = 99;
}
arg.append(1, (char)c);
}
}
if ((argnum == 2) && arg1issystemd && arg2isuser) {
daemons.push_back(onepid);
}
}
}
}
// Go through the daemons array and print the commands to kill all
// user daemons and processes running under them
for (int i = 0; i < daemons.size(); ++i) {
print_tree(daemons[i]);
}
}
The program reads the process tree, finds the systemd user daemons, and prints kill -9 commands to get rid of them and all their child processes. Now when systemd tries to wait on them at reboot, they are already gone, no more 5 minute delays.
But that only helps for the user daemons. If some NFS server is down, systemd will take forever to timeout trying to unmount the filesystem. This command can help there:
umount -l -t nfs -a
The -l option being the vitally important bit that says do the unmount locally, don't try to contact the server. Now this sometimes doesn't work. From time to time, some linux kernels have broken the -l option so it has to timeout anyway, but at least the NFS maintainers seem to be willing to believe this is a bug rather than asserting it is supposed to work that way, so there is a chance a kernel update might fix it and you can go back to rebooting rapidly :-).
Put the pieces together and you can make this handy script:
#!/bin/bash # # echo the commands needed to run before a reboot so the reboot # will happen sometime today. # echo /usr/bin/umount -l -t nfs -a /usr/local/bin/locate-user-daemons echo /sbin/reboot
Then make an alias to run it:
alias reboot="/usr/local/bin/pre-reboot | sudo /bin/bash -i"
If you use the alias in a terminal to reboot, you may reboot rapidly. If you run some GUI reboot button, none of this stuff will happen and the odds are good you'll be stuck again. Perhaps there is a way to make a pre reboot systemd unit somehow. If so, it ought to be easy, because it is a myth that systemd is difficult, but I haven't figured out how to do it yet :-).
User Daemons
In a new development, the systemd user deamon now appears to be responsible for launching dbus when you login. This, <sarcasm>really helpfully</sarcasm>, launches all kinds of random junk when various programs ask about things via dbus. I first noticed this when google-chrome started asking for my gnome keyring password every damn time I started it. Poking around found lots of random crap (most of which I previously disabled in earlier desktop incarnations) running under my systemd --user session.
I went on a day long quest to find out where the devil this stuff comes from. The key hint came from the systemd dbus-daemon man page. That says user sessions are copnfigured by /etc/dbus-1/session.conf. Naturally, that file is obsolete, but it has a comment pointing to /usr/share/dbus-1/session.conf. That file, of course, contains nothing but cryptic gibberish, but I eventually found a description of the <standard_session_servicedirs /> directive in there, which pointed me at the XDG Base Directory Specification. That, in turn, allowed me to write this script to grep for things inside all the directories that might possibly hold .service files which trigger the running of all the crap I see under the user daemon.
#!/bin/bash
#
for p in "${XDG_DATA_HOME}" "${XDG_CONFIG_HOME}" "${XDG_DATA_DIRS}" \
"${XDG_CONFIG_DIRS}" "${XDG_CACHE_HOME}" "${HOME}/.local/share" \
"${HOME}/.config" "/usr/local/share:/usr/share" "/etc/xdg" \
"${HOME}/.cache"
do
if [ -n "$p" ]
then
echo "$p" | tr ':' '\n' | while read onedir
do
if [ -d "$onedir/dbus-1/services" ]
then
fgrep -R -n -e "$1" "$onedir/dbus-1/services"
fi
done
fi
done
So, if I see this sort of crap running:
1474 ? Ss 0:00 /usr/lib/systemd/systemd --user 1477 ? S 0:00 \_ (sd-pam) 1505 ? Ss 0:00 \_ /usr/bin/dbus-daemon --session --address=systemd: 1637 ? S 0:00 \_ /usr/libexec/gconfd-2 1662 ? Ssl 0:00 \_ /usr/libexec/at-spi-bus-launcher 1669 ? S 0:00 | \_ /bin/dbus-daemon --config-file=/usr/share/def 1671 ? Sl 0:00 \_ /usr/libexec/at-spi2-registryd --use-gnome-sessio
and I really want to stop the at-spi stuff from starting, I can do this:
zooty> find-xdg at-spi /usr/share/dbus-1/services/org.a11y.Bus.service:3:Exec=/usr/libexec/at-spi-bus-launcher /usr/share/dbus-1/services/org.a11y.Bus.service:4:SystemdService=at-spi-dbus-bus.service
Now I know where the at-spi-bus-launcher comes from, and I can guess that maybe at-spi2-registryd is started by it (no doubt via the dbus it is running) since I didn't see any explicit mention of that in any service files.
A quick sudo rm -f /usr/share/dbus-1/services/org.a11y.Bus.service should eliminate the ability of anything to start any of the at-spi user deamons now.
Naturally, the next time you get a system update which includes that file, it will come back, so to permanently eradicate it you'll need a big hammer. Actually, permanently is optimistic, since it is inevitable that some OCD developer will decide to rename everything to make his tick stop twitching, then you'll have to rediscover all the new names...
You'd think you could override this on a per-user basis, but none of my experiments with that worked. It always obeyed the service files in the system directory, so I have to eradicate the installed files.