Problem

I'm trying to investigate this weird error where systemd-resolved is constantly crashing on 2 out of 4 machines. The one thing different with these two problematic machines (that I can think of), is that they are mounting a samba/cifs filesystem exported from machine #3 (on which systemd-resolved is running fine).

How do I get to the bottom of this?

OS

# cat /etc/lsb-release 

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

DISTRIB_CODENAME=bionic

DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"



# uname -a

Linux prod-delayed-jobs 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



# systemd --version

systemd 237

+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

Relevant journalctl output

Notice the /dev/null is not a device. line in the logs. This is occurring multiple times - during, before, and after systemd-resolved crashes.

Feb 12 14:23:21 prod-delayed-jobs dhclient[794]: DHCPREQUEST of [REDACTED-IP-ADDRESS] on eth0 to 172.31.1.1 port 67 (xid=0x730cc4e4)

Feb 12 14:23:21 prod-delayed-jobs dhclient[794]: DHCPACK of [REDACTED-IP-ADDRESS] from 172.31.1.1

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopping Network Name Resolution...

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopped Network Name Resolution.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: /dev/null is not a device.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Starting Network Name Resolution...

Feb 12 14:23:21 prod-delayed-jobs systemd[11796]: systemd-resolved.service: Failed to set up mount namespacing: Invalid argument

Feb 12 14:23:21 prod-delayed-jobs systemd[11796]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Failed to start Network Name Resolution.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 1.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopped Network Name Resolution.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: /dev/null is not a device.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Starting Network Name Resolution...

Feb 12 14:23:21 prod-delayed-jobs dhclient[794]: bound to [REDACTED-IP-ADDRESS] -- renewal in 32497 seconds.

Feb 12 14:23:21 prod-delayed-jobs systemd[11813]: systemd-resolved.service: Failed to set up mount namespacing: Invalid argument

Feb 12 14:23:21 prod-delayed-jobs systemd[11813]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Failed to start Network Name Resolution.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 2.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopped Network Name Resolution.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: /dev/null is not a device.

Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Starting Network Name Resolution...

Feb 12 14:23:22 prod-delayed-jobs systemd[11831]: systemd-resolved.service: Failed to set up mount namespacing: Invalid argument

Feb 12 14:23:22 prod-delayed-jobs systemd[11831]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved

Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE

Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.

Feb 12 14:23:22 prod-delayed-jobs systemd[1]: Failed to start Network Name Resolution.

Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.

Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 3.

/etc/fstab contents - in case this is relevant

UUID=2f54e8e6-ff9c-497a-88ea-ce159f6cd283 /               ext4    discard,errors=remount-ro 0       1

/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0

//[REDACTED]/uploaded_files /home/myuser/shared cifs rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser 0 0

Few systemd configuration files are pointing to /dev/null

This is the case on all the 4 machines, however only 2 out of these 4 are having trouble with systemd-resolved. Strangely, hostname.service is pointing to /dev/null - wonder why?

lrwxrwxrwx 1 root root    9 Nov 15 21:45 bootlogd.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 bootlogs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 bootmisc.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 checkfs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 checkroot-bootclean.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 checkroot.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 cryptdisks-early.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 cryptdisks.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 fuse.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 halt.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 hostname.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 hwclock.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 killprocs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Apr 12  2018 lvm2.service -> /dev/null

lrwxrwxrwx 1 root root    9 Jul  2  2018 mdadm.service -> /dev/null

lrwxrwxrwx 1 root root    9 Jul  2  2018 mdadm-waitidle.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 motd.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountall-bootclean.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountall.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountdevsubfs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountkernfs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountnfs-bootclean.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountnfs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Jan 16  2018 nfs-common.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 rc.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 rcS.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 reboot.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 rmnologin.service -> /dev/null

lrwxrwxrwx 1 root root    9 Dec  6 11:10 screen-cleanup.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 sendsigs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 single.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 stop-bootlogd.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 stop-bootlogd-single.service -> /dev/null

lrwxrwxrwx 1 root root    9 Dec  6 11:07 sudo.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 umountfs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 umountnfs.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 umountroot.service -> /dev/null

lrwxrwxrwx 1 root root    9 Nov 15 21:45 x11-common.service -> /dev/null

/lib/systemd/system

[Unit]

Description=Network Name Resolution

Documentation=man:systemd-resolved.service(8)

Documentation=https://www.freedesktop.org/wiki/Software/systemd/resolved

Documentation=https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers

Documentation=https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

DefaultDependencies=no

After=systemd-sysusers.service systemd-networkd.service

Before=network.target nss-lookup.target shutdown.target

Conflicts=shutdown.target

Wants=nss-lookup.target



[Service]

Type=notify

Restart=always

RestartSec=0

ExecStart=!!/lib/systemd/systemd-resolved

WatchdogSec=3min

User=systemd-resolve

CapabilityBoundingSet=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE

AmbientCapabilities=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE

PrivateTmp=yes

PrivateDevices=yes

ProtectSystem=strict

ProtectHome=yes

ProtectControlGroups=yes

ProtectKernelTunables=yes

ProtectKernelModules=yes

MemoryDenyWriteExecute=yes

RestrictRealtime=yes

RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6

SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @reboot @swap