Cfrgtkky

I have a VPS running Ubuntu 16.04.5 that's been going for a number of years now with little issue. Today, however, I found I was unable to access the server using SSH, receiving 'connection refused' errors. I accessed the server using my VPS host's serial console service, and traced the issue down to openssh server failing to start. Here's an output of service status, service start, and sshd -t following a fresh reboot:

root@167:/# service ssh status

● ssh.service - OpenBSD Secure Shell server

   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)

   Active: failed (Result: start-limit-hit) since Fri 2019-01-18 04:56:42 EST; 24min ago

  Process: 983 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)



Jan 18 04:56:42 167 systemd[1]: Failed to start OpenBSD Secure Shell server.

Jan 18 04:56:42 167 systemd[1]: ssh.service: Unit entered failed state.

Jan 18 04:56:42 167 systemd[1]: ssh.service: Failed with result 'exit-code'.

Jan 18 04:56:42 167 systemd[1]: ssh.service: Service hold-off time over, scheduling restart.

Jan 18 04:56:42 167 systemd[1]: Stopped OpenBSD Secure Shell server.

Jan 18 04:56:42 167 systemd[1]: ssh.service: Start request repeated too quickly.

Jan 18 04:56:42 167 systemd[1]: Failed to start OpenBSD Secure Shell server.

Jan 18 04:56:42 167 systemd[1]: ssh.service: Unit entered failed state.

Jan 18 04:56:42 167 systemd[1]: ssh.service: Failed with result 'start-limit-hit'.

root@167:/# service ssh start

Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details.

root@167:/# sshd -t

Missing privilege separation directory: /var/run/sshd

I've attempted some research into this, but nothing that's come up seems to have an actual solution - Just endless cycles of 'I have this problem' with no answers, answers that are outdated, or just generally unhelpful information.

Does anybody have any ideas on what to do next to troubleshoot/resolve this issue? SSH was last working about 12 hours ago when I logged in to run updates and rebooted the server.

UPDATE

Issue was resolved using workaround 1 as provided here: https://askubuntu.com/a/1110843/531533

According to the bug report here, the problem can be worked around by adding the directory creation code to /etc/rc.local.

I've done some research and I still can't figure out what is supposed to create that directory on a normal server startup - there're a couple of scripts under /etc that do this, but they appear to be deprecated as they relate to the old SysV startup and I can't find anything in the SystemD init setup to create that directory.

[Update]: Just noticed @pa4080 comment, and this seems to be it - the file /usr/lib/tmpfiles.d/sshd.conf exists and contains the line d /var/run/sshd 0755 root root, in openssh-server version 1:7.2p2-4ubuntu2.6.

The tmpfiles.d directory is part of the systemd-tmpfiles service and is documented here .

The could be an update anomaly as I've seen relevant bug reports that discuss this as a failed update, and updating the openssh-server to the latest version might fix it.