Cfrgtkky

I have a very strange problem with Ubuntu 18.04.1 Server, where the default resolver, systemd-resolved, isn't resolving some specific domain names.

The one it reliably fails on is stephenreescarter.net:

valorin@wp:~$ dig stephenreescarter.net



; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> stephenreescarter.net

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7015

;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1



;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 65494

;; QUESTION SECTION:

;stephenreescarter.net.         IN      A



;; Query time: 0 msec

;; SERVER: 127.0.0.53#53(127.0.0.53)

;; WHEN: Sun Jan 27 20:01:05 UTC 2019

;; MSG SIZE  rcvd: 50

But the domain itself is fine and works everywhere else:

valorin@wp:~$ dig stephenreescarter.net @1.1.1.1



; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> stephenreescarter.net @1.1.1.1

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45539

;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1



;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 1452

;; QUESTION SECTION:

;stephenreescarter.net.         IN      A



;; ANSWER SECTION:

stephenreescarter.net.  228     IN      A       104.28.2.92

stephenreescarter.net.  228     IN      A       104.28.3.92



;; Query time: 1 msec

;; SERVER: 1.1.1.1#53(1.1.1.1)

;; WHEN: Sun Jan 27 20:00:52 UTC 2019

;; MSG SIZE  rcvd: 82

And other domains work fine, so it's not simply a case of the server not being able to resolve everything:

valorin@wp:~$ dig google.com



; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> google.com

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24208

;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1



;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 65494

;; QUESTION SECTION:

;google.com.                    IN      A



;; ANSWER SECTION:

google.com.             148     IN      A       74.125.24.100

google.com.             148     IN      A       74.125.24.101

google.com.             148     IN      A       74.125.24.102

google.com.             148     IN      A       74.125.24.113

google.com.             148     IN      A       74.125.24.138

google.com.             148     IN      A       74.125.24.139



;; Query time: 2 msec

;; SERVER: 127.0.0.53#53(127.0.0.53)

;; WHEN: Sun Jan 27 20:00:57 UTC 2019

;; MSG SIZE  rcvd: 135

Rebooting the system sometimes solves the problem, likewise with sudo systemd-resolve --flush-caches. However these don't always work, or sometimes need to be attempted multiple times before they start working.

I can reproduce this problem on a newly created Ubuntu 18.04.1 DigitalOcean droplet in the SGP1 region.

In all other ways, systemd-resolve seems to work, so I have no clue what is going on.

Update - debugging info

valorin@wp:~$ ls -al /etc/resolv.conf

lrwxrwxrwx 1 root root 39 Oct  3 16:43 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

valorin@wp:~$ cat /run/resolvconf/resolv.conf

cat: /run/resolvconf/resolv.conf: No such file or directory

1 valorin@wp:~$ cat /run/systemd/resolve/resolv.conf

# This file is managed by man:systemd-resolved(8). Do not edit.

#

# This is a dynamic resolv.conf file for connecting local clients directly to

# all known uplink DNS servers. This file lists all configured search domains.

#

# Third party programs must not access this file directly, but only through the

# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,

# replace this symlink by a static file or a different symlink.

#

# See man:systemd-resolved.service(8) for details about the supported modes of

# operation for /etc/resolv.conf.



nameserver 67.207.67.2

nameserver 67.207.67.3

I think that your /etc/resolv.conf symlink is wrong.

Currently you show...

~$ ls -al /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Oct 3 16:43 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I believe that it should point to resolv.conf, not stub-resolv.conf. To change it, we'll do this...

sudo rm -i /etc/resolv.conf # remove old symlink

sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf # recreate symlink

See if that helps in any way.

The default installation of 18.04 seems to be missing the libnss-resolve package, the installation of which fixes the /etc/nsswitch.conf file so the hosts line looks like

hosts:          files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns myhostname  

If you scan your /var/log/syslog file, you will probably see lines like:

Jan 27 09:33:15 leno systemd-resolved[931]: Using degraded feature set (UDP) for DNS server 192.168.1.1.

Jan 27 10:06:12 leno systemd-resolved[931]: Grace period over, resuming full feature set (UDP+EDNS0) for DNS server 192.168.1.1.  

These indicate that sometimes you are running at a reduced function set over UDP, and large outputs can overflow the usual buffer. See launchpad bugs 1804487 and 1805027.
Other workarounds like redirecting the /etc/resolv.conf link from /run/systemd/resolv/stub-resolv.conf to the .../resolv.conf file basically cut systemd out of the loop, providing a nameserver directly.

You tested resolving with 1.1.1.1, but not a 67... ip. Try:

dig stephenreescarter.net @67.207.67.2  

If that fails, the problem is not within systemd-resolvd which is using that namesserver. That nameserver does not work for me, but maybe it's not public.