DNS: add 9.9.9.9 to resolv.conf

[missytake]

According to #515, some providers misconfigure /etc/resolv.conf. With 9.9.9.9, at least one working DNS resolver is there.

[link2xt]

How does this interact with installing unbound? We are installing our own resolver on 127.0.0.1 that does DNSSEC checks only to add 9.9.9.9 next to it and send plaintext requests there?

[missytake]

How does this interact with installing unbound? We are installing our own resolver on 127.0.0.1 that does DNSSEC checks only to add 9.9.9.9 next to it and send plaintext requests there?

Let me re-run the CI to double-check, but I assume that the unbound installation overwrites /etc/resolv.conf again and our temporary addition of 9.9.9.9 gets removed as soon as we don't need it anymore

[missytake]

How does this interact with installing unbound? We are installing our own resolver on 127.0.0.1 that does DNSSEC checks only to add 9.9.9.9 next to it and send plaintext requests there?

Let me re-run the CI to double-check, but I assume that the unbound installation overwrites /etc/resolv.conf again and our temporary addition of 9.9.9.9 gets removed as soon as we don't need it anymore

The assumption was correct:

$ ssh staging2.testrun.org cat /etc/resolv.conf  # after adding 9.9.9.9, before the install unbound step
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.

nameserver 2a01:4ff:ff00::add:2
nameserver 2a01:4ff:ff00::add:1
nameserver 185.12.64.1
nameserver 9.9.9.9
$ ssh staging2.testrun.org cat /etc/resolv.conf  # after the unbound installation
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.

nameserver 2a01:4ff:ff00::add:2
nameserver 2a01:4ff:ff00::add:1
nameserver 185.12.64.1

[missytake]

(I'll run a second deploy on staging2.testrun.org to make sure 9.9.9.9 isn't re-added on future runs, when unbound isn't installed freshly)

[feld]

My only advice is that if you append the DNS server to the end of the resolv.conf it will still slow down a lot of operations when there are broken DNS servers because glibc will try the DNS servers in the resolv.conf in order, but it is hardcoded to try the server 5 times with a 5 second timeout [1]. This means each failure requires you to wait 25 seconds per broken DNS server.

The cmdeploy will check for 4 DNS records early (A, AAAA, mta-sts CNAME, www CNAME) so the deployment is slowed down by at least 100 seconds just for this operation, not including the DNS lookups required to install packages.

[1] https://github.com/bminor/glibc/blob/master/resolv/resolv.h#L68-L73

[link2xt]

@missytake But this still looks broken, requests should go to 127.0.0.1, not some 185.12.64.1 :/

On hetzner this also looks like this:

# cat /etc/resolv.conf
### Hetzner Online GmbH installimage
# nameserver config
nameserver 127.0.0.1
nameserver 185.12.64.2
nameserver 2a01:4ff:ff00::add:1
nameserver 185.12.64.1
nameserver 2a01:4ff:ff00::add:2

Not something that we actually want, it should only go to 127.0.0.1.

[missytake]

If we write "nameserver 127.0.0.1" to /etc/resolvconf/resolv.conf.d/base after installing unbound, this doesn't make all the other entries disappear, right? I wonder what adds them.

[feld]

systemd has overcomplicated all of this, sadly

One of the most important comments in this doc:

To improve compatibility, /etc/resolv.conf is read in order to discover configured system DNS servers, but only if it is not a symlink to /run/systemd/resolve/stub-resolv.conf, /usr/lib/systemd/resolv.conf or /run/systemd/resolve/resolv.conf (see below).

https://www.freedesktop.org/software/systemd/man/latest/systemd-resolved.service.html