Fedora 35: Name resolver fail - Solved!

One of my boxes started failing after working pretty ok for years. On dnf update it simply said:

Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]

Whoa! Where did that come from?

This is one of my test / devel / not-so-important boxes, so it might have been broken for a while. I simply haven't realized it ever happening.

Troubleshooting

Step 1: Making sure DNS works

Test: dig mirrors.fedoraproject.org

Result:

;; ANSWER SECTION:
mirrors.fedoraproject.org. 106 IN CNAME wildcard.fedoraproject.org.
wildcard.fedoraproject.org. 44 IN A 185.141.165.254
wildcard.fedoraproject.org. 44 IN A 152.19.134.142
wildcard.fedoraproject.org. 44 IN A 18.192.40.85
wildcard.fedoraproject.org. 44 IN A 152.19.134.198
wildcard.fedoraproject.org. 44 IN A 38.145.60.21
wildcard.fedoraproject.org. 44 IN A 18.133.140.134
wildcard.fedoraproject.org. 44 IN A 209.132.190.2
wildcard.fedoraproject.org. 44 IN A 18.159.254.57
wildcard.fedoraproject.org. 44 IN A 38.145.60.20
wildcard.fedoraproject.org. 44 IN A 85.236.55.6

Conclusion: Network works (obviously, I just SSHd into the box) and capability of doing DNS-requests works ok.

Step 2: What's in /etc/resolv.conf?

Test: cat /etc/resolv.conf

Result:

options edns0 trust-ad
; generated by /usr/sbin/dhclient-script
nameserver 185.12.64.1
nameserver 185.12.64.2

Test 2: ls -l /etc/resolv.conf

Result:

lrwxrwxrwx. 1 root root 39 Apr 25 20:31 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Conclusion: Well, well. dhclient isn't supposed to overwrite /etc/resolv.conf if systemd-resolver is used. But is it?

Step 3: Is /etc/nsswitch.conf ok?

Previous one gave hint something was not ok. Just to cover all the bases, need to verify NS switch order.

Test: cat /etc/nsswitch.conf

Result:

# Generated by authselect on Wed Apr 29 05:44:18 2020
# Do not modify this file manually.

# If you want to make changes to nsswitch.conf please modify
# /etc/authselect/user-nsswitch.conf and run 'authselect apply-changes'.

d...

# In order of likelihood of use to accelerate lookup.
hosts:      files myhostname resolve [!UNAVAIL=return] dns

Conclusion: No issues there, all ok.

Step 4: Is systemd-resolved running?

Test: systemctl status systemd-resolved

Result:

● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor pr>
     Active: active (running) since Mon 2022-04-25 20:15:59 CEST; 7min ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configurat>
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 6329 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 2258)
     Memory: 8.5M
        CPU: 91ms
     CGroup: /system.slice/systemd-resolved.service
             └─6329 /usr/lib/systemd/systemd-resolved

Conclusion: No issues there, all ok.

Step 5: If DNS resolution is there, can systemd-resolved do any resolving?

Test: systemd-resolve --status

Result:

Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Test 2: resolvectl query mirrors.fedoraproject.org

Result:

mirrors.fedoraproject.org: resolve call failed: No appropriate name servers or networks for name found

Conclusion: Problem found! systemd-resolved and dhclient are disconnected.

Fix

Edit file /etc/systemd/resolved.conf, have it contain Hetzner recursive resolvers in a line:

DNS=185.12.64.1 185.12.64.2

Make changes effective: systemctl restart systemd-resolved

Test: resolvectl query mirrors.fedoraproject.org

As everything worked and correct result was returned, verify systemd-resolverd status: systemd-resolve --status

Result now:

Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
Current DNS Server: 185.12.64.1
       DNS Servers: 185.12.64.1 185.12.64.2

Link 2 (eth0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Success!

Why it broke? No idea.
I just hate these modern Linuxes where every single thing gets more layers of garbage added between the actual system and admin. Ufff.