fedoraproject.org DNSSEC Outage: 2017-09-06

Updated: September 21, 2017

Overview

This page gives some details on the fedoraproject.org DNSSEC outage on September 6, 2017. It also affected fedorahosted.org.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from September 6, 2017:

September 6, 2017 fedoraproject.org DNSSEC outage

Zonemaster

Twitter

This DNSSEC outage was discussed on Twitter.

@reseauxsansfil writes: Looks like the @fedora Project's #DNSSEC problems have been fixed for the moment. Algo 10 keys removed, DS record fixed.

@PCTuning_OW writes: #SystemD's AI decided it hates #DNSSEC, so it killed [Fedora]

@PlatformPatrick writes: @fedora I'm unable to resolve http://fedoraproject.org against googledns dig @8.8.8.8 http://fedoraproject.org SERVFAIL. DNSSEC issues?

@marcodavids writes: Kaputt... #DNSSEC @fedora http://dnsviz.net/d/fedoraproject.org/Wa_ljA/dnssec/

@ptvician writes: Hi @fedora, you have broken DNSSEC on http://fedoraproject.org , see http://dnsviz.net/d/fedoraproject.org/dnssec/ Could you please fix it?

@bortzmeyer writes: #DNSSEC blunder by the Fedora people http://dnsviz.net/d/fedoraproject.org/WbALHQ/dnssec/   Now fixed but some DNS resolvers still have the old info (time to live, 24 h)

OARC Public DNS, with/without DNSSEC

With DNSSEC enabled, queries fail:

$ dig +dnssec a fedoraproject.org. @184.105.193.73

; <<>> DiG 9.10.3-P4-Debian <<>> +dnssec a fedoraproject.org. @184.105.193.73
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52229
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;fedoraproject.org. IN A

;; Query time: 645 msec
;; SERVER: 184.105.193.73#53(184.105.193.73)
;; WHEN: Wed Sep 06 07:48:00 UTC 2017
;; MSG SIZE rcvd: 46

You have to disable DNSSEC to make DNS work:

$ dig +cd a fedoraproject.org. @184.105.193.73

; <<>> DiG 9.10.3-P4-Debian <<>> +cd a fedoraproject.org. @184.105.193.73
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21947
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 10, AUTHORITY: 3, ADDITIONAL: 6

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;fedoraproject.org. IN A

;; ANSWER SECTION:
fedoraproject.org. 60 IN A 209.132.181.15
fedoraproject.org. 60 IN A 8.43.85.67
fedoraproject.org. 60 IN A 209.132.181.16
fedoraproject.org. 60 IN A 140.211.169.206
fedoraproject.org. 60 IN A 67.219.144.68
fedoraproject.org. 60 IN A 67.203.2.67
fedoraproject.org. 60 IN A 152.19.134.142
fedoraproject.org. 60 IN A 66.35.62.162
fedoraproject.org. 60 IN A 152.19.134.198
fedoraproject.org. 60 IN A 140.211.169.196

;; AUTHORITY SECTION:
fedoraproject.org. 85805 IN NS ns02.fedoraproject.org.
fedoraproject.org. 85805 IN NS ns04.fedoraproject.org.
fedoraproject.org. 85805 IN NS ns05.fedoraproject.org.

;; ADDITIONAL SECTION:
ns02.fedoraproject.org. 85805 IN A 152.19.134.139
ns04.fedoraproject.org. 85805 IN A 209.132.181.17
ns05.fedoraproject.org. 85805 IN A 85.236.55.10
ns02.fedoraproject.org. 85805 IN AAAA 2610:28:3090:3001:dead:beef:cafe:fed5
ns05.fedoraproject.org. 85805 IN AAAA 2001:4178:2:1269:dead:beef:cafe:fed5

;; Query time: 623 msec
;; SERVER: 184.105.193.73#53(184.105.193.73)
;; WHEN: Wed Sep 06 07:48:00 UTC 2017
;; MSG SIZE rcvd: 367

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

;; Domain: fedoraproject.org.
;; Signature ok but no chain to a trusted key or ds record
[S] fedoraproject.org. 300 IN DNSKEY 257 3 10 ;{id = 39894 (ksk), size = 4096b}
fedoraproject.org. 300 IN DNSKEY 257 3 5 ;{id = 16207 (ksk), size = 2048b}
fedoraproject.org. 300 IN DNSKEY 256 3 10 ;{id = 34913 (zsk), size = 4096b}
fedoraproject.org. 300 IN DNSKEY 256 3 5 ;{id = 7725 (zsk), size = 1024b}
[S] fedoraproject.org. 60 IN A 152.19.134.198
fedoraproject.org. 60 IN A 67.219.144.68
fedoraproject.org. 60 IN A 67.203.2.67
fedoraproject.org. 60 IN A 140.211.169.196
fedoraproject.org. 60 IN A 66.35.62.162
fedoraproject.org. 60 IN A 140.211.169.206
fedoraproject.org. 60 IN A 209.132.181.16
fedoraproject.org. 60 IN A 152.19.134.142
fedoraproject.org. 60 IN A 209.132.181.15
fedoraproject.org. 60 IN A 8.43.85.67
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples