gsu.edu DNSSEC Outage: 2019-06-04
Date: June 4, 2019
Overview
This page gives some details on the gsu.edu (Georgia State University) DNSSEC outage on June 4, 2019.
Timeline / DNSViz
At the time of this writing, DNSViz historical archives have been failing for a couple months, and may be permanently down. DNSSEC makes its users completely give up.
I've included screenshots of DNSViz output since DNSSEC people don't seem to care if things work or not.
- 2019-06-04 11:14:03 UTC — first personally observed gsu.edu DNSSEC failure
- 2019-06-04 15:18:56 UTC — last personally observed gsu.edu DNSSEC failure
DNSSEC Debugger
Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from June 4, 2019.
Zonemaster
- zonemaster.iis.se archived "Trying to verify NSEC RRset with RRSIG 59262 gave error 'DNSSEC signature has expired'."
- zonemaster.labs.nic.cz archived "Trying to verify NSEC RRset with RRSIG 59262 gave error 'DNSSEC signature has expired'."
Google Public DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:
$ dig +dnssec a gsu.edu. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +dnssec a gsu.edu. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 42484
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;gsu.edu. IN A
;; Query time: 69 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Jun 4 12:02:11 2019
;; MSG SIZE rcvd: 36
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a gsu.edu. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +cd a gsu.edu. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21782
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;gsu.edu. IN A
;; ANSWER SECTION:
gsu.edu. 3599 IN A 166.78.46.98
;; Query time: 66 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Jun 4 12:02:11 2019
;; MSG SIZE rcvd: 41
dns.google.com
dns.google.com is related to but separate from Google Public DNS. During this DNSSEC outage, dns.google.com showed the following for gsu.edu:
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):
;; Domain: gsu.edu.
[T] gsu.edu. 3600 IN DNSKEY 257 3 5 ;{id = 30514 (ksk), size = 2048b}
gsu.edu. 3600 IN DNSKEY 256 3 5 ;{id = 59262 (zsk), size = 1024b}
gsu.edu. 3600 IN DNSKEY 256 3 5 ;{id = 33046 (zsk), size = 1024b}
gsu.edu. 3600 IN DNSKEY 257 3 5 ;{id = 3547 (ksk), size = 2048b}
[B] gsu.edu. 3600 IN A 166.78.46.98
;; Error: DNSSEC signature has expired
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1559646843] unbound[28610:0] info: validation failure <tinman.cs.gsu.edu. A IN>: signature expired from 131.96.7.116 for DS cs.gsu.edu. while building chain of trust
- [1559651654] unbound[28610:0] info: validation failure <gsu.edu. A IN>: signature expired from 131.144.4.9
- [1559661536] unbound[28610:0] info: validation failure <tinman.cs.gsu.edu. A IN>: signature expired from 131.96.7.124 for DS cs.gsu.edu. while building chain of trust