ucr.edu DNSSEC Outage:
2018-04-04 to 2018-04-05
Updated: April 16, 2018
Overview
This page gives some details on the ucr.edu (University of California, Riverside) DNSSEC outage from April 4 to April 5, 2018. UCR has around 23,000 students.
Timeline / DNSViz
- 2018-04-04 22:58:02 UTC — Bogus DNSSEC delegation
- 2018-04-04 22:59:38 UTC — Bogus DNSSEC delegation
- 2018-04-04 23:02:02 UTC — Bogus DNSSEC delegation
- 2018-04-04 23:04:25 UTC — Bogus DNSSEC delegation
- 2018-04-04 23:17:41 UTC — Bogus DNSSEC delegation
- 2018-04-04 23:42:52 UTC — Bogus DNSSEC delegation
- 2018-04-05 00:15:18 UTC — last personally observed DNSSEC failure
- 2018-04-05 00:32:18 UTC — Outage over, but DNSSEC outage debris remain
DNSSEC Debugger
Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from April 4, 2018.
Zonemaster
- zonemaster.net archived "Delegation from parent to child is not properly signed (no_dnskey)."
DNS-OARC: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:
$ dig +dnssec a ucr.edu. @184.105.193.74
; <<>> DiG 9.4.2-P2 <<>> +dnssec a ucr.edu. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 45461
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;ucr.edu. IN A
;; Query time: 36 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Wed Apr 4 22:59:17 2018
;; MSG SIZE rcvd: 36
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a ucr.edu. @184.105.193.74
; <<>> DiG 9.4.2-P2 <<>> +cd a ucr.edu. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 223
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 4
;; QUESTION SECTION:
;ucr.edu. IN A
;; ANSWER SECTION:
ucr.edu. 900 IN A 138.23.226.208
;; AUTHORITY SECTION:
ucr.edu. 28800 IN NS ns2.ucr.edu.
ucr.edu. 28800 IN NS ns1.ucr.edu.
ucr.edu. 28800 IN NS sdns1.berkeley.edu.
;; ADDITIONAL SECTION:
ns1.ucr.edu. 28800 IN AAAA 2607:f290:9::100
ns2.ucr.edu. 28800 IN AAAA 2607:f290:9:1::100
ns1.ucr.edu. 28800 IN A 138.23.80.10
ns2.ucr.edu. 28800 IN A 138.23.80.20
;; Query time: 20 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Wed Apr 4 22:59:17 2018
;; MSG SIZE rcvd: 194
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):
;; Domain: ucr.edu.
[B] ucr.edu. 172800 IN DNSKEY 256 3 10 ;{id = 26666 (zsk), size = 1024b}
ucr.edu. 172800 IN DNSKEY 256 3 10 ;{id = 15005 (zsk), size = 1024b}
[B] ucr.edu. 900 IN A 138.23.226.208
;; Error: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1522882752] unbound[26322:0] info: validation failure <ucr.edu. A IN>: no keys have a DS with algorithm RSASHA512 from 138.23.80.10 for key ucr.edu. while building chain of trust
- [1522883434] unbound[26322:0] info: validation failure <cs.ucr.edu. A IN>: no keys have a DS with algorithm RSASHA512 from 138.23.80.10 for key ucr.edu. while building chain of trust
- [1522887318] unbound[26322:0] info: validation failure <faculty.ucr.edu. A IN>: no keys have a DS with algorithm RSASHA512 from 128.32.136.251 for key ucr.edu. while building chain of trust