uwm.edu DNSSEC Outage:
2017-12-29 - 2018-01-02

Updated: January 3, 2018

Overview

This page gives some details on the uwm.edu (University of Wisconsin-Milwaukee) DNSSEC outage from December 29, 2017 to January 2, 2018. UWM has around 26,000 students. The outage lasted 4 days and was fixed by completely removing DNSSEC support.

Timeline / DNSViz

2017-12-29 13:11:54 UTC — Bogus DNSSEC delegation
2017-12-29 13:22:38 UTC — Bogus DNSSEC delegation
2017-12-29 17:50:37 UTC — Bogus DNSSEC delegation
2017-12-29 19:46:43 UTC — No DNSKEY, No RRSIGs
2017-12-30 00:03:43 UTC — No DNSKEY, No RRSIGs
2017-12-31 09:37:13 UTC — No DNSKEY, No RRSIGs
2018-01-01 00:00:38 UTC — No DNSKEY, No RRSIGs
2018-01-02 04:04:22 UTC — No DNSKEY, No RRSIGs
2018-01-02 15:15:47 UTC — last personally observed DNSSEC failure
2018-01-02 16:21:15 UTC — Unsigned; DNSSEC outage over

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from December 29, 2017:

Google Public DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec www.uwm.edu. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec www.uwm.edu. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19496
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.uwm.edu. IN A

;; Query time: 68 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 29 18:01:21 2017
;; MSG SIZE rcvd: 40

You have to disable DNSSEC to make DNS queries work:

$ dig +cd www.uwm.edu. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd www.uwm.edu. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23634
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.uwm.edu. IN A

;; ANSWER SECTION:
www.uwm.edu. 599 IN CNAME UWM-PROD-LBR-906207684.us-east-1.elb.amazonaws.com.
UWM-PROD-LBR-906207684.us-east-1.elb.amazonaws.com. 59 IN A 52.7.25.236
UWM-PROD-LBR-906207684.us-east-1.elb.amazonaws.com. 59 IN A 52.54.196.228

;; Query time: 89 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 29 18:01:21 2017
;; MSG SIZE rcvd: 125

Logfile examples