af.mil (US Air Force) partial DNSSEC Outage: 20151211 - 20151212

Updated: December 12, 2015

Overview

This page gives some details on the af.mil partial DNSSEC outage from December 11 to December 12, 2015. This particular outage is a good example of DNSSEC's fragility resulting from unnecessary complexity.

Some resolvers failed to resolve names under af.mil, and some didn't. Some queries in unbound were reported as DNSSEC validation failures in the logs, while some simply returned SERVFAIL while producing no log entries. Some DNSSEC test sites called this a DNSSEC outage and some didn't.

DNSViz / Timeline

Zonemaster

Zonemaster archived this af.mil DNSSEC outage, reporting, "Delegation from parent to child is not properly signed (no_dnskey_packet)."

Logfile examples

OpenDNS & Google Public DNS

OpenDNS does not support DNSSEC, instead supporting DNSCurve. Google Public DNS currently supports only DNSSEC, and thus, Google's users saw SERVFAIL for queries under af.mil during this outage.

With OpenDNS, queries succeed:

$ dig www.af.mil. @resolver1.opendns.com.

; <<>> DiG 9.4.2-P2 <<>> www.af.mil. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17621
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.af.mil. IN A

;; ANSWER SECTION:
www.af.mil. 167 IN CNAME www.af.mil.edgesuite.net.
www.af.mil.edgesuite.net. 167 IN CNAME a499.dscb.akamai.net.
a499.dscb.akamai.net. 20 IN A 216.206.30.11
a499.dscb.akamai.net. 20 IN A 216.206.30.24

;; Query time: 159 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Dec 11 03:10:11 2015
;; MSG SIZE rcvd: 129

With Google Public DNS, with DNSSEC, queries fail:

$ dig +dnssec www.af.mil. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16313
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.af.mil. IN A

;; Query time: 18 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 11 03:10:25 2015
;; MSG SIZE rcvd: 39

Google Public DNS: Look Closer

Let's check Google Public DNS with and without DNSSEC.

With checking disabled (dig +cd), queries succeed:

$ dig +cd www.af.mil. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28696
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.af.mil. IN A

;; ANSWER SECTION:
www.af.mil. 146 IN CNAME www.af.mil.edgesuite.net.
www.af.mil.edgesuite.net. 299 IN CNAME a499.dscb.akamai.net.
a499.dscb.akamai.net. 19 IN A 216.206.30.11
a499.dscb.akamai.net. 19 IN A 216.206.30.24

;; Query time: 56 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 11 04:33:21 2015
;; MSG SIZE rcvd: 129

With DNSSEC checking, queries fail:

$ dig +dnssec www.af.mil. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 61060
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.af.mil. IN A

;; Query time: 22 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 11 04:33:19 2015
;; MSG SIZE rcvd: 39

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from December 11, 2015:

December 11 2015 af.mil (US Air Force) DNSSEC outage