af.mil (US Air Force) partial DNSSEC Outage: 20151211 - 20151212
Updated: December 12, 2015
Overview
This page gives some details on the af.mil partial DNSSEC outage from December 11 to December 12, 2015. This particular outage is a good example of DNSSEC's fragility resulting from unnecessary complexity.
Some resolvers failed to resolve names under af.mil, and some didn't. Some queries in unbound were reported as DNSSEC validation failures in the logs, while some simply returned SERVFAIL while producing no log entries. Some DNSSEC test sites called this a DNSSEC outage and some didn't.
DNSViz / Timeline
- 2015-12-11 00:17:48 UTC: first personally observed DNSSEC failure (see logfile examples)
- 2015-12-11 02:19:39 UTC: Errors and issues with non-obvious implications
- 2015-12-11 03:10:55 UTC: Errors and issues with non-obvious implications
- 2015-12-11 03:10:25 UTC: DNSSEC failure seen by Google (below)
- 2015-12-11 07:03:22 UTC: personally observed queries to unbound succeeding
- 2015-12-11 07:05:26 UTC: Warnings but no errors
- 2015-12-11 07:09:18 UTC: DNSSEC queries to Google Public DNS still failing
- 2015-12-11 15:20:05 UTC: DNSSEC queries to Google Public DNS still failing
- 2015-12-12 00:15:16 UTC: continued DNSSEC failures at Google Public DNS
Zonemaster
Zonemaster archived this af.mil DNSSEC outage, reporting, "Delegation from parent to child is not properly signed (no_dnskey_packet)."
Logfile examples
- [1449793068] unbound[31752:0] info: validation failure <af.mil. A IN>: no signatures from 132.3.13.10
- [1449793227] unbound[31752:0] info: validation failure <af.mil. A IN>: no signatures from 132.3.13.10
- [1449822929] unbound[31752:0] info: validation failure <af.mil. A IN>: no signatures from 132.3.13.10
- [1449823082] unbound[31752:0] info: validation failure <af.mil. A IN>: no signatures from 132.3.13.10
OpenDNS & Google Public DNS
OpenDNS does not support DNSSEC, instead supporting DNSCurve. Google Public DNS currently supports only DNSSEC, and thus, Google's users saw SERVFAIL for queries under af.mil during this outage.
With OpenDNS, queries succeed:
$ dig www.af.mil. @resolver1.opendns.com.
; <<>> DiG 9.4.2-P2 <<>> www.af.mil. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17621
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.af.mil. IN A
;; ANSWER SECTION:
www.af.mil. 167 IN CNAME www.af.mil.edgesuite.net.
www.af.mil.edgesuite.net. 167 IN CNAME a499.dscb.akamai.net.
a499.dscb.akamai.net. 20 IN A 216.206.30.11
a499.dscb.akamai.net. 20 IN A 216.206.30.24
;; Query time: 159 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Dec 11 03:10:11 2015
;; MSG SIZE rcvd: 129
With Google Public DNS, with DNSSEC, queries fail:
$ dig +dnssec www.af.mil. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +dnssec www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16313
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.af.mil. IN A
;; Query time: 18 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 11 03:10:25 2015
;; MSG SIZE rcvd: 39
Google Public DNS: Look Closer
Let's check Google Public DNS with and without DNSSEC.
With checking disabled (dig +cd), queries succeed:
$ dig +cd www.af.mil. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +cd www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28696
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.af.mil. IN A
;; ANSWER SECTION:
www.af.mil. 146 IN CNAME www.af.mil.edgesuite.net.
www.af.mil.edgesuite.net. 299 IN CNAME a499.dscb.akamai.net.
a499.dscb.akamai.net. 19 IN A 216.206.30.11
a499.dscb.akamai.net. 19 IN A 216.206.30.24
;; Query time: 56 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 11 04:33:21 2015
;; MSG SIZE rcvd: 129
With DNSSEC checking, queries fail:
$ dig +dnssec www.af.mil. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +dnssec www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 61060
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.af.mil. IN A
;; Query time: 22 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Dec 11 04:33:19 2015
;; MSG SIZE rcvd: 39
DNSSEC Debugger
Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from December 11, 2015:
