af.mil DNSSEC Outage: 2019-05-04 to 2019-05-06

Updated: May 7, 2019

Overview

This page gives some details on the af.mil DNSSEC outage from May 4 to May 6, 2019. This is one of numerous af.mil DNSSEC outages, and this particular one lasted about 53 hours. The US Air Force has over half a million employees, a budget of $161 Billion, 170 military satellites and enough nuclear weapons to end the human species.

Timeline / DNSViz

DNSViz has been down for about a month at the time of this writing. DNSSEC makes its users completely give up.

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from May 4, 2019:

May 4, 2019 af.mil DNSSEC outage

Google DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

With DNSSEC, DNS queries fail:

$ dig +dnssec www.af.mil. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 55883
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.af.mil. IN A

;; Query time: 166 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat May 4 22:05:56 2019
;; MSG SIZE rcvd: 39

You have to disable DNSSEC to make DNS queries work:

$ dig +cd www.af.mil. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46123
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.af.mil. IN A

;; ANSWER SECTION:
www.af.mil. 87 IN CNAME www.afpw.af.mil.edgekey.net.
www.afpw.af.mil.edgekey.net. 299 IN CNAME e11218.dscb.akamaiedge.net.
e11218.dscb.akamaiedge.net. 19 IN A 23.39.18.201

;; Query time: 60 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat May 4 22:06:05 2019
;; MSG SIZE rcvd: 122

Zonemaster

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

;; Domain: af.mil.
[B] af.mil. 144858 IN DNSKEY 257 3 8 ;{id = 48652 (ksk), size = 2048b}
af.mil. 144858 IN DNSKEY 256 3 8 ;{id = 14268 (zsk), size = 2048b}
[B] Error verifying denial of existence for af.mil. type A: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples