af.mil DNSSEC Outage: 2020-09-16
Date: September 16, 2020
Overview
This page gives some details on the af.mil DNSSEC outage on September 16, 2020. This is one of numerous af.mil DNSSEC outages. The US Air Force has over half a million employees, a budget of $161 Billion, 170 military satellites and enough nuclear weapons to end the human species.
Timeline / DNSViz
I've included a screenshot of DNSViz output since DNSSEC people don't care if things work or not.
- 2020-09-16 04:11:50 UTC — Bogus RRSIGs
- 2020-09-16 05:43:40 UTC — Bogus RRSIGs
- 2020-09-16 18:51:55 UTC — Bogus RRSIGs
- 2020-09-16 19:25:34 UTC — DNSSEC outage kind of over
- 2020-09-16 21:09:09 UTC — last personally observed af.mil DNSSEC failure

DNSSEC Debugger
Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from September 16, 2020:

drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
;; Domain: af.mil.
[B] af.mil. 60016 IN DNSKEY 256 3 8 ;{id = 47250 (zsk), size = 2048b}
af.mil. 60016 IN DNSKEY 256 3 8 ;{id = 64236 (zsk), size = 2048b}
af.mil. 60016 IN DNSKEY 256 3 8 ;{id = 44976 (zsk), size = 2048b}
af.mil. 60016 IN DNSKEY 257 3 8 ;{id = 56521 (ksk), size = 2048b}
af.mil. 60016 IN DNSKEY 257 3 8 ;{id = 9826 (ksk), size = 2048b}
[B] Error verifying denial of existence for af.mil. type A: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
These logs come from different servers in different geographical regions:
- [1600225450] unbound[88846:0] info: validation failure <www.af.mil. A IN>: signature crypto failed from 132.3.29.10 for key af.mil. while building chain of trust
- [1600236842] unbound[25550:0] info: validation failure <af.mil. A IN>: signature crypto failed from 132.3.57.10 for key af.mil. while building chain of trust
- [1600231622] unbound[88846:0] info: validation failure <www.af.mil. A IN>: signature crypto failed from 132.3.65.10 for key af.mil. while building chain of trust
- [1600244363] unbound[25550:0] info: validation failure <www.af.mil. A IN>: signature crypto failed from 132.3.41.10 for key af.mil. while building chain of trust
- [1600241372] unbound[88846:0] info: validation failure <af.mil. A IN>: signature crypto failed from 132.3.25.10 for key af.mil. while building chain of trust
- [1600287518] unbound[25550:0] info: validation failure <af.mil. A IN>: signature crypto failed from 132.3.13.10 for key af.mil. while building chain of trust
- [1600290549] unbound[88846:0] info: validation failure <af.mil. A IN>: signature crypto failed from 132.3.57.10 for key af.mil. while building chain of trust