af.mil DNSSEC Outage: 2015-04-28 to 2015-04-29
Updated: April 29, 2015
Overview
This page gives some details on the af.mil DNSSEC outages beginning April 28, 2015, and continuing to April 29, 2015. The outage began with subdomains and then expanded to all of af.mil. The outage lasted about a day.
Verisign's DNSSEC Debugger
Verisign doesn't archive test results, unlike DNSViz. So here's a screenshot I took on April 29, 2015:
Timeline / DNSViz
af.mil failures:
- 2015-04-28 23:53:17 UTC — bogus af.mil signatures
- 2015-04-29 00:38:05 UTC — bogus af.mil signatures
- 2015-04-29 03:08:49 UTC — bogus af.mil signatures
- 2015-04-29 07:23:32 UTC — bogus af.mil signatures
- 2015-04-29 11:36:15 UTC — bogus af.mil signatures
- 2015-04-29 12:27:35 UTC — bogus af.mil signatures
- 2015-04-29 14:33:30 UTC — bogus DNSSEC delegation
- 2015-04-29 15:54:58 UTC — bogus DNSSEC delegation
- 2015-04-29 18:57:54 UTC — af.mil is unsigned and DNSSEC removed; outage fixed
Subdomain failures from earlier in the day on April 28, 2015:
- 2015-04-28 09:49:42 UTC — spangdahlem.af.mil/{MX,A,SOA}
- 2015-04-28 13:17:22 UTC — robins.af.mil/{MX,SOA}
- 2015-04-28 13:49:27 UTC — aetc.af.mil/{MX,SOA}
- 2015-04-28 14:36:23 UTC — afrc.af.mil/{A,SOA,MX,TXT}
- 2015-04-28 15:47:46 UTC — law.af.mil/SOA
- 2015-04-28 16:12:26 UTC — pacaf.af.mil/{MX,SOA}
- 2015-04-28 16:24:31 UTC — edwards.af.mil/{MX,SOA}
- 2015-04-28 16:26:15 UTC — www.edwards.af.mil/A
- 2015-04-28 16:29:38 UTC — www.afrc.af.mil/A
- 2015-04-28 17:58:02 UTC — www.aetc.af.mil/A
- 2015-04-28 18:20:20 UTC — www.robins.af.mil/A
- 2015-04-28 19:11:57 UTC — www.law.af.mil/A
- etc.
OpenDNS & Google Public DNS
OpenDNS does not support DNSSEC, instead supporting DNSCurve. Google Public DNS currently supports only DNSSEC, and thus, Google's users saw SERVFAIL for queries under af.mil during this outage.
With OpenDNS, queries succeed:
$ dig www.af.mil. @resolver1.opendns.com
; <<>> DiG 9.4.2-P2 <<>> www.af.mil. @resolver1.opendns.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29681
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.af.mil. IN A
;; ANSWER SECTION:
www.af.mil. 300 IN CNAME www.af.mil.edgesuite.net.
www.af.mil.edgesuite.net. 300 IN CNAME a499.dscb.akamai.net.
a499.dscb.akamai.net. 20 IN A 23.212.53.206
a499.dscb.akamai.net. 20 IN A 23.212.53.196
;; Query time: 175 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Wed Apr 29 15:41:59 2015
;; MSG SIZE rcvd: 129
With Google Public DNS, with DNSSEC, queries fail:
$ dig www.af.mil. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> www.af.mil. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30636
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.af.mil. IN A
;; Query time: 96 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Apr 29 15:42:04 2015
;; MSG SIZE rcvd: 28
dnscheck
dnscheck.iis.se archived a DNSSEC outage at 2015-04-28 18:54:02 (requires javascript).
dnscheck.labs.nic.cz archived a DNSSEC outage at 2015-04-28 18:54:38 (requires javascript).
Zonemaster
Zonemaster archived this af.mil DNSSEC outage.
Logfile examples
- [1430233475] unbound[18328:0] info: validation failure <pacaf.af.mil. A IN>: signature crypto failed for <pacaf.af.mil. SOA IN> from 132.3.57.10
- [1430233858] unbound[18328:0] info: validation failure <osan.af.mil. A IN>: signature crypto failed for <osan.af.mil. SOA IN> from 132.3.13.10
- [1430234149] unbound[18328:0] info: validation failure <302aw.afrc.af.mil. A IN>: signature crypto failed from 132.3.13.10 for key afrc.af.mil. while building chain of trust
- [1430234214] unbound[18328:0] info: validation failure <usafe.af.mil. A IN>: signature crypto failed from 132.3.25.10
- [1430234235] unbound[18328:0] info: validation failure <westover.afrc.af.mil. A IN>: signature crypto failed from 132.3.57.10 for key afrc.af.mil. while building chain of trust
- [1430234463] unbound[18328:0] info: validation failure <minneapolis.afrc.af.mil. A IN>: signature crypto failed from 132.3.29.10 for key afrc.af.mil. while building chain of trust
- [1430234520] unbound[18328:0] info: validation failure <offutt.af.mil. A IN>: signature crypto failed for <offutt.af.mil. SOA IN> from 132.3.45.10
- [1430234705] unbound[18328:0] info: validation failure <holloman.af.mil. A IN>: signature crypto failed for <holloman.af.mil. SOA IN> from 132.3.53.10
- [1430234754] unbound[18328:0] info: validation failure <arpc.afrc.af.mil. A IN>: signature crypto failed from 132.3.25.10 for key afrc.af.mil. while building chain of trust
- [1430234759] unbound[18328:0] info: validation failure <445aw.afrc.af.mil. A IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <arpc.afrc.af.mil. A IN>: signature crypto failed from 132.3.25.10 for key afrc.af.mil. while building chain of trust
- [1430235048] unbound[18328:0] info: validation failure <law.af.mil. A IN>: signature crypto failed from 132.3.53.10 for key law.af.mil. while building chain of trust
- [1430235192] unbound[18328:0] info: validation failure <edwards.af.mil. A IN>: signature crypto failed for <edwards.af.mil. SOA IN> from 132.3.53.10
- [1430235322] unbound[18328:0] info: validation failure <patrick.af.mil. A IN>: signature crypto failed for <patrick.af.mil. SOA IN> from 132.3.13.10
- [1430235899] unbound[18328:0] info: validation failure <mountainhome.af.mil. A IN>: signature crypto failed for <mountainhome.af.mil. SOA IN> from 132.3.65.10
- [1430236075] unbound[18328:0] info: validation failure <916arw.afrc.af.mil. A IN>: signature crypto failed from 132.3.45.10 for key afrc.af.mil. while building chain of trust
- [1430236084] unbound[18328:0] info: validation failure <afrc.af.mil. A IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <916arw.afrc.af.mil. A IN>: signature crypto failed from 132.3.45.10 for key afrc.af.mil. while building chain of trust
- [1430236497] unbound[18328:0] info: validation failure <eglin.af.mil. A IN>: signature crypto failed for <eglin.af.mil. SOA IN> from 132.3.49.10
- [1430237009] unbound[18328:0] info: validation failure <spangdahlem.af.mil. A IN>: signature crypto failed from 132.3.13.10
- [1430237215] unbound[18328:0] info: validation failure <wbits.afrc.af.mil. A IN>: signature crypto failed from 132.3.45.10 for key afrc.af.mil. while building chain of trust
- [1430237281] unbound[18328:0] info: validation failure <5af.pacaf.af.mil. A IN>: signature crypto failed for <pacaf.af.mil. SOA IN> from 132.3.13.10
- [1430238145] unbound[18328:0] info: validation failure <afnoc.af.mil. A IN>: signature crypto failed for <AFNOC.AF.MIL. SOA IN> from 132.3.1.10
- [1430243835] unbound[27543:0] info: validation failure <www.aetc.af.mil. A IN>: signature crypto failed for CNAME from 167.142.232.220 and 23.61.199.64 and 132.3.13.10
- [1430243852] unbound[27543:0] info: validation failure <www.edwards.af.mil. A IN>: signature crypto failed for CNAME from 24.143.193.61 and 23.74.25.64 and 132.3.53.10
- [1430243884] unbound[27543:0] info: validation failure <www.tinker.af.mil. A IN>: signature crypto failed for CNAME from 167.142.232.215 and 184.85.248.65 and 132.3.25.10
- [1430243920] unbound[27543:0] info: validation failure <www.robins.af.mil. A IN>: signature crypto failed for CNAME from 167.142.232.215 and 23.61.199.64 and 132.3.13.10
- [1430243993] unbound[27543:0] info: validation failure <www.usafe.af.mil. A IN>: signature crypto failed for CNAME from 167.142.232.213 and 193.108.91.2 and 132.3.13.10
- [1430244052] unbound[27543:0] info: validation failure <www.eglin.af.mil. A IN>: signature crypto failed for CNAME from 184.51.40.7 and 23.61.199.64 and 132.3.49.10
- [1430247963] unbound[552:0] info: validation failure <aetc.af.mil. MX IN>: signature crypto failed from 132.3.13.10
- [1430247976] unbound[552:0] info: validation failure <edwards.af.mil. MX IN>: signature crypto failed from 132.3.21.10
- [1430247989] unbound[552:0] info: validation failure <minneapolis.afrc.af.mil. MX IN>: signature crypto failed from 132.3.13.10 for key afrc.af.mil. while building chain of trust
- [1430248005] unbound[552:0] info: validation failure <302aw.afrc.af.mil. MX IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <minneapolis.afrc.af.mil. MX IN>: signature crypto failed from 132.3.13.10 for key afrc.af.mil. while building chain of trust
- [1430248018] unbound[552:0] info: validation failure <spangdahlem.af.mil. MX IN>: signature crypto failed from 132.3.13.10
- [1430248035] unbound[552:0] info: validation failure <robins.af.mil. MX IN>: signature crypto failed from 132.3.13.10
- [1430248048] unbound[552:0] info: validation failure <arpc.afrc.af.mil. MX IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <minneapolis.afrc.af.mil. MX IN>: signature crypto failed from 132.3.13.10 for key afrc.af.mil. while building chain of trust
- [1430248061] unbound[552:0] info: validation failure <445aw.afrc.af.mil. MX IN>: signature crypto failed from 132.3.57.10 for key afrc.af.mil. while building chain of trust
- [1430248075] unbound[552:0] info: validation failure <law.af.mil. MX IN>: signature crypto failed from 132.3.37.10 for key law.af.mil. while building chain of trust
- [1430248087] unbound[552:0] info: validation failure <916arw.afrc.af.mil. MX IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <445aw.afrc.af.mil. MX IN>: signature crypto failed from 132.3.57.10 for key afrc.af.mil. while building chain of trust
- [1430248101] unbound[552:0] info: validation failure <usafe.af.mil. MX IN>: signature crypto failed from 132.3.13.10
- [1430248114] unbound[552:0] info: validation failure <wbits.afrc.af.mil. MX IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <445aw.afrc.af.mil. MX IN>: signature crypto failed from 132.3.57.10 for key afrc.af.mil. while building chain of trust
- [1430248127] unbound[552:0] info: validation failure <5af.pacaf.af.mil. MX IN>: signature crypto failed for <pacaf.af.mil. SOA IN> from 132.3.13.10
- [1430248140] unbound[552:0] info: validation failure <eglin.af.mil. MX IN>: signature crypto failed from 132.3.61.10
- [1430248152] unbound[552:0] info: validation failure <westover.afrc.af.mil. MX IN>: signature crypto failed from 132.3.45.10 for key afrc.af.mil. while building chain of trust
- [1430248165] unbound[552:0] info: validation failure <pacaf.af.mil. MX IN>: signature crypto failed from 132.3.13.10
- [1430248177] unbound[552:0] info: validation failure <afrc.af.mil. MX IN>: key for validation afrc.af.mil. is marked as invalid because of a previous validation failure <westover.afrc.af.mil. MX IN>: signature crypto failed from 132.3.45.10 for key afrc.af.mil. while building chain of trust
- [1430273193] unbound[3218:0] info: validation failure <af.mil. A IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.25.10
- [1430323186] unbound[11904:0] info: validation failure <www.af.mil. A IN>: No DNSKEY record from 132.3.13.10 for key af.mil. while building chain of trust
- [1430328878] unbound[559:0] info: validation failure <af.mil. NS IN>: No DNSKEY record from 132.3.13.10 for key af.mil. while building chain of trust