af.mil partial DNSSEC Outage: 2014-06-09
Date: June 9, 2014
Overview
This page gives some details on the partial af.mil DNSSEC outage of June 9, 2014.
Verisign's DNSSEC Debugger
Verisign doesn't archive test results, unlike DNSViz. So here's a screenshot I took on June 9, 2014:
DNSViz
DNSViz archived DNSSEC issues that don't show the full impact of the outage at 2014-06-10 02:10:51 UTC and 2014-06-10 02:30:04 UTC. Some subdomains, such as hq.af.mil and macdill.af.mil were outright failing.
OpenDNS & Google Public DNS
OpenDNS does not support DNSSEC, instead supporting DNSCurve. Google Public DNS currently supports only DNSSEC, and thus, Google's users saw SERVFAIL for many but not all queries for names under af.mil during this outage.
With OpenDNS, queries succeed:
$ dig mx af.mil @resolver1.opendns.com
; <<>> DiG 9.4.2-P2 <<>> mx af.mil @resolver1.opendns.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10910
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;af.mil. IN MX
;; AUTHORITY SECTION:
af.mil. 500 IN SOA
langley-ns10.afnoc.af.mil. dnsman.afnet.af.mil. 2014060802 3600 360 604800 500
;; Query time: 292 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Mon Jun 9 21:18:21 2014
;; MSG SIZE rcvd: 104
With Google Public DNS, queries fail:
$ dig mx af.mil @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> mx af.mil @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 40389
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;af.mil. IN MX
;; Query time: 103 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 9 21:18:02 2014
;; MSG SIZE rcvd: 24
Logfile examples
- [1402366312] unbound[20031:0] info: validation failure <afcaf.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.29.10
- [1402366373] unbound[20031:0] info: validation failure <afhso.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.65.10
- [1402366568] unbound[20031:0] info: validation failure <wingmanmagazine.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402366667] unbound[20031:0] info: validation failure <353sog.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402366669] unbound[20031:0] info: validation failure <retirees.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.25.10
- [1402366738] unbound[20031:0] info: validation failure <housing.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402366768] unbound[20031:0] info: validation failure <afhra.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402366933] unbound[20031:0] info: validation failure <beready.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402366937] unbound[20031:0] info: validation failure <netcents.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402366991] unbound[20031:0] info: validation failure <usafeuropeband.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367011] unbound[20031:0] info: validation failure <afms.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367019] unbound[20031:0] info: validation failure <whasc.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367062] unbound[20031:0] info: validation failure <afsec.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367096] unbound[20031:0] info: validation failure <chaplaincorps.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367203] unbound[20031:0] info: validation failure <music.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.65.10
- [1402367262] unbound[20031:0] info: validation failure <62aw.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367290] unbound[20031:0] info: validation failure <nationalmuseum.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.65.10
- [1402367438] unbound[20031:0] info: validation failure <airforcehollywood.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367460] unbound[20031:0] info: validation failure <89aw.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.25.10
- [1402367610] unbound[20031:0] info: validation failure <jble.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367685] unbound[20031:0] info: validation failure <5wing.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367727] unbound[20031:0] info: validation failure <afoutreach.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367738] unbound[20031:0] info: validation failure <79mdw.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402367890] unbound[20031:0] info: validation failure <37trw.af.mil. NS IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402368435] unbound[20031:0] info: validation failure <housing.af.mil. MX IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.65.10
- [1402368440] unbound[20031:0] info: validation failure <housing.af.mil. ANY IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.25.10
- [1402368446] unbound[20031:0] info: validation failure <af.mil. ANY IN>: signature crypto failed from 132.3.57.10
- [1402369055] unbound[20031:0] info: validation failure <af.mil. MX IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402369079] unbound[20031:0] info: validation failure <af.mil. AAAA IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402369183] unbound[20031:0] info: validation failure <housing.af.mil. ANY IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402369184] unbound[20031:0] info: validation failure <62aw.af.mil. ANY IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402369185] unbound[20031:0] info: validation failure <nationalmuseum.af.mil. ANY IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402369186] unbound[20031:0] info: validation failure <jble.af.mil. ANY IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.13.10
- [1402369311] unbound[20031:0] info: validation failure <af.mil. TXT IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.25.10
- [1402369329] unbound[20031:0] info: validation failure <housing.af.mil. TXT IN>: signature crypto failed for <af.mil. SOA IN> from 132.3.29.10
- [1402438595] unbound[17483:0] info: validation failure <qrzvffyruoqdxehr.af.mil. SOA IN>: signature crypto failed for <F6TLK41R806N05HGBCSM2UKCJ28MAM7R.af.mil. NSEC3 IN> from 132.3.45.10
- [1402439009] unbound[17483:0] info: validation failure <qlpqgyyounrrfxsy.62aw.af.mil. MX IN>: signature crypto failed for <NMA3M9UN511IPE7OP7G57907N729LKO5.af.mil. NSEC3 IN> from 132.3.29.10
- [1402439035] unbound[17483:0] info: validation failure <fmyjfbjimgqpkmji.353sog.af.mil. A IN>: signature crypto failed for <NMA3M9UN511IPE7OP7G57907N729LKO5.af.mil. NSEC3 IN> from 132.3.25.10
- [1402439091] unbound[17483:0] info: validation failure <txbireqbrmftarlx.nationalmuseum.af.mil. A IN>: signature crypto failed for <EC1JEL6NPOR9V9NGUD6DOK3CHRNFHHBG.af.mil. NSEC3 IN> from 132.3.29.10