nasa.gov DNSSEC Outage: 2015-08-14 to 2015-08-15
Updated: August 15, 2015
Overview
This page gives some details on the nasa.gov DNSSEC outage from August 14 to 15, 2015. It lasted over 4 hours. From the variety of DNSSEC failure types in logs, and the multiple different kinds of failures shown in DNSViz and the DNSSEC Debugger, it appears the NASA DNS administrators solved the DNSSEC outage by trying random things until one of them finally worked.
Verisign's DNSSEC Debugger
Verisign doesn't archive test results, unlike DNSViz. So here's a screenshot I took on August 14, 2015:
Timeline / DNSViz
- 2015-08-14 20:47:32 UTC: first personally observed nasa.gov DNSSEC failure
- 2015-08-14 21:22:33 UTC: bogus DNSSEC delegation
- 2015-08-14 22:23:22 UTC: different bogus DNSSEC delegation
- 2015-08-14 22:45:24 UTC: a third type of bogus DNSSEC delegation
- 2015-08-14 22:50:09 UTC: same (above) bogus DNSSEC delegation
- 2015-08-14 23:01:44 UTC: same (above) bogus DNSSEC delegation
- 2015-08-14 23:11:24 UTC: same (above) bogus DNSSEC delegation
- 2015-08-14 23:15:41 UTC: same, but with 8 bogus SOA RRSIGs
- 2015-08-14 23:17:32 UTC: same, but with extra bogus SOA RRSIGs removed
- 2015-08-14 23:18:29 UTC: same (above) bogus DNSSEC delegation
- 2015-08-14 23:43:27 UTC: original bogus DNSSEC delegation type
- 2015-08-14 23:56:10 UTC: back to the third type of bogus DNSSEC delegation
- 2015-08-15 00:06:21 UTC: original bogus DNSSEC delegation again — really
- 2015-08-15 00:33:12 UTC: fourth type of bogus DNSSEC delegation
- 2015-08-15 00:48:31 UTC: same as above
- 2015-08-15 01:00:08 UTC: DNSSEC outage over
OpenDNS & Google Public DNS
OpenDNS does not support DNSSEC, instead supporting DNSCurve. Google Public DNS currently supports only DNSSEC, and thus, Google's users saw SERVFAIL for queries under nasa.gov during this outage.
With OpenDNS, queries succeed:
$ dig www.nasa.gov. @resolver1.opendns.com.
; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53946
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.nasa.gov. IN A
;; ANSWER SECTION:
www.nasa.gov. 489 IN CNAME www.nasawestprime.com.
www.nasawestprime.com. 11 IN CNAME iznasa.hs.llnwd.net.
iznasa.hs.llnwd.net. 144 IN A 208.111.171.236
;; Query time: 36 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Aug 14 21:45:57 2015
;; MSG SIZE rcvd: 114
With Google Public DNS, with DNSSEC, queries fail:
$ dig www.nasa.gov. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63576
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.nasa.gov. IN A
;; Query time: 127 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Aug 14 21:46:23 2015
;; MSG SIZE rcvd: 30
dnscheck
dnscheck.labs.nic.cz archived a DNSSEC outage at 2015-08-14 21:24:03 (requires javascript).
dnscheck.iis.se archived a DNSSEC outage at 2015-08-14 21:23:18 (requires javascript).
Zonemaster
Zonemaster archived this nasa.gov DNSSEC outage here.