nasa.gov DNSSEC Outage: 2015-10-02

Updated: October 3, 2015

Overview

This page gives some details on the nasa.gov partial DNSSEC outage on October 2, 2015. It may have been limited to the Unbound resolver.

Timeline / DNSViz

DNS-OARC: BIND vs. Unbound

DNS-OARC provides open DNSSEC resolvers. Here's a comparison between Unbound (which fails) and BIND (which does not).

First, BIND:

$ dig +dnssec nasa.gov @149.20.64.20

; <<>> DiG 9.4.2-P2 <<>> +dnssec nasa.gov @149.20.64.20
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59651
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;nasa.gov. IN A

;; ANSWER SECTION:
nasa.gov. 335 IN A 52.0.14.116
nasa.gov. 335 IN RRSIG A 5 2 600 20151011205503 20150911195503 4476 nasa.gov. bRh2h/DaP+E1UXRa21U5AtfgOvDiXvogtZoYKUA7hUNZLVSohOGAPn1J /dxxFGjkhFBs6P06C8kobKtDYyPLD8kom5IeohdEo56PoQBOdbxFGtVU ZGZrd5LO/zJ4tzH9QApB0Ce/PtRXp+lSzTOJqg06sJw7wooGRZlLGbP1 FlQ=
nasa.gov. 335 IN RRSIG A 5 2 600 20151011205503 20150911195503 8703 nasa.gov. KGoQqAL2SiClfj37UbmI9HFQctvG4t3Us3zfNk9AXqqGz0bkP/BXr58Y DAYlVVhuzMcs8pQid7GT1fTFuVSwg7Ejtq19OKfN2guW4SPH5d0IFulx OzrjChg5YzK+CmikfSUbq4PoKh9inITaK/iJ1nxXfR+DZVmNaXsvInbm Bjg=
nasa.gov. 335 IN RRSIG A 5 2 600 20151011205503 20150911195503 12315 nasa.gov. U5I6vzzqfWevD8wxRfd/iebibvq8yRSYwxJ3cAYCo51gWi7hMJwWACBb arviBZdUh6NvIXzCHl9hI692L4bZbNnkQKJ6TL36cR+uH9fIumMOheVl xXEVDcY1XU29ebt3nVv9KrmfAWxnmYiZQbQIVm70azto4sIqB9oD0s+e RVE=

;; Query time: 795 msec
;; SERVER: 149.20.64.20#53(149.20.64.20)
;; WHEN: Fri Oct 2 16:46:53 2015
;; MSG SIZE rcvd: 557

Now, Unbound:

$ dig +dnssec nasa.gov @149.20.64.21

; <<>> DiG 9.4.2-P2 <<>> +dnssec nasa.gov @149.20.64.21
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 492
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;nasa.gov. IN A

;; Query time: 1357 msec
;; SERVER: 149.20.64.21#53(149.20.64.21)
;; WHEN: Fri Oct 2 16:46:49 2015
;; MSG SIZE rcvd: 37

OpenDNS and Verisign Public DNS

OpenDNS does not support DNSSEC, instead supporting DNSCurve. Verisign Public DNS currently supports only DNSSEC, and thus, Verisign's users saw SERVFAIL for queries under nasa.gov during this outage.

With OpenDNS, queries succeed:

$ dig www.nasa.gov. @resolver1.opendns.com.

; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6890
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.nasa.gov. IN A

;; ANSWER SECTION:
www.nasa.gov. 7 IN CNAME www.nasawestprime.com.
www.nasawestprime.com. 30 IN CNAME iznasa.hs.llnwd.net.
iznasa.hs.llnwd.net. 116 IN A 208.111.171.236

;; Query time: 88 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Oct 2 22:55:45 2015
;; MSG SIZE rcvd: 114

With Verisign Public DNS, with DNSSEC, queries fail:

$ dig www.nasa.gov. @64.6.64.6

; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @64.6.64.6
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4790
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.nasa.gov. IN A

;; Query time: 288 msec
;; SERVER: 64.6.64.6#53(64.6.64.6)
;; WHEN: Fri Oct 2 22:56:25 2015
;; MSG SIZE rcvd: 30

Verisign's other public resolver also fails:

$ dig www.nasa.gov. @64.6.65.6

; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @64.6.65.6
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20715
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.nasa.gov. IN A

;; Query time: 319 msec
;; SERVER: 64.6.65.6#53(64.6.65.6)
;; WHEN: Fri Oct 2 22:56:53 2015
;; MSG SIZE rcvd: 30

Twitter

@charlesgruener on Twitter reported DNSSEC problems here: "@nasa I think DNSSEC for nasa.gov has problems dnsviz.net/d/nasa.gov/Vg5Gvw/dnssec/"

@hadmonitor on Twitter reported the DNSSEC outage but got the cause wrong, suggesting a "Non-DNSSEC" error: "ERROR: NASA.GOV appears #DNSSEC BOGUS. Possible cause: Some Non-DNSSEC error occurred"

HAD Monitor has the right idea but gets the details wrong.

Logfile examples