nasa.gov DNSSEC Outage: 2015-10-02
Updated: October 3, 2015
Overview
This page gives some details on the nasa.gov partial DNSSEC outage on October 2, 2015. It may have been limited to the Unbound resolver.
Timeline / DNSViz
- 2015-10-02 06:14:45 UTC: first personally observed nasa.gov DNSSEC failure
- 2015-10-02 08:24:45 UTC: numerous errors
- 2015-10-02 08:56:31 UTC: numerous errors
- 2015-10-02 13:02:19 UTC: still numerious errors...
- 2015-10-02 13:49:21 UTC: numerous errors
- 2015-10-02 14:11:33 UTC: numerous errors
- 2015-10-02 16:56:09 UTC: numerous errors
- 2015-10-02 17:40:58 UTC: numerous errors
- 2015-10-02 18:17:19 UTC: numerous errors
- 2015-10-02 22:07:27 UTC: numerous errors
- 2015-10-03 00:00:31 UTC: last personally observed nasa.gov DNSSEC failure
- 2015-10-03 02:26:48 UTC: nasa.gov partial DNSSEC outage over
DNS-OARC: BIND vs. Unbound
DNS-OARC provides open DNSSEC resolvers. Here's a comparison between Unbound (which fails) and BIND (which does not).
First, BIND:
$ dig +dnssec nasa.gov @149.20.64.20
; <<>> DiG 9.4.2-P2 <<>> +dnssec nasa.gov @149.20.64.20
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59651
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;nasa.gov. IN A
;; ANSWER SECTION:
nasa.gov. 335 IN A 52.0.14.116
nasa.gov. 335 IN RRSIG A 5 2 600 20151011205503 20150911195503 4476 nasa.gov. bRh2h/DaP+E1UXRa21U5AtfgOvDiXvogtZoYKUA7hUNZLVSohOGAPn1J /dxxFGjkhFBs6P06C8kobKtDYyPLD8kom5IeohdEo56PoQBOdbxFGtVU ZGZrd5LO/zJ4tzH9QApB0Ce/PtRXp+lSzTOJqg06sJw7wooGRZlLGbP1 FlQ=
nasa.gov. 335 IN RRSIG A 5 2 600 20151011205503 20150911195503 8703 nasa.gov. KGoQqAL2SiClfj37UbmI9HFQctvG4t3Us3zfNk9AXqqGz0bkP/BXr58Y DAYlVVhuzMcs8pQid7GT1fTFuVSwg7Ejtq19OKfN2guW4SPH5d0IFulx OzrjChg5YzK+CmikfSUbq4PoKh9inITaK/iJ1nxXfR+DZVmNaXsvInbm Bjg=
nasa.gov. 335 IN RRSIG A 5 2 600 20151011205503 20150911195503 12315 nasa.gov. U5I6vzzqfWevD8wxRfd/iebibvq8yRSYwxJ3cAYCo51gWi7hMJwWACBb arviBZdUh6NvIXzCHl9hI692L4bZbNnkQKJ6TL36cR+uH9fIumMOheVl xXEVDcY1XU29ebt3nVv9KrmfAWxnmYiZQbQIVm70azto4sIqB9oD0s+e RVE=
;; Query time: 795 msec
;; SERVER: 149.20.64.20#53(149.20.64.20)
;; WHEN: Fri Oct 2 16:46:53 2015
;; MSG SIZE rcvd: 557
Now, Unbound:
$ dig +dnssec nasa.gov @149.20.64.21
; <<>> DiG 9.4.2-P2 <<>> +dnssec nasa.gov @149.20.64.21
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 492
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;nasa.gov. IN A
;; Query time: 1357 msec
;; SERVER: 149.20.64.21#53(149.20.64.21)
;; WHEN: Fri Oct 2 16:46:49 2015
;; MSG SIZE rcvd: 37
OpenDNS and Verisign Public DNS
OpenDNS does not support DNSSEC, instead supporting DNSCurve. Verisign Public DNS currently supports only DNSSEC, and thus, Verisign's users saw SERVFAIL for queries under nasa.gov during this outage.
With OpenDNS, queries succeed:
$ dig www.nasa.gov. @resolver1.opendns.com.
; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6890
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.nasa.gov. IN A
;; ANSWER SECTION:
www.nasa.gov. 7 IN CNAME www.nasawestprime.com.
www.nasawestprime.com. 30 IN CNAME iznasa.hs.llnwd.net.
iznasa.hs.llnwd.net. 116 IN A 208.111.171.236
;; Query time: 88 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Oct 2 22:55:45 2015
;; MSG SIZE rcvd: 114
With Verisign Public DNS, with DNSSEC, queries fail:
$ dig www.nasa.gov. @64.6.64.6
; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @64.6.64.6
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4790
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.nasa.gov. IN A
;; Query time: 288 msec
;; SERVER: 64.6.64.6#53(64.6.64.6)
;; WHEN: Fri Oct 2 22:56:25 2015
;; MSG SIZE rcvd: 30
Verisign's other public resolver also fails:
$ dig www.nasa.gov. @64.6.65.6
; <<>> DiG 9.4.2-P2 <<>> www.nasa.gov. @64.6.65.6
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20715
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.nasa.gov. IN A
;; Query time: 319 msec
;; SERVER: 64.6.65.6#53(64.6.65.6)
;; WHEN: Fri Oct 2 22:56:53 2015
;; MSG SIZE rcvd: 30
@charlesgruener on Twitter reported DNSSEC problems here: "@nasa I think DNSSEC for nasa.gov has problems dnsviz.net/d/nasa.gov/Vg5Gvw/dnssec/"
@hadmonitor on Twitter reported the DNSSEC outage but got the cause wrong, suggesting a "Non-DNSSEC" error: "ERROR: NASA.GOV appears #DNSSEC BOGUS. Possible cause: Some Non-DNSSEC error occurred"
HAD Monitor has the right idea but gets the details wrong.
Logfile examples
- [1443766485] unbound[10165:0] info: validation failure <nasa.gov. A IN>: no keys have a DS with algorithm RSASHA256 from 198.116.4.185 for key nasa.gov. while building chain of trust
- [1443767419] unbound[10165:0] info: validation failure <nasa.gov. A IN>: no keys have a DS with algorithm RSASHA256 from 198.116.4.181 for key nasa.gov. while building chain of trust
- [1443805903] unbound[10165:0] info: validation failure <nasa.gov. A IN>: no keys have a DS with algorithm RSASHA256 from 198.116.4.189 for key nasa.gov. while building chain of trust