nist.gov DNSSEC Outage: 2016-09-12

Updated: September 12, 2016

Overview

This page gives some details on the nist.gov DNSSEC outage on September 12, 2016. It was not the first DNSSEC outage in nist.gov, and not even the first nist.gov DNSSEC outage in the last 30 days. This was a complete DNSSEC outage affecting all names under nist.gov, including the websites, NTP service (time.nist.gov), and all other nist.gov Internet services requiring functioning DNS service.

Timeline / DNSViz

2016-09-12 04:48:23 UTC — Bogus RRSIGs
2016-09-12 06:11:57 UTC — Bogus RRSIGs
2016-09-12 10:24:17 UTC — Bogus RRSIGs
2016-09-12 14:46:54 UTC — Bogus RRSIGs
2016-09-12 15:20:03 UTC — Outage debris, but DNSSEC outage over

OpenDNS & Google Public DNS

OpenDNS does not support DNSSEC, instead supporting DNSCurve. Google Public DNS currently supports only DNSSEC, and thus, Google's users saw SERVFAIL for queries under www.nist.gov during this outage.

With OpenDNS, which doesn't support DNSSEC, queries succeed:

$ dig time.nist.gov. @resolver1.opendns.com.

; <<>> DiG 9.4.2-P2 <<>> time.nist.gov. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59716
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;time.nist.gov. IN A

;; ANSWER SECTION:
time.nist.gov. 611 IN CNAME ntp1.glb.nist.gov.
ntp1.glb.nist.gov. 29 IN A 216.229.0.179

;; Query time: 5 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Mon Sep 12 15:01:14 2016
;; MSG SIZE rcvd: 70

With Google Public DNS, because of DNSSEC, queries fail:

$ dig +dnssec time.nist.gov. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec time.nist.gov. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 40427
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;time.nist.gov. IN A

;; Query time: 29 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Sep 12 15:01:14 2016
;; MSG SIZE rcvd: 42

dnscheck

dnscheck.labs.nic.cz reports RSA verification failures.
dnscheck.iis.se reports RSA verification failures, and also claims one of the nameservers wasn't responding. I wonder if the DNSSEC outage caused a spike in DNS queries that overloaded the server?

Zonemaster

zonemaster.net archived this nist.gov DNSSEC outage, noting "Delegation from parent to child is not properly signed (signature: Bogus DNSSEC signature; signature: Bogus DNSSEC signature)."
zonemaster.fr archived this nist.gov DNSSEC outage, noting "Delegation from parent to child is not properly signed (signature: Bogus DNSSEC signature; signature: Bogus DNSSEC signature)."

Logfile examples

[1473659608] unbound[4014:0] info: validation failure <time.nist.gov. A IN>: signature crypto failed from 129.6.13.3 for key nist.gov. while building chain of trust
[1473693262] unbound[4014:0] info: validation failure <nist.gov. A IN>: signature crypto failed from 132.163.4.10 for key nist.gov. while building chain of trust
[1473660622] unbound[4014:0] info: validation failure <www.nist.gov. A IN>: signature crypto failed from 132.163.4.10 for key nist.gov. while building chain of trust