nsf.gov DNSSEC Outage: 2019-11-22 to 2019-11-23

Date: November 23, 2019

Overview

This page gives some details on the nsf.gov DNSSEC outage from November 22 to November 23, 2019. It was not the first DNSSEC outage for the National Science Foundation.

Timeline / DNSViz

(At the time of this writing, DNSViz historical archives have been down for months. DNSSEC makes its users think downtime doesn't matter.)

2019-11-22 11:37:09 UTC — first personally observed nsf.gov DNSSEC failure
2019-11-23 00:08:35 UTC — last personally observed nsf.gov DNSSEC failure

Here's a screenshot of DNSViz output:

November 22, 2019 nsf.gov DNSSEC outage in DNSViz

DNSSEC Debugger

Here's a screenshot of my web browser's output from November 3, 2019:

Please also see this archive.org copy

DNS-OARC: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

$ dig +dnssec a nsf.gov. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +dnssec a nsf.gov. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23940
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;nsf.gov. IN A

;; Query time: 87 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Thu Nov 21 11:37:13 2019
;; MSG SIZE rcvd: 36

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a nsf.gov. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +cd a nsf.gov. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17174
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;nsf.gov. IN A

;; ANSWER SECTION:
nsf.gov. 300 IN A 128.150.4.107

;; Query time: 19 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Thu Nov 21 11:37:13 2019
;; MSG SIZE rcvd: 41

Zonemaster

zonemaster.labs.nic.cz archived this DNSSEC outage, noting "No DS record had a DNSKEY with a matching keytag."
zonemaster.iis.se archived "Det finns ingen DS-post som har en DNSKEY-post med matchande keytag." (here's an archive.org copy of the same)
zonemaster.fr archived "No DS record had a DNSKEY with a matching keytag."

Logfile examples

These unbound error logs are from two different servers in different geographical regions.

[1574336229] unbound[77132:0] info: validation failure <nsf.gov. A IN>: no keys have a DS with algorithm RSASHA1 from 128.150.2.56 for key nsf.gov. while building chain of trust
[1574336335] unbound[230:0] info: validation failure <nsf.gov. A IN>: no keys have a DS with algorithm RSASHA1 from 204.14.134.242 for key nsf.gov. while building chain of trust
[1574337057] unbound[77132:0] info: validation failure <www.nsf.gov. A IN>: no keys have a DS with algorithm RSASHA1 from 128.150.2.55 for key nsf.gov. while building chain of trust
[1574381183] unbound[77132:0] info: validation failure <nsf.gov. A IN>: no keys have a DS with algorithm RSASHA1 from 204.14.134.242 for key nsf.gov. while building chain of trust
[1574381315] unbound[230:0] info: validation failure <www.nsf.gov. A IN>: no keys have a DS with algorithm RSASHA1 from 128.150.2.55 for key nsf.gov. while building chain of trust