nih.gov DNSSEC Outage: 2021-04-22
Date: April 22, 2021
Overview
This page gives some details on the US National Institutes of Health (nih.gov) DNSSEC outage on April 22, 2021.
Timeline / DNSViz
- 2021-04-22 16:29:38 UTC — Bogus DNSSEC delegation
- 2021-04-22 17:33:12 UTC — Bogus DNSSEC delegation
- 2021-04-22 18:27:48 UTC — Bogus DNSSEC delegation
- 2021-04-22 18:48:58 UTC — Bogus DNSSEC delegation
- 2021-04-22 19:07:18 UTC — Bogus DNSSEC delegation
- 2021-04-22 19:37:04 UTC — Bogus DNSSEC delegation
- 2021-04-22 19:55:01 UTC — Bogus DNSSEC delegation
- 2021-04-22 20:01:40 UTC — Bogus DNSSEC delegation
- 2021-04-22 20:04:22 UTC — Bogus DNSSEC delegation
- 2021-04-22 20:10:34 UTC — Bogus DNSSEC delegation
- 2021-04-22 20:15:29 UTC — DNSSEC outage over
Since DNSViz has lots its archives multiple times, here are some 3rd party copies:
And here's a screenshot just in case:
DNSSEC Debugger
Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from April 22, 2021:
You can also see this snapshot thanks to archive.is.
Google DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.
$ dig +dnssec a nih.gov. @8.8.8.8.
; <<>> dig 9.10.8-P1 <<>> +dnssec a nih.gov. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 13359
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;nih.gov. IN A
;; Query time: 3308 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Apr 22 17:27:15 UTC 2021
;; MSG SIZE rcvd: 36
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a nih.gov. @8.8.8.8.
; <<>> dig 9.10.8-P1 <<>> +cd a nih.gov. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35389
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;nih.gov. IN A
;; ANSWER SECTION:
nih.gov. 3331 IN A 54.235.145.223
;; Query time: 7 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Apr 22 17:27:15 UTC 2021
;; MSG SIZE rcvd: 52
Zonemaster
- zonemaster.iis.se (archive.is copy)
- zonemaster.net (archive.is copy)
- zonemaster.labs.nic.cz (archive.is copy)
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
[T] nih.gov. 3600 IN DS 6318 7 2 6dc9d762020fdf991df60a72d2bddacdfe761f58a0e58cec8998450a5a565793
nih.gov. 3600 IN DS 35355 7 2 3d1ff2755fbacc8958c29fea7a71750848cc259ccd4fb392aa1ca372751c9c25
;; Domain: nih.gov.
;; Signature ok but no chain to a trusted key or ds record
[S] nih.gov. 172800 IN DNSKEY 257 3 7 ;{id = 58971 (ksk), size = 2048b}
nih.gov. 172800 IN DNSKEY 256 3 8 ;{id = 22698 (zsk), size = 1024b}
nih.gov. 172800 IN DNSKEY 256 3 7 ;{id = 2264 (zsk), size = 1024b}
nih.gov. 172800 IN DNSKEY 256 3 7 ;{id = 56039 (zsk), size = 1024b}
nih.gov. 172800 IN DNSKEY 256 3 7 ;{id = 35669 (zsk), size = 1024b}
nih.gov. 172800 IN DNSKEY 257 3 8 ;{id = 17224 (ksk), size = 2048b}
nih.gov. 172800 IN DNSKEY 257 3 8 ;{id = 35449 (ksk), size = 2048b}
[S] nih.gov. 7200 IN A 54.235.145.223
;;[S] self sig OK; [B] bogus; [T] trusted
dns.google.com
dns.google.com saw this outage — here's a screenshot:
A snapshot of the dns.google.com view was saved by archive.is.
Logfile examples
- [1619112367] unbound[85039:0] info: validation failure <nih.gov. AAAA IN>: no keys have a DS with algorithm RSASHA1-NSEC3-SHA1 from 165.112.4.230 for key nih.gov. while building chain of trust
- [1619112367] unbound[85039:0] info: validation failure <nih.gov. A IN>: no keys have a DS with algorithm RSASHA1-NSEC3-SHA1 from 165.112.4.230 for key nih.gov. while building chain of trust
- [1619112367] unbound[85039:0] info: validation failure <medlineplus.gov. A IN>: no keys have a DS with algorithm RSASHA1-NSEC3-SHA1 from 165.112.4.230 for key nih.gov. while building chain of trust
- [1619112367] unbound[85039:0] info: validation failure <medlineplus.gov. AAAA IN>: no keys have a DS with algorithm RSASHA1-NSEC3-SHA1 from 165.112.4.230 for key nih.gov. while building chain of trust
- [1619123044] unbound[85039:0] info: validation failure <nih.gov. A IN>: key for validation nih.gov. is marked as invalid because of a previous validation failure <medlineplus.gov. AAAA IN>: no keys have a DS with algorithm RSASHA1-NSEC3-SHA1 from 128.231.128.251 for key nih.gov. while building chain of trust