dnssec-validator.cz DNSSEC Outage: 2020-04-18 to 2020-05-11
Updated date: June 12, 2020
Overview
This page gives some details on the long dnssec-validator.cz DNSSEC outage that began on April 18, 2020. The DNSSEC outage was caused by bogus NSEC3 records. According to my unbound logs, it began on April 18 and ended on May 11.
Timeline
- 2020-04-18 12:55:08 UTC — first personally observed DNSSEC failure
- 2020-05-11 12:51:29 UTC — last personally observed DNSSEC failure
Google DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNS SEC.
With Google Public DNS (8.8.8.8), because of DNSSEC, queries fail:
$ dig +dnssec a dnssec-validator.cz. @8.8.8.8
; <<>> DiG 9.10.3-P4-Debian <<>> +dnssec a dnssec-validator.cz. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 25916
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;dnssec-validator.cz. IN A
;; Query time: 265 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Apr 18 13:21:29 UTC 2020
;; MSG SIZE rcvd: 48
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a dnssec-validator.cz. @8.8.8.8
; <<>> DiG 9.10.3-P4-Debian <<>> +cd a dnssec-validator.cz. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45870
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;dnssec-validator.cz. IN A
;; ANSWER SECTION:
dnssec-validator.cz. 1799 IN A 217.31.192.130
;; Query time: 101 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Apr 18 13:21:29 UTC 2020
;; MSG SIZE rcvd: 64
dns.google.com
The website dns.google.com is not the public DNS service known as 8.8.8.8. dns.google.com is another DNSSEC-related analysis tool. During this DNSSEC outage, dns.google.com showed a DNSSEC failure for dnssec-validator.cz:
In addition, archive.is was kind enough to provide an archive of this output.
Logfile examples
Note that the bogus NSEC3 records are given by multiple servers over a long period of time. Here's a complete log excerpt.
- [1587214508] unbound[50740:0] info: validation failure <dnssec-validator.cz. A IN>: covering NSEC3 was not opt-out in an opt-out DS NOERROR/NODATA case from 194.0.13.1 for DS dnssec-validator.cz. while building chain of trust
- [1587214865] unbound[50740:0] info: validation failure <www.dnssec-validator.cz. A IN>: covering NSEC3 was not opt-out in an opt-out DS NOERROR/NODATA case from 194.0.13.1 for DS dnssec-validator.cz. while building chain of trust
- [1587215100] unbound[50740:0] info: validation failure <dnssec-validator.cz. A IN>: covering NSEC3 was not opt-out in an opt-out DS NOERROR/NODATA case from 194.0.14.1 for DS dnssec-validator.cz. while building chain of trust
- [1587215453] unbound[50740:0] info: validation failure <www.dnssec-validator.cz. A IN>: covering NSEC3 was not opt-out in an opt-out DS NOERROR/NODATA case from 193.29.206.1 for DS dnssec-validator.cz. while building chain of trust
- [1589201489] unbound[50740:0] info: validation failure <www.dnssec-validator.cz. A IN>: covering NSEC3 was not opt-out in an opt-out DS NOERROR/NODATA case from 194.0.12.1 for DS dnssec-validator.cz. while building chain of trust