dnssec-validator.cz DNSSEC Outage: 2020-04-18 to 2020-05-11

Updated date: June 12, 2020

Overview

This page gives some details on the long dnssec-validator.cz DNSSEC outage that began on April 18, 2020. The DNSSEC outage was caused by bogus NSEC3 records. According to my unbound logs, it began on April 18 and ended on May 11.

Timeline

Google DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNS SEC.

With Google Public DNS (8.8.8.8), because of DNSSEC, queries fail:

$ dig +dnssec a dnssec-validator.cz. @8.8.8.8

; <<>> DiG 9.10.3-P4-Debian <<>> +dnssec a dnssec-validator.cz. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 25916
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;dnssec-validator.cz. IN A

;; Query time: 265 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Apr 18 13:21:29 UTC 2020
;; MSG SIZE rcvd: 48


You have to disable DNSSEC to make DNS queries work:

$ dig +cd a dnssec-validator.cz. @8.8.8.8

; <<>> DiG 9.10.3-P4-Debian <<>> +cd a dnssec-validator.cz. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45870
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;dnssec-validator.cz. IN A

;; ANSWER SECTION:
dnssec-validator.cz. 1799 IN A 217.31.192.130

;; Query time: 101 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Apr 18 13:21:29 UTC 2020
;; MSG SIZE rcvd: 64

dns.google.com

The website dns.google.com is not the public DNS service known as 8.8.8.8. dns.google.com is another DNSSEC-related analysis tool. During this DNSSEC outage, dns.google.com showed a DNSSEC failure for dnssec-validator.cz:

dns.google.com output showing DNSSEC outage for dnssec-validator.cz

In addition, archive.is was kind enough to provide an archive of this output.

Logfile examples

Note that the bogus NSEC3 records are given by multiple servers over a long period of time. Here's a complete log excerpt.