dnssek.info DNSSEC Outage: 2018-09-17 to 2018-10-06

Updated: October 11, 2018

Overview

This page gives some details on the dnssek.info DNSSEC outage from September 17 to October 6, 2018. dnssek.info provides what it calls the "DNSSEC EARLY WARNING SYSTEM (DEWS)" for expiring and expired TLD RRSIGs...

Timeline / DNSViz

2018-09-17 09:30:44 UTC — RRSIGs expire
2018-09-17 09:32:48 UTC — expired RRSIGs
2018-09-17 09:40:35 UTC — expired RRSIGs
2018-09-17 09:44:55 UTC — expired RRSIGs
2018-09-17 09:47:15 UTC — expired RRSIGs
2018-09-17 15:51:52 UTC — expired RRSIGs
2018-09-18 03:07:42 UTC — expired RRSIGs
2018-09-18 15:08:11 UTC — expired RRSIGs
2018-09-19 01:25:37 UTC — expired RRSIGs
2018-09-20 02:33:33 UTC — expired RRSIGs
2018-09-21 02:04:40 UTC — expired RRSIGs
2018-10-05 05:21:18 UTC — expired RRSIGs
2018-10-06 01:34:16 UTC — last personally observed DNSSEC failure

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from September 17, 2018:

September 17, 2018 dnssek.info DNSSEC outage

Zonemaster

zonemaster.iis.se archived "The apex DNSKEY RRset was not correctly signed" and other DNSSEC problems.
zonemaster.labs.nic.cz archived "The apex DNSKEY RRset was not correctly signed" and other DNSSEC problems.

Google Public DNS

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

With Google Public DNS, because of DNSSEC, queries fail:

$ dig +dnssec a www.dnssek.info. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.dnssek.info. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 35395
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.dnssek.info. IN A

;; Query time: 216 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Sep 17 09:34:53 2018
;; MSG SIZE rcvd: 44

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a www.dnssek.info. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd a www.dnssek.info. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49528
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.dnssek.info. IN A

;; ANSWER SECTION:
www.dnssek.info. 3378 IN A 192.101.186.107

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Sep 17 09:34:53 2018
;; MSG SIZE rcvd: 49

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

;; Domain: dnssek.info.
[B] dnssek.info. 3600 IN DNSKEY 257 3 8 ;{id = 532 (ksk), size = 2048b}
dnssek.info. 3600 IN DNSKEY 256 3 8 ;{id = 38286 (zsk), size = 2048b}
[B] Error verifying denial of existence for dnssek.info. type A: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples

[1537176892] unbound[11465:0] info: validation failure <www.dnssek.info. A IN>: signature expired from 172.105.242.234 for key dnssek.info. while building chain of trust
[1537177212] unbound[11465:0] info: validation failure <dnssek.info. A IN>: signature expired from 45.56.97.49 for key dnssek.info. while building chain of trust
[1537199241] unbound[11465:0] info: validation failure <dnssek.info. MX IN>: signature expired from 45.56.97.49 for key dnssek.info. while building chain of trust
[1538789656] unbound[927:0] info: validation failure <www.dnssek.info. A IN>: signature expired from 45.56.97.49 for key dnssek.info. while building chain of trust