.xn--qxam TLD DNSSEC Outage: 2019-11-03

Date: November 3, 2019

Overview

This page gives some details on the .xn--qxam TLD DNSSEC outage on November 3, 2019.

Timeline / DNSViz

(At the time of this writing, DNSViz historical archives have been down for months. DNSSEC makes its users think downtime doesn't matter.)

2019-11-03 09:31:28 UTC — first personally observed xn--qxam DNSSEC failure
2019-11-03 10:03:27 UTC — last personally observed xn--qxam DNSSEC failure

DNSSEC Debugger

Here's a screenshot of my web browser's output from November 3, 2019:

November 3, 2019 .xn--qxam TLD DNSSEC outage

Google DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

$ dig +dnssec ns xn--qxam. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec ns xn--qxam. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 49082
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;xn--qxam. IN NS

;; Query time: 45 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Nov 3 09:31:30 2019
;; MSG SIZE rcvd: 37

You have to disable DNSSEC to make DNS queries work:

$ dig +cd ns xn--qxam. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd ns xn--qxam. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57545
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;xn--qxam. IN NS

;; ANSWER SECTION:
xn--qxam. 10799 IN NS gr-c.ics.forth.gr.
xn--qxam. 10799 IN NS estia.ics.forth.gr.
xn--qxam. 10799 IN NS gr-d.ics.forth.gr.
xn--qxam. 10799 IN NS gr-m.ics.forth.gr.
xn--qxam. 10799 IN NS gr-at.ics.forth.gr.
xn--qxam. 10799 IN NS grdns.ics.forth.gr.

;; Query time: 199 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Nov 3 09:31:30 2019
;; MSG SIZE rcvd: 155

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

[T] xn--qxam. 86400 IN DS 55983 7 1 ee1acba7831fee4d79f434131be29bd9962901d4
xn--qxam. 86400 IN DS 55983 7 2 3e135aa5a7c428e65386925a99da9404b0135eaa5f5a3fc4e747d75c99618e0c
;; Domain: xn--qxam.
;; No DNSKEY record found for xn--qxam.
[U] No data found for: xn--qxam. type A
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples

These Unbound log entries come from different Unbound instances, each on different servers in different geographical regions.

[1572773488] unbound[73855:0] info: validation failure <xn--qxam. NS IN>: no signatures from 139.91.1.1
[1572773919] unbound[73855:0] info: validation failure <xn--qxam. NS IN>: No DNSKEY record from 139.91.1.1 for key xn--qxam. while building chain of trust
[1572774119] unbound[73855:0] info: validation failure <xn--qxam. NS IN>: No DNSKEY record from 194.0.11.102 for key xn--qxam. while building chain of trust
[1572775374] unbound[185:0] info: validation failure : No DNSKEY record from 78.104.145.227 for key xn--qxam. while building chain of trust
[1572775407] unbound[73855:0] info: validation failure <xn--qxam. NS IN>: No DNSKEY record from 139.91.191.3 for key xn--qxam. while building chain of trust