tm TLD DNSSEC Outage: 2022-09-28
Date: September 28, 2022
Overview
This page gives some details on the tm (Turkmenistan) TLD DNSSEC outage on September 28, 2022.
Verisign's DNSSEC Debugger
Verisign doesn't archive test results, unlike DNSViz. So here's a screenshot I took on September 28, 2022:
Missing DNSSEC records
The nameservers weren't supplying DNSSEC records in responses. Here's an example of an empty answer section DNSKEY query response.
Timeline / DNSViz
- 2022-09-28 13:50:42 UTC — No DNSSEC records
- 2022-09-28 13:52:27 UTC — No DNSSEC records
- 2022-09-28 13:56:53 UTC — No DNSSEC records
- 2022-09-28 14:01:33 UTC — No DNSSEC records
- 2022-09-28 14:02:51 UTC — No DNSSEC records
- 2022-09-28 14:31:55 UTC — last personally observed tm DNSSEC failure
Here's a screenshot example:
Google Public DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:
$ dig +dnssec ns tm. @8.8.8.8.
; <<>> dig 9.10.8-P1 <<>> +dnssec ns tm. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 9714
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;tm. IN NS
;; Query time: 70 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Sep 28 13:52:12 UTC 2022
;; MSG SIZE rcvd: 31
You have to disable DNSSEC to make DNS work:
$ dig +cd ns tm. @8.8.8.8.
; <<>> dig 9.10.8-P1 <<>> +cd ns tm. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58288
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;tm. IN NS
;; ANSWER SECTION:
tm. 21600 IN NS ns-a2.tm.
tm. 21600 IN NS ns-y1.tm.
tm. 21600 IN NS ns-a3.tm.
tm. 21600 IN NS ns-a4.tm.
tm. 21600 IN NS ns-l1.tm.
tm. 21600 IN NS ns-d1.tm.
tm. 21600 IN NS ns-a1.tm.
;; Query time: 78 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Sep 28 13:52:12 UTC 2022
;; MSG SIZE rcvd: 171
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):
[T] tm. 86400 IN DS 36787 8 2 af4a3a54aa5f73c114f9c6e4d9c3ef0040e387806a9f92ae85276ff38f33bcb5
;; Domain: tm.
;; No DNSKEY record found for tm.
[U] No data found for: tm. type A
;;[S] self sig OK; [B] bogus; [T] trusted; [U] unsigned
Logfile examples
These logfile examples come from 3 different unbound instances running on different operating systems in different geographical locations.
- [1664373127] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 74.116.178.22 for key tm. while building chain of trust
- [1664373319] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 64.251.31.180 for key tm. while building chain of trust
- [1664373381] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 194.0.2.22 for key tm. while building chain of trust
- [1664374323] unbound[424:0] info: validation failure <tm. NS IN>: No DNSKEY record from 74.116.178.22 for key tm. while building chain of trust
- [1664374904] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 64.251.31.180 for key tm. while building chain of trust
- [1664375034] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 91.208.95.22 for key tm. while building chain of trust
- [1664375220] unbound[46624:0] info: validation failure <tm. NS IN>: No DNSKEY record from 91.208.95.22 for key tm. while building chain of trust
- [1664375232] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 80.249.100.44 for key tm. while building chain of trust
- [1664375303] unbound[46624:0] info: validation failure <tm. NS IN>: No DNSKEY record from 74.116.178.22 for key tm. while building chain of trust
- [1664375341] unbound[424:0] info: validation failure <tm. NS IN>: No DNSKEY record from 194.0.1.22 for key tm. while building chain of trust
- [1664375363] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 194.0.2.22 for key tm. while building chain of trust
- [1664375379] unbound[46624:0] info: validation failure <tm. NS IN>: No DNSKEY record from 194.0.2.22 for key tm. while building chain of trust
- [1664375446] unbound[46624:0] info: validation failure <tm. NS IN>: No DNSKEY record from 194.0.1.22 for key tm. while building chain of trust
- [1664375478] unbound[77840:0] info: validation failure <tm. NS IN>: No DNSKEY record from 194.0.1.22 for key tm. while building chain of trust
- [1664375515] unbound[46624:0] info: validation failure <tm. NS IN>: No DNSKEY record from 80.249.100.44 for key tm. while building chain of trust