.ss TLD DNSSEC Outage: 2020-10-01
Date: October 1, 2020
Overview
This page gives some details on the .ss (South Sudan) TLD DNSSEC outage on October 1, 2020.
Timeline / DNSViz
- 2020-10-01 18:47:59 UTC — first personally observed ss DNSSEC failure
- 2020-10-01 18:48:14 UTC — DNSSEC spinning out of control
- 2020-10-01 19:00:38 UTC — Bogus DNSSEC
- 2020-10-01 19:21:29 UTC — Bogus DNSSEC
- 2020-10-01 19:36:41 UTC — Bogus DNSSEC
- 2020-10-01 19:39:53 UTC — Bogus DNSSEC
- 2020-10-01 19:43:49 UTC — Bogus DNSSEC
- 2020-10-01 19:47:27 UTC — Bogus DNSSEC
- 2020-10-01 19:56:50 UTC — Bogus DNSSEC
- 2020-10-01 20:15:49 UTC — Bogus DNSSEC
- 2020-10-01 20:19:11 UTC — last personally observed ss DNSSEC failure
Here's a screenshot of DNSViz output showing what the outage looked like:
Since DNSViz loses data regularly, there is an archive at archive.is. There is also a copy courtesy of archive.org.
DNSSEC Debugger
Here's a screenshot of my web browser's output from October 1, 2020.
There is a copy of DNSSEC Debugger output saved by archive.is.
Google DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.
$ dig +dnssec ns ss. @8.8.8.8
; <<>> dig 9.10.8-P1 <<>> +dnssec ns ss. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20896
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;ss. IN NS
;; Query time: 921 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Oct 01 18:48:01 UTC 2020
;; MSG SIZE rcvd: 31
You have to disable DNSSEC to make DNS queries work:
$ dig +cd ns ss. @8.8.8.8
; <<>> dig 9.10.8-P1 <<>> +cd ns ss. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25474
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;ss. IN NS
;; ANSWER SECTION:
ss. 21599 IN NS ssnic.anycastdns.cz.
ss. 21599 IN NS root.nic.ss.
ss. 21599 IN NS ns-ss.afrinic.net.
ss. 21599 IN NS b.ns.tznic.or.tz.
ss. 21599 IN NS pch.nic.ss.
;; Query time: 17 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Oct 01 18:48:01 UTC 2020
;; MSG SIZE rcvd: 166
Zonemaster
Note: Zonemaster requires javascript to display text.
- zonemaster.net noted the outage (archive.is copy)
- zonemaster.iis.se also noted the outage (archive.is copy)
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
ss. 86400 IN DS 43817 8 2 f85d36a2e056ff8cbb8dc79c748e4ec1e862c4f739891f9595669aa3384a252d
;; Domain: ss.
;; Signature ok but no chain to a trusted key or ds record
[S] ss. 3600 IN DNSKEY 257 3 8 ;{id = 47197 (ksk), size = 2048b}
ss. 3600 IN DNSKEY 256 3 8 ;{id = 36876 (zsk), size = 1024b}
[S] Existence denied: ss. A
;;[S] self sig OK; [B] bogus; [T] trusted
dns.google.com
dns.google.com is related to but separate from Google Public DNS. During this DNSSEC outage, dns.google.com showed the following for ss:
There's also a copy thanks to archive.is
Logfile examples
These Unbound log entries come from different Unbound instances, each on different servers in different geographical regions.
- [1601578079] unbound[69311:0] info: validation failure <ss. NS IN>: no keys have a DS with algorithm RSASHA256 from 185.28.194.194 for key ss. while building chain of trust
- [1601578277] unbound[69311:0] info: validation failure <ss. NS IN>: no keys have a DS with algorithm RSASHA256 from 204.61.216.130 for key ss. while building chain of trust
- [1601583362] unbound[8912:0] info: validation failure <ss. NS IN>: no keys have a DS with algorithm RSASHA256 from 204.61.216.130 for key ss. while building chain of trust
- [1601583525] unbound[8912:0] info: validation failure <ss. NS IN>: no keys have a DS with algorithm RSASHA256 from 185.28.194.194 for key ss. while building chain of trust
- [1601583551] unbound[69311:0] info: validation failure <ss. NS IN>: no keys have a DS with algorithm RSASHA256 from 204.61.216.130 for key ss. while building chain of trust