.my TLD DNSSEC Outage: 2018-06-15 to 2018-06-16

Updated: June 18, 2018

Overview

This page gives some details on the my (Malaysia) TLD DNSSEC outage from June 15 to June 16, 2018. The outage began with a DNSSEC delegation to an orphaned, revoked DNSKEY and a ghost DNSKEY. Half a day later, at around 2018-06-15 20:14:01 UTC, all RRSIGs were removed from the .my server at 194.0.1.30 (check logfile examples at the bottom) and others followed. However, the DS from . to my remained, thus maintaining the DNSSEC outage. Then around 2018-06-16 01:44:11 UTC, a pile of bogus RRSIGs were added to the zone (because why not).

Check the DNSViz links and the DNSSEC Debugger screenshots for visual descriptions of the issues.

Timeline / DNSViz

Verisign's DNSSEC Debugger

Here's a screenshot I took on June 15, 2018, of the DNSSEC Debugger output:

June 15, 2018 .my TLD DNSSEC outage

Later, the .my administrators removed all RRSIGs (without disabling DNSSEC), making the DNSSEC Debugger output look like this:

June 15, 2018 .my TLD DNSSEC outage

Later still, the .my administrators added a bunch of bogus RRSIGs to the domain and let them marinate for about 4 hours.

June 16, 2018 .my TLD DNSSEC outage

Google Public DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec google.com.my. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec google.com.my. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 406
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;google.com.my. IN A

;; Query time: 345 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jun 15 14:36:26 2018
;; MSG SIZE rcvd: 42

You have to disable DNSSEC to make DNS queries work:

$ dig +cd google.com.my. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd google.com.my. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17823
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.my. IN A

;; ANSWER SECTION:
google.com.my. 299 IN A 172.217.3.195

;; Query time: 10 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jun 15 14:36:26 2018
;; MSG SIZE rcvd: 47

Cloudflare acknowledgment

This outage was discussed on Twitter by a Cloudflare employee:

cloudflare acknowledges .my TLD DNSSEC outage

MYNIC

MYNIC acknowledged this DNSSEC outage on its website (requires javascript).

It also acknowledged the outage via the MYNIC Twitter account:

MYNIC acknowledges .my TLD DNSSEC outage

Other News Sources

thestar.com.my reported that "All Malaysian websites whose addresses end with the ".my" domain, including The Star Online, will be experiencing difficulties after the domain registrar was hit by a technical glitch."

lowyat.net reports that "[Update] 4.50am – MYNIC confirms via a tweet that they are experiencing technical issues with their DNSSEC chain."

malaysiakini.com reports ".my domain outage, report urges caution when banking online"

freemalaysiatoday.com writes "The ".my" address registrar, MyNIC Berhad, said the problem lay in "some Technical Issue related to DNSSEC chain with IANA", a reference to a chain of security certificates used by domain name servers linked to IANA, the internet's governing name authority."

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: my.
;; Signature ok but no chain to a trusted key or ds record
[S] my. 172800 IN DNSKEY 257 3 8 ;{id = 63366 (ksk), size = 2048b}
my. 172800 IN DNSKEY 385 3 8 ;{id = 26120, size = 2048b}
my. 172800 IN DNSKEY 257 3 8 ;{id = 174 (ksk), size = 2048b}
my. 172800 IN DNSKEY 256 3 8 ;{id = 44156 (zsk), size = 1024b}
my. 172800 IN DNSKEY 256 3 8 ;{id = 48851 (zsk), size = 1024b}
[S] Existence denied: my. A
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples