co.bw DNSSEC Outage: 2018-07-08

Date: July 8, 2018

Overview

This page gives some details on the co.bw DNSSEC outage on July 8, 2018.

Timeline / DNSViz

DNS-OARC: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec ns co.bw. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +dnssec ns co.bw. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 2707
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;co.bw. IN NS

;; Query time: 2870 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Sun Jul 8 16:17:15 2018
;; MSG SIZE rcvd: 34

You have to disable DNSSEC to make DNS queries work:

$ dig +cd ns co.bw. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +cd ns co.bw. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41035
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;co.bw. IN NS

;; ANSWER SECTION:
co.bw. 3598 IN NS pch.nic.net.bw.
co.bw. 3598 IN NS master.btc.net.bw.
co.bw. 3598 IN NS dns2.nic.net.bw.
co.bw. 3598 IN NS ns-bw.afrinic.net.
co.bw. 3598 IN NS dns1.nic.net.bw.

;; Query time: 51 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Sun Jul 8 16:17:15 2018
;; MSG SIZE rcvd: 143

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: bw.
[T] bw. 3600 IN DNSKEY 257 3 8 ;{id = 18880 (ksk), size = 2048b}
bw. 3600 IN DNSKEY 256 3 8 ;{id = 38424 (zsk), size = 1024b}
bw. 3600 IN DNSKEY 256 3 8 ;{id = 46158 (zsk), size = 1024b}
;; No DS for co.bw.;; No ds record for delegation
;; Domain: co.bw.
;; No DNSKEY record found for co.bw.
[U] No data found for: co.bw. type A
;;[S] self sig OK; [B] bogus; [T] trusted

To make it clear, since DNSSEC is confusing and bizarre, the above shows a bogus DNSSEC delegation. A proper delegation from a DNSSEC parent to a non-DNSSEC child would include a line such as:

[T] Existence denied: google.com. DS

But bw broke the delegation to co.bw, causing this DNSSEC outage. Also it looks like there's a minor bug in the drill output (a missing newline) in the line beginning ";; No DS for co.bw." Heh.

Logfile examples