census.gov DNSSEC Outage: 2018-11-27

Date: November 27, 2018

Overview

This page gives some details on the census.gov DNSSEC outage on November 27, 2018.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from November 27, 2018:

November 27, 2018 census.gov DNSSEC outage

DNS-OARC: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec a census.gov. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +dnssec a census.gov. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31911
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;census.gov. IN A

;; Query time: 1051 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Tue Nov 27 16:09:17 2018
;; MSG SIZE rcvd: 39

You have to disable DNSSEC to make DNS work:

$ dig +cd a census.gov. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +cd a census.gov. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14625
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 4

;; QUESTION SECTION:
;census.gov. IN A

;; ANSWER SECTION:
census.gov. 14389 IN A 148.129.75.218

;; AUTHORITY SECTION:
census.gov. 28774 IN NS ns1e.census.gov.
census.gov. 28774 IN NS ns2e.census.gov.

;; ADDITIONAL SECTION:
ns1e.census.gov. 28774 IN A 148.129.129.22
ns2e.census.gov. 28774 IN A 148.129.75.22
ns1e.census.gov. 28774 IN AAAA 2610:20:2000:101::16:0
ns2e.census.gov. 28774 IN AAAA 2610:20:2010:a04::16:0

;; Query time: 20 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Tue Nov 27 16:09:17 2018
;; MSG SIZE rcvd: 170

Zonemaster

Note: zonemaster requires javascript.

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: census.gov.
;; Signature ok but no chain to a trusted key or ds record
[S] census.gov. 14400 IN DNSKEY 257 3 7 ;{id = 34566 (ksk), size = 2048b}
census.gov. 14400 IN DNSKEY 256 3 7 ;{id = 3725 (zsk), size = 1024b}
census.gov. 14400 IN DNSKEY 256 3 7 ;{id = 13964 (zsk), size = 1024b}
census.gov. 14400 IN DNSKEY 257 3 7 ;{id = 8151 (ksk), size = 2048b}
census.gov. 14400 IN DNSKEY 256 3 7 ;{id = 31118 (zsk), size = 1024b}
[S] census.gov. 14400 IN A 148.129.75.218
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples