hhs.gov DNSSEC Outage: 2018-09-20

Date: September 20, 2018

Overview

This page gives some details on the hhs.gov DNSSEC outage on September 20, 2018. The U.S. Department of Health & Human Services has a budget of over 1 Trillion dollars per year.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from September 20, 2018:

September 20, 2018 hhs.gov DNSSEC outage

Google Public DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec mx hhs.gov. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec mx hhs.gov. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19696
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;hhs.gov. IN MX

;; Query time: 1156 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Sep 20 13:21:38 2018
;; MSG SIZE rcvd: 36

You have to disable DNSSEC to make DNS work:

$ dig +cd mx hhs.gov. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd mx hhs.gov. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64636
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hhs.gov. IN MX

;; ANSWER SECTION:
hhs.gov. 899 IN MX 10 smtp2.ees.hhs.gov.
hhs.gov. 899 IN MX 10 smtp.ees.hhs.gov.

;; Query time: 81 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Sep 20 13:21:38 2018
;; MSG SIZE rcvd: 72

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: hhs.gov.
;; Signature ok but no chain to a trusted key or ds record
[S] hhs.gov. 900 IN DNSKEY 257 3 8 ;{id = 10939 (ksk), size = 2048b}
hhs.gov. 900 IN DNSKEY 256 3 8 ;{id = 14863 (zsk), size = 1024b}
hhs.gov. 900 IN DNSKEY 256 3 8 ;{id = 42115 (zsk), size = 1024b}
hhs.gov. 900 IN DNSKEY 257 3 8 ;{id = 25769 (ksk), size = 2048b}
[S] hhs.gov. 900 IN A 52.7.111.176
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples