epa.gov DNSSEC Outage: 2021-04-19

Date: April 19, 2021

Overview

This page gives some details on the US Environmental Protection Agency (epa.gov) DNSSEC outage on April 19, 2021.

Timeline / DNSViz

Since DNSViz has lots its archives multiple times, here's a 3rd party copy from archive.is:

And here's a screenshot just in case:

April 19, 2021 epa.gov DNSSEC outage, DNSViz screenshot

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from April 19, 2021:

April 19, 2021 epa.gov DNSSEC outage

Google DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

$ dig +dnssec a epa.gov. @8.8.8.8.

; <<>> dig 9.10.8-P1 <<>> +dnssec a epa.gov. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 56945
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;epa.gov. IN A

;; Query time: 176 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 19 03:13:44 UTC 2021
;; MSG SIZE rcvd: 36

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a epa.gov. @8.8.8.8.

; <<>> dig 9.10.8-P1 <<>> +cd a epa.gov. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50730
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;epa.gov. IN A

;; ANSWER SECTION:
epa.gov. 59 IN A 134.67.21.34

;; Query time: 90 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 19 03:13:45 UTC 2021
;; MSG SIZE rcvd: 52

Zonemaster

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

[T] epa.gov. 3600 IN DS 36574 8 2 680e7b221d71d31e313af1e34a753c7dad4db695ddcf4650df2e63b49e83cf70
epa.gov. 3600 IN DS 61894 8 2 4aa4fe64b5cec8cd1a67bf89d26afe90952e5874ed5561c487980593a56c39db
;; Domain: epa.gov.
;; Signature ok but no chain to a trusted key or ds record
[S] epa.gov. 1296000 IN DNSKEY 256 3 8 ;{id = 17859 (zsk), size = 1024b}
epa.gov. 1296000 IN DNSKEY 257 3 8 ;{id = 505 (ksk), size = 2048b}
epa.gov. 1296000 IN DNSKEY 256 3 8 ;{id = 45300 (zsk), size = 1024b}
[S] epa.gov. 60 IN A 134.67.21.34
;;[S] self sig OK; [B] bogus; [T] trusted

dns.google.com

dns.google.com saw this outage. See the historical view courtesy of archive.is. Here's a screenshot:

dns.google.com view of April 19, 2021 epa.gov DNSSEC outage

Logfile examples