mail.mil DNSSEC Outage: 2022-09-13

Date: September 13, 2022

Overview

This page gives some details on the mail.mil DNSSEC outage on September 13, 2022.

Timeline / DNSViz

Verisign's DNSSEC Debugger

Here's a screenshot I took on September 13, 2022, of the DNSSEC Debugger output:

September 13, 2022 kg TLD DNSSEC outage

Cloudflare DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

With DNSSEC, DNS queries fail:

$ dig +dnssec mx mail.mil. @1.1.1.1.

;; Truncated, retrying in TCP mode.

; <<>> DiG 9.11.5-P4-5.1+deb10u7-Debian <<>> +dnssec mx mail.mil. @1.1.1.1.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 54446
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 53 20 66 6f 75 6e 64 20 66 6f 72 20 6d 61 69 6c 2e 6d 69 6c 2e ("..no SEP matching the DS found for mail.mil.")
;; QUESTION SECTION:
;mail.mil. IN MX

;; Query time: 3441 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Sep 13 03:25:57 EDT 2022
;; MSG SIZE rcvd: 85

You have to disable DNSSEC to make DNS queries work:

$ dig +cd mx mail.mil. @1.1.1.1.

;; Truncated, retrying in TCP mode.

; <<>> DiG 9.11.5-P4-5.1+deb10u7-Debian <<>> +cd mx mail.mil. @1.1.1.1.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43360
;; flags: qr rd ra ad cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;mail.mil. IN MX

;; ANSWER SECTION:
mail.mil. 1292 IN MX 10 pri-jeemsg.eemsg.mail.mil.
mail.mil. 1292 IN MX 20 sec-jeemsg.eemsg.mail.mil.

;; Query time: 5 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Sep 13 03:25:57 EDT 2022
;; MSG SIZE rcvd: 97

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

[T] MAIL.MIL. 10800 IN DS 39829 8 1 5423d1e6fe7faaaa82943258017ac102068d35b0
MAIL.MIL. 10800 IN DS 39829 8 2 1ae3433f2b520815b9370d7933528d7758c76a9f9fa000876c041109513a9282
;; Domain: mail.mil.
[U] No data found for: mail.mil. type A
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples