mail.mil DNSSEC Outage: 2022-11-02
Date: November 2, 2022
Overview
This page gives some details on the mail.mil DNSSEC outage on November 2, 2022.
Timeline / DNSViz
- 2022-11-02 12:07:13 UTC — No DNSKEYs
Google Public DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.
With DNSSEC, DNS queries fail:
$ dig +dnssec mx mail.mil. @8.8.8.8.
; <<>> DiG 9.11.5-P4-5.1+deb10u8-Debian <<>> +dnssec mx mail.mil. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 11990
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;mail.mil. IN MX
;; Query time: 3894 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Nov 02 08:04:45 EDT 2022
;; MSG SIZE rcvd: 37
You have to disable DNSSEC to make DNS queries work:
$ dig +cd mx mail.mil. @8.8.8.8.
; <<>> DiG 9.11.5-P4-5.1+deb10u8-Debian <<>> +cd mx mail.mil. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16303
;; flags: qr rd ra ad cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mail.mil. IN MX
;; ANSWER SECTION:
mail.mil. 499 IN MX 10 pri-jeemsg.eemsg.mail.mil.
mail.mil. 499 IN MX 20 sec-jeemsg.eemsg.mail.mil.
;; Query time: 7 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Nov 02 08:04:45 EDT 2022
;; MSG SIZE rcvd: 97
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
;; Domain: mail.mil.
[T] mail.mil. 125 IN DNSKEY 257 3 8 ;{id = 39829 (ksk), size = 2048b}
mail.mil. 125 IN DNSKEY 256 3 8 ;{id = 6051 (zsk), size = 1024b}
mail.mil. 125 IN DNSKEY 257 3 8 ;{id = 48547 (ksk), size = 2048b}
[U] No data found for: mail.mil. type A
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1667390643] unbound[25581:0] info: validation failure <mail.mil. A IN>: No DNSKEY record for key mail.mil. while building chain of trust
- [1667392996] unbound[25581:0] info: validation failure <mail.mil. A IN>: No DNSKEY record for key mail.mil. while building chain of trust
- [1667393016] unbound[25581:0] info: validation failure <mail.mil. A IN>: key for validation mail.mil. is marked as invalid because of a previous validation failure <mail.mil. A IN>: No DNSKEY record for key mail.mil. while building chain of trust