tamu.edu DNSSEC Outage: 2023-01-05

Date: January 5, 2023

Overview

This page gives some details on the tamu.edu (Texas A&M University) DNSSEC outage on January 5, 2023. Texas A&M has around 72,000 students.

Timeline / DNSViz

CloudFlare Public DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec caa tamu.edu. @1.1.1.1.

; <<>> dig 9.10.8-P1 <<>> +dnssec caa tamu.edu. @1.1.1.1.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 35790
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; EDE: 7 (Signature Expired): 66 6f 72 20 44 4e 53 4b 45 59 20 74 61 6d 75 2e 65 64 75 2e 2c 20 69 64 20 3d 20 33 30 32 37 34 3a 20 52 52 53 49 47 20 74 61 6d 75 2e 65 64 75 2e 2c 20 65 78 70 69 72 61 74 69 6f 6e 20 3d 20 31 36 37 32 34 30 39 31 30 33 ("for DNSKEY tamu.edu., id = 30274: RRSIG tamu.edu., expiration = 1672409103")
;; QUESTION SECTION:
;tamu.edu. IN CAA

;; Query time: 331 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Jan 05 20:12:26 UTC 2023
;; MSG SIZE rcvd: 117

$ dig +cd caa tamu.edu. @1.1.1.1.

; <<>> dig 9.10.8-P1 <<>> +cd caa tamu.edu. @1.1.1.1.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30102
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 7 (Signature Expired): 66 6f 72 20 44 4e 53 4b 45 59 20 74 61 6d 75 2e 65 64 75 2e 2c 20 69 64 20 3d 20 33 30 32 37 34 3a 20 52 52 53 49 47 20 74 61 6d 75 2e 65 64 75 2e 2c 20 65 78 70 69 72 61 74 69 6f 6e 20 3d 20 31 36 37 32 34 30 39 31 30 33 ("for DNSKEY tamu.edu., id = 30274: RRSIG tamu.edu., expiration = 1672409103")
;; QUESTION SECTION:
;tamu.edu. IN CAA

;; AUTHORITY SECTION:
tamu.edu. 900 IN SOA csce-info-grid.net.tamu.edu. infoblox.tamu.edu. 3183055 14400 1440 2419200 900

;; Query time: 273 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Jan 05 20:12:26 UTC 2023
;; MSG SIZE rcvd: 181

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: tamu.edu.
;; Signature ok but no chain to a trusted key or ds record
[S] tamu.edu. 172800 IN DNSKEY 256 3 8 ;{id = 43116 (zsk), size = 2048b}
tamu.edu. 172800 IN DNSKEY 257 3 5 ;{id = 30274 (ksk), size = 2048b}
tamu.edu. 172800 IN DNSKEY 257 3 8 ;{id = 54682 (ksk), size = 2048b}
tamu.edu. 172800 IN DNSKEY 256 3 8 ;{id = 14109 (zsk), size = 2048b}
tamu.edu. 172800 IN DNSKEY 256 3 5 ;{id = 18378 (zsk), size = 1024b}
tamu.edu. 172800 IN DNSKEY 256 3 8 ;{id = 8370 (zsk), size = 2048b}
tamu.edu. 172800 IN DNSKEY 256 3 5 ;{id = 57843 (zsk), size = 1024b}
tamu.edu. 172800 IN DNSKEY 257 3 5 ;{id = 32956 (ksk), size = 2048b}
tamu.edu. 172800 IN DNSKEY 256 3 5 ;{id = 55857 (zsk), size = 1024b}
[S] tamu.edu. 300 IN A 165.91.22.70
;;[S] self sig OK; [B] bogus; [T] trusted; [U] unsigned

Logfile example