caltech.edu DNSSEC Outage: 2020-04-06
Date: April 6, 2020
Overview
This page gives some details on the caltech.edu DNSSEC outage on April 6, 2020. The California Institute of Technology (Caltech) is a highly prestigious university with deep ties to the scientific world and an endowment of $2.98 billion.
Timeline / DNSViz
- 2020-04-06 14:54:10 UTC — expired RRSIGs
- 2020-04-06 19:01:48 UTC — expired RRSIGs
- 2020-04-06 22:08:54 UTC — last personally observed DNSSEC failure
- 2020-04-06 22:21:39 UTC — DNSSEC outage mostly over
DNSSEC Debugger
Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from April 6, 2020:
Google DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNS SEC.
With Google Public DNS (8.8.8.8), because of DNSSEC, queries fail:
$ dig +dnssec a caltech.edu. @8.8.8.8
; <<>> DiG 9.10.3-P4-Debian <<>> +dnssec a caltech.edu. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63608
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;caltech.edu. IN A
;; Query time: 3 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 06 11:12:27 UTC 2020
;; MSG SIZE rcvd: 40
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a caltech.edu. @8.8.8.8
; <<>> DiG 9.10.3-P4-Debian <<>> +cd a caltech.edu. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46152
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;caltech.edu. IN A
;; ANSWER SECTION:
caltech.edu. 21599 IN A 52.26.125.194
;; Query time: 3 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 06 11:12:27 UTC 2020
;; MSG SIZE rcvd: 56
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
;; Domain: caltech.edu.
[T] caltech.edu. 172800 IN DNSKEY 256 3 8 ;{id = 9606 (zsk), size = 1024b}
caltech.edu. 172800 IN DNSKEY 256 3 8 ;{id = 3435 (zsk), size = 1024b}
caltech.edu. 172800 IN DNSKEY 257 3 8 ;{id = 46795 (ksk), size = 2048b}
[B] caltech.edu. 43200 IN A 52.26.125.194
;; Error: DNSSEC signature has expired
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1586168379] unbound[58922:0] info: validation failure <its.caltech.edu. A IN>: signature expired from 131.215.139.100 for <7A4G1UAV9NAGTN4NQ8RQ6I11HRTSGHL4.caltech.edu. NSEC3 IN>
- [1586169549] unbound[58922:0] info: validation failure <cds.caltech.edu. A IN>: signature expired from 131.215.254.100
- [1586170125] unbound[58922:0] info: validation failure <its.caltech.edu. A IN>: signature expired from 131.215.139.100 for <7A4G1UAV9NAGTN4NQ8RQ6I11HRTSGHL4.caltech.edu. NSEC3 IN>
- [1586210853] unbound[58922:0] info: validation failure <caltech.edu. A IN>: signature expired from 131.215.254.99 for key caltech.edu. while building chain of trust
- [1586210934] unbound[58922:0] info: validation failure <cds.caltech.edu. A IN>: signature expired from 131.215.254.99 for key caltech.edu. while building chain of trust