caltech.edu DNSSEC Outage: 2020-04-06

Date: April 6, 2020

Overview

This page gives some details on the caltech.edu DNSSEC outage on April 6, 2020. The California Institute of Technology (Caltech) is a highly prestigious university with deep ties to the scientific world and an endowment of $2.98 billion.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from April 6, 2020:

April 6, 2020 caltech.edu DNSSEC outage

Google DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNS SEC.

With Google Public DNS (8.8.8.8), because of DNSSEC, queries fail:

$ dig +dnssec a caltech.edu. @8.8.8.8

; <<>> DiG 9.10.3-P4-Debian <<>> +dnssec a caltech.edu. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63608
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;caltech.edu. IN A

;; Query time: 3 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 06 11:12:27 UTC 2020
;; MSG SIZE rcvd: 40

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a caltech.edu. @8.8.8.8

; <<>> DiG 9.10.3-P4-Debian <<>> +cd a caltech.edu. @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46152
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;caltech.edu. IN A

;; ANSWER SECTION:
caltech.edu. 21599 IN A 52.26.125.194

;; Query time: 3 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 06 11:12:27 UTC 2020
;; MSG SIZE rcvd: 56

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):

;; Domain: caltech.edu.
[T] caltech.edu. 172800 IN DNSKEY 256 3 8 ;{id = 9606 (zsk), size = 1024b}
caltech.edu. 172800 IN DNSKEY 256 3 8 ;{id = 3435 (zsk), size = 1024b}
caltech.edu. 172800 IN DNSKEY 257 3 8 ;{id = 46795 (ksk), size = 2048b}
[B] caltech.edu. 43200 IN A 52.26.125.194
;; Error: DNSSEC signature has expired
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples