www.cloudflare.com DNSSEC Outage: 2019-03-21

Date: March 21, 2019

Overview

This page gives some details on the www.cloudflare.com DNSSEC outage on March 21, 2019. Cloudflare is one of the largest DNSSEC providers. I saw this DNSSEC outage at DNSViz, Verisign's DNSSEC Debugger, Google Public DNS, and DNS-OARC (both Unbound and BIND!), in addition to my 3 Unbound instances. This particular outage was caused by a less common type of DNSSEC failure that I've only seen in Cloudflare and TinyDNSSEC.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from March 21, 2019:

March 21, 2019 www.cloudflare.com DNSSEC outage

Since the above image is confusing — why should a DS record for www.cloudflare.com matter?! — I've included an example screenshot of what this analysis is supposed to look like:

www.cloudflare.com analysis

(Apparently www.cloudflare.com has NS records):

$ dig +short ns www.cloudflare.com.
jule.ns.cloudflare.com.
vin.ns.cloudflare.com.

Google Public DNS

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

With Google Public DNS, because of DNSSEC, queries fail:

$ dig +dnssec a www.cloudflare.com. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.cloudflare.com. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 60321
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.cloudflare.com. IN A

;; Query time: 43 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Mar 21 21:42:15 2019
;; MSG SIZE rcvd: 47

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a www.cloudflare.com. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd a www.cloudflare.com. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35624
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.cloudflare.com. IN A

;; ANSWER SECTION:
www.cloudflare.com. 187 IN A 104.17.209.9
www.cloudflare.com. 187 IN A 104.17.210.9

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Mar 21 21:42:15 2019
;; MSG SIZE rcvd: 68

DNS-OARC: BIND

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNS SEC.

With DNS-OARC's BIND instance, because of DNSSEC, queries fail:

$ dig +dnssec a www.cloudflare.com. @184.105.193.73

; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.cloudflare.com. @184.105.193.73
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 268
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;www.cloudflare.com. IN A

;; Query time: 176 msec
;; SERVER: 184.105.193.73#53(184.105.193.73)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 47

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a www.cloudflare.com. @184.105.193.73

; <<>> DiG 9.4.2-P2 <<>> +cd a www.cloudflare.com. @184.105.193.73
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50288
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.cloudflare.com. IN A

;; ANSWER SECTION:
www.cloudflare.com. 294 IN A 104.17.209.9
www.cloudflare.com. 294 IN A 104.17.210.9

;; Query time: 21 msec
;; SERVER: 184.105.193.73#53(184.105.193.73)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 68

DNS-OARC: Unbound

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNS SEC.

With DNS-OARC's Unbound instance, because of DNSSEC, queries fail:

$ dig +dnssec a www.cloudflare.com. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.cloudflare.com. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 14194
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;www.cloudflare.com. IN A

;; Query time: 21 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 47

You have to disable DNSSEC to make DNS queries work:

$ dig +cd a www.cloudflare.com. @184.105.193.74

; <<>> DiG 9.4.2-P2 <<>> +cd a www.cloudflare.com. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47381
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.cloudflare.com. IN A

;; ANSWER SECTION:
www.cloudflare.com. 295 IN A 104.17.210.9
www.cloudflare.com. 295 IN A 104.17.209.9

;; Query time: 24 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 68

Logfile examples

These log entries come from 3 different Unbound instances, all on different computers in different geographical regions.