arin.net DNSSEC Outage: 2019-01-11

Date: January 11, 2019

Overview

This page gives some details on the arin.net DNSSEC outage on January 11, 2019. ARIN is the American Registry for Internet Numbers, one of the Internet's 5 RIRs. The DNSSEC outage was fixed by removing the DS record from the .net servers.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from January 11, 2019:

January 11, 2019 arin.net DNSSEC outage

Google Public DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC. With DNSSEC, DNS queries result in SERVFAIL:

$ dig +dnssec ns arin.net. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec ns arin.net. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 44632
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;arin.net. IN NS

;; Query time: 51 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jan 11 13:20:10 2019
;; MSG SIZE rcvd: 37

You have to disable DNSSEC to make DNS work:

$ dig +cd ns arin.net. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd ns arin.net. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39545
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;arin.net. IN NS

;; ANSWER SECTION:
arin.net. 21599 IN NS u.arin.net.
arin.net. 21599 IN NS ns1.arin.net.
arin.net. 21599 IN NS ns3.arin.net.
arin.net. 21599 IN NS ns2.arin.net.

;; Query time: 17 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jan 11 13:20:10 2019
;; MSG SIZE rcvd: 96

Zonemaster

Note: Zonemaster requires javascript.

DNS-OARC

This DNSSEC outage was discussed on the dns-operations mailing list hosted by DNS-OARC, in the thread [dns-operations] Signatures expired at arin.net.

NANOG

This DNSSEC outage was also discussed on the NANOG mailing list in the thread ARIN NS down?

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: arin.net.
[B] arin.net. 43200 IN DNSKEY 257 3 5 ;{id = 49918 (ksk), size = 1024b}
arin.net. 43200 IN DNSKEY 257 3 5 ;{id = 48281 (ksk), size = 1024b}
arin.net. 43200 IN DNSKEY 256 3 5 ;{id = 5040 (zsk), size = 1024b}
arin.net. 43200 IN DNSKEY 256 3 5 ;{id = 11986 (zsk), size = 1024b}
[B] arin.net. 600 IN A 199.43.0.44
arin.net. 600 IN A 199.43.0.43
;; Error: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples