validatorsearch.verisignlabs.com 4.5-year DNSSEC Outage: 2013-2018

Updated: August 14, 2018

Overview

This page gives some details on the validatorsearch.verisignlabs.com DNSSEC outage that began on September 10, 2013, lasting until around March 22, 2018.

How validatorsearch.verisignlabs.com is intended to work, according to the site itself: "We elicit retry behavior from validators by omitting signature records from an initial response... A recursive name server that retries within a short period of time is marked as a validator." [emphasis mine]

In reality, DNS responses contain expired RRSIGs which are different from missing RRSIGs, though both are a common source of DNSSEC failures.

Timeline / DNSViz

DNSSEC Debugger

Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from October 20, 2017:

validatorsearch.verisignlabs.com dnssec outage

Google Public DNS: with and without DNSSEC

DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.

With DNSSEC, DNS queries fail:

$ dig +dnssec validatorsearch.verisignlabs.com. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +dnssec validatorsearch.verisignlabs.com. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 14017
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;validatorsearch.verisignlabs.com. IN A

;; Query time: 160 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 20 15:05:45 2017
;; MSG SIZE rcvd: 61

You have to disable DNSSEC to make DNS queries work:

$ dig +cd validatorsearch.verisignlabs.com. @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> +cd validatorsearch.verisignlabs.com. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15770
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;validatorsearch.verisignlabs.com. IN A

;; ANSWER SECTION:
validatorsearch.verisignlabs.com. 7199 IN A 72.13.58.64

;; Query time: 83 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 20 15:05:45 2017
;; MSG SIZE rcvd: 66

dnscheck

Zonemaster

drill trace

Since DNSSEC contains so much garbage, I put the complete drill trace into its own file with the relevant portion below (emphasis added):

;; Domain: validatorsearch.verisignlabs.com.
[B] validatorsearch.verisignlabs.com. 7200 IN DNSKEY 256 3 5 ;{id = 58962 (zsk), size = 1024b}
validatorsearch.verisignlabs.com. 7200 IN DNSKEY 257 3 5 ;{id = 38317 (ksk), size = 1280b}
[B] validatorsearch.verisignlabs.com. 7200 IN A 72.13.58.64
;; Error: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted

Logfile examples