lb (Lebanon) DNSSEC Outage: 2015-03-12

Updated: March 14, 2015

Overview

This page gives some details on the lb (Lebanon) DNSSEC outage on March 12, 2015. The outage lasted approximately 11 hours.

Timeline / DNSViz

2015-03-12 14:06:01 UTC — RRSIGs expire
2015-03-12 15:42:01 UTC — expired RRSIGs
2015-03-12 16:42:50 UTC — expired RRSIGs
2015-03-12 18:36:38 UTC — expired RRSIGs
2015-03-12 19:37:48 UTC — expired RRSIGs
2015-03-12 20:09:13 UTC — expired RRSIGs
2015-03-12 21:09:29 UTC — expired RRSIGs
2015-03-12 22:29:36 UTC — expired RRSIGs
2015-03-13 00:09:14 UTC — expired RRSIGs
2015-03-13 01:00:09 UTC — outage essentially over
2015-03-13 02:01:27 UTC — some bogus RRSIGs remain
2015-03-13 06:53:33 UTC — some bogus RRSIGs remain
2015-03-13 11:08:10 UTC — some bogus RRSIGs remain
2015-03-13 15:52:19 UTC — some bogus RRSIGs remain
2015-03-13 21:52:19 UTC — bogus RRSIGs finally cleared

Verisign's DNSSEC Debugger

Here's a screenshot I took on March 12, 2015, of the DNSSEC Debugger output:

OpenDNS vs. Google Public DNS

While Google Public DNS supports DNSSEC, OpenDNS supports the superior DNSCurve, which is (among other advantages) immune to DNSSEC failures. During this outage, Google failed to resolve names under lb while OpenDNS worked normally.

With OpenDNS, queries succeed:

$ dig www.google.com.lb @resolver1.opendns.com

; <<>> DiG 9.4.2-P2 <<>> www.google.com.lb @resolver1.opendns.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45044
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.lb. IN A

;; ANSWER SECTION:
www.google.com.lb. 300 IN A 216.58.216.227

;; Query time: 29 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Thu Mar 12 16:46:28 2015
;; MSG SIZE rcvd: 51

With Google Public DNS, queries fail:

$ dig www.google.com.lb @8.8.8.8

; <<>> DiG 9.4.2-P2 <<>> www.google.com.lb @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27176
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.lb. IN A

;; Query time: 251 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Mar 12 16:46:22 2015
;; MSG SIZE rcvd: 35

dnscheck

dnscheck.iis.se archived a DNSSEC outage at 2015-03-12 11:10:15 (requires javascript).

dnscheck.labs.nic.cz archived a DNSSEC outage at 2015-03-12 11:11:04 (requires javascript).

Logfile examples

[1426177197] unbound[4862:0] info: validation failure <www.empire.com.lb. A IN>: signature expired from 203.119.56.132 for key lb. while building chain of trust
[1426178264] unbound[4862:0] info: validation failure <lb. NS IN>: signature expired from 77.72.229.254 for key lb. while building chain of trust
[1426179977] unbound[19032:0] info: validation failure <www.youtube.com.lb. A IN>: signature expired from 77.72.229.254 for key lb. while building chain of trust
[1426180680] unbound[4862:0] info: validation failure <lb. SOA IN>: signature expired from 193.188.128.14 for key lb. while building chain of trust
[1426198918] unbound[4862:0] info: validation failure <lb. NS IN>: signature expired from 77.72.229.254 for key lb. while building chain of trust
[1426199421] unbound[19032:0] info: validation failure <www.google.com.lb. A IN>: signature expired from 193.188.128.14 for key lb. while building chain of trust
[1426201891] unbound[19032:0] info: validation failure <lb. SOA IN>: signature expired from 193.188.128.14 for key lb. while building chain of trust
[1426207450] unbound[4862:0] info: validation failure <lb. NS IN>: signature expired from 77.72.229.254 for key lb. while building chain of trust