.az (Azerbaijan) DNSSEC Outage: 20160112 - 20160114

Updated: January 14, 2016

Overview

This page gives some details on the .az (Azerbaijan) TLD partial DNSSEC outage from January 12 to January 14, 2016. It affected some resolvers (e.g. Unbound, Verisign Public DNS, and others) but not all.

Timeline / DNSViz

• 2016-01-12 16:56:48 UTC — .az is not yet signed
• 2016-01-12 19:31:24 UTC — .az gets DS, is DNSSEC-signed
• 2016-01-12 22:12:23 UTC — first personally observed edu.az DNSSEC failure (that didn't take long)
• 2016-01-12 22:15:42 UTC — first personally observed info.az DNSSEC failure
• 2016-01-12 22:15:55 UTC — first personally observed int.az DNSSEC failure
• 2016-01-12 22:18:39 UTC — first personally observed name.az DNSSEC failure
• 2016-01-12 22:18:54 UTC — first personally observed net.az DNSSEC failure
• 2016-01-12 22:19:52 UTC — first personally observed org.az DNSSEC failure
• 2016-01-12 22:21:23 UTC — first personally observed pp.az DNSSEC failure
• 2016-01-12 22:28:18 UTC — first personally observed biz.az DNSSEC failure
• 2016-01-12 22:29:49 UTC — first personally observed com.az DNSSEC failure
• 2016-01-13 03:19:23 UTC — com.az is busted
• 2016-01-13 03:28:38 UTC — biz.az is busted
• 2016-01-13 03:28:45 UTC — edu.az is busted
• 2016-01-13 03:29:11 UTC — info.az is busted
• 2016-01-13 03:29:37 UTC — int.az is busted
• 2016-01-13 03:30:03 UTC — name.az is busted
• 2016-01-13 03:30:57 UTC — pp.az is busted
• 2016-01-13 03:30:32 UTC — org.az is busted
• 2016-01-13 03:30:16 UTC — net.az is busted
• ...
• 2016-01-14 14:42:52 UTC — last personally observed edu.az DNSSEC failure
• 2016-01-14 14:50:00 UTC — last personally observed name.az DNSSEC failure
• 2016-01-14 14:50:20 UTC — last personally observed net.az DNSSEC failure
• 2016-01-14 14:51:34 UTC — last personally observed org.az DNSSEC failure
• 2016-01-14 14:52:32 UTC — last personally observed pp.az DNSSEC failure
• 2016-01-14 14:56:59 UTC — last personally observed biz.az DNSSEC failure
• 2016-01-14 14:57:50 UTC — last personally observed com.az DNSSEC failure
• 2016-01-14 15:02:42 UTC — last personally observed info.az DNSSEC failure
• 2016-01-14 15:02:55 UTC — last personally observed int.az DNSSEC failure

OpenDNS & Verisign Public DNS

OpenDNS does not support DNSSEC, instead supporting DNSCurve. Verisign Public DNS currently supports only DNSSEC, and thus, Verisign's users saw SERVFAIL for queries under .az during this outage.

With OpenDNS, queries succeed:

$ dig +dnssec www.google.com.az. @resolver1.opendns.com.

; <<>> DiG 9.4.2-P2 <<>> +dnssec www.google.com.az. @resolver1.opendns.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28380
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com.az. IN A

;; ANSWER SECTION:
www.google.com.az. 300 IN A 216.58.216.227

;; Query time: 172 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Wed Jan 13 03:37:36 2016
;; MSG SIZE rcvd: 62

With Verisign Public DNS, using DNSSEC, queries fail:

$ dig +dnssec www.google.com.az. @64.6.64.6

; <<>> DiG 9.4.2-P2 <<>> +dnssec www.google.com.az. @64.6.64.6
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 42408
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;www.google.com.az. IN A

;; Query time: 41 msec
;; SERVER: 64.6.64.6#53(64.6.64.6)
;; WHEN: Wed Jan 13 03:37:36 2016
;; MSG SIZE rcvd: 46

With Verisign Public DNS, WITH DNSSEC DISABLED (+cd), queries succeed:

$ dig +cd www.google.com.az. @64.6.64.6

; <<>> DiG 9.4.2-P2 <<>> +cd www.google.com.az. @64.6.64.6
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50513
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.az. IN A

;; ANSWER SECTION:
www.google.com.az. 300 IN A 216.58.216.227

;; Query time: 626 msec
;; SERVER: 64.6.64.6#53(64.6.64.6)
;; WHEN: Wed Jan 13 03:37:36 2016
;; MSG SIZE rcvd: 51

Logfile examples

• [1452655177] unbound[23878:0] info: validation failure <www.google.com.az. A IN>: no NSEC3 closest encloser from 147.28.0.39 for DS com.az. while building chain of trust
• [1452660261] unbound[23878:0] info: validation failure <www.renault.com.az. A IN>: no NSEC3 closest encloser from 147.28.0.39 for DS com.az. while building chain of trust
• [1452660372] unbound[23878:0] info: validation failure <youtube.com.az. A IN>: no NSEC3 closest encloser from 195.47.253.13 for DS com.az. while building chain of trust
• [1452660417] unbound[23878:0] info: validation failure <dnssec.org.az. A IN>: nameerror proof failed from 195.47.253.13
• [1452660833] unbound[23878:0] info: validation failure <www.visa.com.az. A IN>: no NSEC3 closest encloser from 194.87.0.9 for DS com.az. while building chain of trust