libreswan.org DNSSEC Outage: 2019-05-20 to 2019-05-21
Updated: May 22, 2019
Overview
This page gives some details on the libreswan.org DNSSEC outage on May 21, 2019. It was probably related a DNSSEC failure within nohats.ca, which provides DNS service for libreswan.org. This time, affected domains were libreswan.net, libreswan.com, libreswan.org, and nohats.ca.
DNSViz / Timeline
DNSViz historical archives have been down for over a month at the time of this writing. DNSSEC makes its users completely give up.
I've included screenshots of DNSViz output since DNSSEC people don't care if things work or not.
- 2019-05-20 05:11:12 UTC — first personally observed libreswan.net DNSSEC failure
- 2019-05-20 05:38:52 UTC — first personally observed nohats.ca DNSSEC failure
- 2019-05-20 13:10:49 UTC — first personally observed libreswan.org DNSSEC failure
- 2019-05-20 18:49:02 UTC — first personally observed libreswan.com DNSSEC failure
- 2019-05-21 12:00:14 UTC — last personally observed libreswan.org DNSSEC failure
- 2019-05-21 12:04:28 UTC — last personally observed nohats.ca DNSSEC failure
- 2019-05-21 12:05:58 UTC — last personally observed libreswan.com DNSSEC failure
- 2019-05-21 12:06:10 UTC — last personally observed libreswan.net DNSSEC failure


DNSSEC Debugger
Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here are screenshots of my web browser's output from May 20, 2019:


Zonemaster
- zonemaster.iis archived for libreswan.org "Trying to verify NSEC3 RRset with RRSIG 22009 gave error 'DNSSEC signature has expired'."
- zonemaster.labs.nic.cz archived for libreswan.org "Trying to verify NSEC3 RRset with RRSIG 22009 gave error 'DNSSEC signature has expired'."
Google DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.
With DNSSEC, DNS queries fail:
$ dig +dnssec a libreswan.org. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +dnssec a libreswan.org. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 12879
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;libreswan.org. IN A
;; Query time: 223 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue May 21 03:43:34 2019
;; MSG SIZE rcvd: 42
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a libreswan.org. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +cd a libreswan.org. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9476
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;libreswan.org. IN A
;; ANSWER SECTION:
libreswan.org. 6581 IN A 188.127.201.229
;; Query time: 7 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue May 21 03:43:34 2019
;; MSG SIZE rcvd: 47
dns.google.com
dns.google.com is related to but separate from Google Public DNS. During this DNSSEC outage, dns.google.com showed the following for www.libreswan.org:

This data is also archived by web.archive.org.
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
;; Domain: libreswan.org.
[B] libreswan.org. 3600 IN DNSKEY 257 3 8 ;{id = 15232 (ksk), size = 2048b}
libreswan.org. 3600 IN DNSKEY 257 3 8 ;{id = 2644 (ksk), size = 2048b}
libreswan.org. 3600 IN DNSKEY 256 3 8 ;{id = 22009 (zsk), size = 2048b}
[B] libreswan.org. 7200 IN A 188.127.201.229
;; Error: No keys with the keytag and algorithm from the RRSIG found
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1558329072] unbound[51938:0] info: validation failure <libreswan.net. A IN>: signature expired from 149.56.110.74 for key libreswan.net. while building chain of trust
- [1558330732] unbound[51938:0] info: validation failure <pdc.nohats.ca. A IN>: signature expired from 149.56.110.74
- [1558357849] unbound[51938:0] info: validation failure <www.libreswan.org. A IN>: signature expired from 193.110.157.102
- [1558378142] unbound[51938:0] info: validation failure <libreswan.com. A IN>: signature expired from 193.110.157.102 for key libreswan.com. while building chain of trust
- [1558440014] unbound[51938:0] info: validation failure <www.libreswan.org. A IN>: signature expired from 149.56.110.74 for key libreswan.org. while building chain of trust
- [1558440268] unbound[51938:0] info: validation failure <gw.nohats.ca. A IN>: key for validation nohats.ca. is marked as invalid because of a previous validation failure <devconf.nohats.ca. A IN>: signature expired from 188.127.201.225 for key nohats.ca. while building chain of trust
- [1558440358] unbound[51938:0] info: validation failure <libreswan.com. A IN>: signature expired from 188.127.201.225 for key libreswan.com. while building chain of trust
- [1558440370] unbound[51938:0] info: validation failure <libreswan.net. A IN>: signature expired from 193.110.157.102 for key libreswan.net. while building chain of trust