nohats.ca DNSSEC Outage: 2022-07-05
Date: July 5, 2022
Overview
This page gives some details on the nohats.ca DNSSEC outage on July 5, 2022. This isn't the first DNSSEC outage for nohats.ca.
DNSViz / Timeline
- 2022-07-05 06:11:02 UTC — Bogus DNSSEC delegation
- 2022-07-05 06:23:36 UTC — Bogus DNSSEC delegation
- 2022-07-05 18:17:42 UTC — last personally observed nohats.ca DNSSEC failure
Here's a screenshot of DNSViz output during the nohats.ca DNSSEC outage:

This data was also archive.ph and archive.org.
DNSSEC Debugger
Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from July 5, 2022:

Cloudflare DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.
With DNSSEC, DNS queries fail:
$ dig +dnssec a nohats.ca. @1.1.1.1.
; <<>> DiG 9.11.5-P4-5.1+deb10u7-Debian <<>> +dnssec a nohats.ca. @1.1.1.1.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 42807
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 53 20 66 6f 75 6e 64 20 66 6f 72 20 6e 6f 68 61 74 73 2e 63 61 2e ("..no SEP matching the DS found for nohats.ca.")
;; QUESTION SECTION:
;nohats.ca. IN A
;; Query time: 219 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Jul 05 02:23:19 EDT 2022
;; MSG SIZE rcvd: 87
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a nohats.ca. @1.1.1.1.
; <<>> DiG 9.11.5-P4-5.1+deb10u7-Debian <<>> +cd a nohats.ca. @1.1.1.1.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46719
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 53 20 66 6f 75 6e 64 20 66 6f 72 20 6e 6f 68 61 74 73 2e 63 61 2e ("..no SEP matching the DS found for nohats.ca.")
;; QUESTION SECTION:
;nohats.ca. IN A
;; ANSWER SECTION:
nohats.ca. 3600 IN A 193.110.157.133
;; Query time: 209 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Jul 05 02:23:19 EDT 2022
;; MSG SIZE rcvd: 103
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
[T] nohats.ca. 86400 IN DS 56170 13 2 62f03ef6f2dd636c9c1b3fd5ad47bd086feff4a42d11e7502761a9f890af9f25
;; Domain: nohats.ca.
;; Signature ok but no chain to a trusted key or ds record
[S] nohats.ca. 3600 IN DNSKEY 256 3 13 ;{id = 14569 (zsk), size = 256b}
nohats.ca. 3600 IN DNSKEY 256 3 13 ;{id = 7001 (zsk), size = 256b}
nohats.ca. 3600 IN DNSKEY 257 3 13 ;{id = 25698 (ksk), size = 256b}
[S] nohats.ca. 3600 IN A 193.110.157.133
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1657001426] unbound[467:0] info: validation failure <bofh.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 193.110.157.102 for key nohats.ca. while building chain of trust
- [1657001541] unbound[467:0] info: validation failure <ns0.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657001654] unbound[467:0] info: validation failure <www.xn--nohts-5qa.com. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 188.127.201.225 for key nohats.ca. while building chain of trust
- [1657001797] unbound[467:0] info: validation failure <rogue.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 193.110.157.102 for key nohats.ca. while building chain of trust
- [1657002009] unbound[467:0] info: validation failure <cia.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 188.127.201.225 for key nohats.ca. while building chain of trust
- [1657002073] unbound[467:0] info: validation failure <mx-rogue.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657002145] unbound[467:0] info: validation failure <nssec.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 188.127.201.225 for key nohats.ca. while building chain of trust
- [1657002210] unbound[467:0] info: validation failure <thinkpad.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 193.110.157.102 for key nohats.ca. while building chain of trust
- [1657002288] unbound[467:0] info: validation failure <www.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 188.127.201.225 for key nohats.ca. while building chain of trust
- [1657002370] unbound[467:0] info: validation failure <gw.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 193.110.157.102 for key nohats.ca. while building chain of trust
- [1657003193] unbound[467:0] info: validation failure <ns2.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 193.110.157.102 for key nohats.ca. while building chain of trust
- [1657005108] unbound[467:0] info: validation failure <shrew.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657005828] unbound[467:0] info: validation failure <ns1.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657013217] unbound[467:0] info: validation failure <paola.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657013279] unbound[467:0] info: validation failure <tor.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657018120] unbound[467:0] info: validation failure <test.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 149.56.110.74 for key nohats.ca. while building chain of trust
- [1657045062] unbound[467:0] info: validation failure <ubuntu.nohats.ca. A IN>: key for validation nohats.ca. is marked as invalid because of a previous validation failure <rogue.nohats.ca. A IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 193.110.157.102 for key nohats.ca. while building chain of trust