.fj (Fiji) DNSSEC Outage: 2022-03-07 to 2022-03-08
Date: March 8, 2022
Overview
This page gives some details on the .fj (Fiji) DNSSEC outage from March 7 to March 8, 2022.
Timeline / DNSViz
- 2022-03-07 23:55:46 UTC — first personally observed .fj DNSSEC failure
- 2022-03-08 00:02:05 UTC — Bogus DNSSEC delegation
- 2022-03-08 00:03:40 UTC — Bogus DNSSEC delegation
- 2022-03-08 00:40:58 UTC — Bogus DNSSEC delegation
- 2022-03-08 00:55:52 UTC — Bogus DNSSEC delegation
- 2022-03-08 04:46:15 UTC — Bogus DNSSEC delegation
- 2022-03-08 06:05:53 UTC — Bogus DNSSEC delegation
- 2022-03-08 06:22:18 UTC — Bogus DNSSEC delegation
- 2022-03-08 06:34:30 UTC — Bogus DNSSEC delegation
- 2022-03-08 06:38:47 UTC — Bogus DNSSEC delegation
- [many many pages snipped]
- 2022-03-08 14:17:52 UTC — Bogus DNSSEC delegation
- 2022-03-08 14:31:55 UTC — last personally observed .au DNSSEC failure
DNSSEC Debugger
Unlike DNSViz, Verisign's DNSSEC Debugger doesn't archive results, so here's a screenshot of my web browser's output from March 8, 2022.
Google Public DNS: with and without DNSSEC
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let's compare DNS queries with and without DNSSEC.
$ dig +dnssec aaaa fj. @8.8.8.8.
; <<>> dig 9.10.8-P1 <<>> +dnssec aaaa fj. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30425
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;fj. IN AAAA
;; Query time: 221 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Mar 07 23:57:53 UTC 2022
;; MSG SIZE rcvd: 31
You have to disable DNSSEC to make DNS queries work:
$ dig +cd aaaa fj. @8.8.8.8.
; <<>> dig 9.10.8-P1 <<>> +cd aaaa fj. @8.8.8.8.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31002
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;fj. IN AAAA
;; AUTHORITY SECTION:
fj. 1800 IN SOA ns1.fj. domreg.usp.ac.fj. 2022030811 21600 3600 604800 3600
;; Query time: 16 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Mar 07 23:57:53 UTC 2022
;; MSG SIZE rcvd: 85
Cloudflare
Cloudflare wrote about this DNSSEC outage in a blog post titled DNSSEC issues take Fiji domains offline.
Zonemaster
A couple zonemaster sites reported this DNSSEC outage:
- zonemaster.net (archive.ph copy)
- zonemaster.iis.se (archive.ph copy)
- zonemaster.labs.nic.cz didn't allow a test due to a bug in its webapp
Outages mailing list
This Fiji DNSSEC outage was discussed on the outages mailing list in the thread [outages] .fj TLD down.
There are copies courtesy of archive.org and archive.ph.
dns-operations mailing list
It was also discussed on the dns-operations mailing list hosted by DNS-OARC in the thread [dns-operations] TLD .fj broken (DNSSEC issue).
There are copies courtesy of archive.org and archive.ph.
dns.google.com
dns.google.com noticed the outage. And here's a screenshot:
drill trace
Since DNSSEC contains so much garbage, I put the complete drill trace into its own file, with the relevant portion below (emphasis added):
[T] fj. 86400 IN DS 18952 8 2 b22f5938ad822a76499a3ac295e061cc07fce36d7956e26a4f51aede1717f993
;; Domain: fj.
;; Signature ok but no chain to a trusted key or ds record
[S] fj. 3600 IN DNSKEY 257 3 8 ;{id = 12931 (ksk), size = 2048b}
fj. 3600 IN DNSKEY 256 3 8 ;{id = 24459 (zsk), size = 2048b}
[S] Existence denied: fj. A
;;[S] self sig OK; [B] bogus; [T] trusted
Logfile examples
- [1646697346] unbound[95966:0] info: validation failure <fj. NS IN>: signature missing from 144.120.146.65 for key fj. while building chain of trust
- [1646697433] unbound[95966:0] info: validation failure <fj. NS IN>: signature missing from 185.28.194.194 for key fj. while building chain of trust
- [1646697518] unbound[95966:0] info: validation failure <fj. NS IN>: signature missing from 185.38.108.108 for key fj. while building chain of trust
- [1646697677] unbound[95966:0] info: validation failure <fj. NS IN>: no keys have a DS with algorithm RSASHA256 from 144.120.146.1 for key fj. while building chain of trust
- [1646697778] unbound[95966:0] info: validation failure <fj. NS IN>: no keys have a DS with algorithm RSASHA256 from 144.120.146.1 for key fj. while building chain of trust
- [1646749866] unbound[95966:0] info: validation failure <fj. NS IN>: no keys have a DS with algorithm RSASHA256 from 144.120.146.1 for key fj. while building chain of trust
- [1646749915] unbound[95966:0] info: validation failure <fj. NS IN>: key for validation fj. is marked as invalid because of a previous validation failure <fj. NS IN>: no keys have a DS with algorithm RSASHA256 from 144.120.146.1 for key fj. while building chain of trust