archive.is stopped loading: EDNS Client Subnet

archive.is stopped loading at home. Every other site was fine, the name resolved instantly, and it had an A record, so my first assumption was that this was not a DNS problem. It was a DNS problem, just not a lookup failure: resolution worked and handed back an address that went nowhere.

The first measurement already looked odd. The host answered pings and port 80, but port 443 timed out rather than refusing:

A bare nginx 404 is not what archive.is serves. Asking a few public resolvers gave a different address from each one, and all of those worked:

local resolver     95.213.142.50     :443 timeout
1.1.1.1            104.36.80.106     200
8.8.8.8            168.222.241.49    200
9.9.9.9            104.36.80.107     200
208.67.222.222     123.30.186.252    200

So archive.is was reachable from my connection the whole time. Only the address my own resolver produced was dead. For comparison, a real archive.is node answers port 80 with nginx/1.9.2, sets a cookie and redirects to HTTPS. The one I had been given was some unrelated host.

Different answer for every querier

I run Blocky as the resolver for the house, and it had the bad address cached with the TTL counting down. Flushing the cache produced a working node, which fixes the symptom and explains nothing.

Querying archive.is authoritative servers directly gave a stable answer with a 300 second TTL, and no rotation at all across 40 queries. The variation was not random. It tracked who was asking, which points at EDNS Client Subnet: the resolver passes a truncated prefix of the client address to the authoritative server, which uses it to pick a nearby node.

dig +subnet sets that field by hand:

The first subnet is my ISP prefix and maps to a node that works. The second is the egress address of the upstream resolver Blocky happened to be talking to, and maps to the dead one. That reproduces every time, at /24, /20 and /16.

Blocky was configured with three encrypted upstreams raced against each other:

upstreams = {
  groups.default = [
    "quic:dns.quad9.net:853"
    "quic:unfiltered.adguard-dns.com:853"
    "quic:dns.nextdns.io:853"
  ];
  strategy = "parallel_best";
};

Quad9 and NextDNS strip ECS, which is a deliberate privacy choice on their part. With the field gone, archive.is sees only the resolver egress, which for that path was a Vultr address in the US, and returns the node it maps to that location. AdGuard forwards ECS, so when it won the race the answer was fine. Which upstream wins is a race, so the failure came and went.

You can see what a resolver forwards by asking Google’s authoritative server to reflect it back:

A one hour floor on a 300 second TTL

The second half is the cache. My config had this:

caching = {
  minTime = "1h";
  maxTime = "12h";
  prefetching = true;
};

minTime raises any shorter TTL to the floor. archive.is publishes 300 seconds, so a bad answer that should have aged out in five minutes was pinned for an hour, for every client in the house. I had added the floor for cache hit rate without thinking about what short TTLs are usually for: failover, geo steering, CDN cutovers.

The floor was also doing less than I assumed. Blocky’s metrics show 1.41M cached responses at 0.14 ms mean against 116k upstream resolves at 16 ms median, and of those cached hits about 91% were prefetch hits. The prefetcher keeps the hot set warm on its own, and the working set is small enough that the floor adds little. I removed minTime and kept maxTime as a ceiling.

For the upstreams I dropped the two that strip ECS. That leaves less choice than I expected: the resolvers that support DNS-over-QUIC mostly strip ECS by design, and the ones that forward it are the large operators on DNS-over-TLS. Measured from my resolver host, AdGuard unfiltered forwards ECS over QUIC at 49 ms median, and Google forwards it over TLS at 67 ms with no QUIC endpoint. Quad9’s dns11 and OpenDNS forward ECS too, but both filter. Since parallel_best races everything in the group, one stripping entry is enough to bring the fault back intermittently, so the set has to be uniform.

That is the part I am least settled on. ECS leaks a truncated client prefix to every authoritative server you touch, which is exactly what Quad9 and NextDNS are avoiding, and I gave that up to make geo steering work. The failure mode that pushed me there is not specific to archive.is: any CDN or geo steered service can hand you a badly placed node the same way. It usually shows up as slow rather than broken, which is harder to notice than a port that times out.