Reading · John Mueller's randoms

Sometimes you might find, that after a CMS migration, some things are missing. Obviously nobody would do such a thing on purpose, right? But it can happen. Suddenly, instead of something complete, you’re left with just guitars.

Let’s do a quick & dirty hack to check for important, lost URLs.

curl "http://web.archive.org/cdx/search/cdx?url=colored.house.com*&output=txt&from=20241201&to=20241231"  --output - \
| grep " text/html 200 " | awk '{print $3}' | sed 's/\?.*//' | sort | uniq -c | sort -nr \
| awk '{print $2}' | head -n 50 | xargs -I {} curl -s -o /dev/null -w "%{http_code} %{url_effective}\\n" {} \
| grep -v "^[23]"

There you are, the top “important” URLs from the site that worked in December, but which don’t work now (neither redirecting nor returning content). A serious site owner could then check the URLs for things that they either want to maintain (perhaps historical content), if there are new URLs to replace (perhaps one department is now replaced with another), or maybe they just want to keep things 404. SEOs might want to maintain the value of some of the old URLs, even if the replacements are just kinda close.

That command line is a bit extreme. Let’s break it down:

Getting URLs

I remembered that the Internet Archive has a way of extracting the URLs that it knows about. A quick search found a blog post that has examples. That lead to the initial curl URL, which looks for known URLs from start to end of December 2024 (adjust as needed).

Let’s just capture the output into a text file, since repeatedly calling archive.org is a bit rude. Also, sometimes these requests fail, in which case you should upload a donation of $5 and try again.

curl "http://web.archive.org/cdx/search/cdx?url=colored.house.com*&output=txt&from=20241201&to=20241231" \
 --output - > archive-wh-urls.txt

screenshot of an Internet Archive text list

The archive includes URLs that don’t work and ones which aren’t HTML, so we filter them out with grep " text/html 200 ". This also filters out ‘http’ URLs because, hopefully, you were redirecting to ‘https’. Then, we just keep the URLs from each line (which is the 3rd element in the line): awk '{print $3}'. So far, so good?

Canonicalization

Now, we have a set of URLs, but there’s still a lot of cruft - query parameters. Let’s cut off anything after a question mark: sed 's/\?.*//' (I never understand sed, but I think this works). I don’t think IA has fragment ("#") URLs, so we can skip those. (Hashtags are JavaScript, right, and who wants that?) This is a bit like the basic canonicalization that a search engine does. I’m sure you could use an LLM to do this too, but come on.

Ranking

Anyuway, now we have clean URLs, but a lot of duplicates. My assumption is that IA will check URLs with a frequency that’s comparable to their importance, so we’ll use that as our ranking function. Many search engines do something similar: if a URL is important, let’s check it more often. Ok, for ranking, we’ll train a machine learning model and feed it with cosine similarity embeddings of the URLs. Just kidding, we’ll just sort by number of times the URL occurs: sort | uniq -c | sort -nr | awk '{print $2}' (you have to sort before you can pull the unique entries, then we sort again by the count, and we’ll drop the count). Well, we’ve now done ranking by authority. We’re basically Google (minus the GPUs).

If you want to save the intermediate step, that would be:

cat archive-wh-urls.txt | grep " text/html 200 " | awk '{print $3}' \
 | sed 's/\?.*//' | sort | uniq -c | sort -nr | awk '{print $2}' \
 > archive-wh-urls-clean.txt

Live fetch without frogs

Now comes the funky part that I needed to gemini around for: checking URLs to see if they still work. For something reasonable, I’d just iterate over the list and make a simple, readable script. But we’re not here to be reasonable.

Taking the top URLs is the easy part: head -n 50. The funky part is iterating over the URLs and putting them into ‘curl’ to check their status code. We use ‘xargs’ here because it looks fancy. Do people actually do this in real life? It’s so wild. But then again, we live in weird times.

The ‘xargs’ basically calls ‘curl’ for each URL (the ‘{}’), we only want to see the status code & the URL, on separate lines: xargs -I {} curl -s -o /dev/null -w "%{http_code} %{url_effective}\\n" {}. This looks like rocket science, but who really knows what rocket scientists do all day? Apparently they’re not setting up clean CMS migrations, otherwise we wouldn’t be here.

Finally, we filter out the successful URLs (good job), to focus on failure: grep -v "^[23]" (status codes 2xx and 3xx are considered successful). Using regex here is fancy, and it’s a good way to show off your skills (omg this autocomplete is snarky, sorry).

In conclusion

And that’s it. Let’s put it all together, on separate lines this time:

curl "http://web.archive.org/cdx/search/cdx?url=colored.house.com*&output=txt&from=20241201&to=20241231" \
 --output - > archive-wh-urls.txt

cat archive-wh-urls.txt | grep " text/html 200 " | awk '{print $3}' | sed 's/\?.*//' \
 | sort | uniq -c | sort -nr | awk '{print $2}' > archive-wh-urls-clean.txt

cat archive-wh-urls-clean.txt | head -n 50 \
 | xargs -I {} curl -s -o /dev/null -w "%{http_code} %{url_effective}\\n" {} \
 | grep -v "^[23]"

And that’s how you can check for lost URLs after a CMS migration. If you want to clean things up, it’s best to do it within a few days - neither search engines nor users will wait 4 years for you to get your act together.

How did this one-liner turn into a long blog post? It’s wild. So much delving. Anyway.

Comments / questions

There's currently no commenting functionality here. If you'd like to comment, please use Bluesky and mention me there: @johnmu.com. Thanks!

Checking for lost URLs after a CMS migration