Let’s connect a static site generator, like Hugo, to IndexNow’s search engine submissions, using automatic Github Actions whenever you make changes. Will this change how your site appears in all search engines? Will your site’s SEO suddenly explode? No. Also No. I’m just here because I wanted to see if it was possible. And tl;dr: yes, you can. (Well, technically it doesn’t just send the last updates, but read on.)
You’ll need a static site generator, storage of your site in Github, a Github Action to build & publish your site, and a sitemap file that’s automatically generated.
Background
Static site generators like Hugo are great - they result in fast, static sites, which are easy to host anywhere (and … I’ve never seen them in the “Infected CMS distribution” of hacked sites). Getting them set up is a bit of a hassle at first, but you can set them up to publish automatically from Github, which gives you free backups too. Github has a browser-based editor, so it’s … almost like running a CMS with a database (not quite, but reasonably close).
IndexNow is a system originally from Bing which allows sites to submit new URLs to some search engines. Does it actually bring you traffic? Who knows, but it’s easy to implement. It sends the URLs to Bing, Yandex (Russia), Naver (South Korea), Seznam (Czech Republic), and a number of others. Practically speaking, I find it results in getting visited by the Yandex crawler and a set of, uhm, SEO tools. You can block the crawlers that you don’t like with robots.txt.
Publishing Hugo
For Hugo, peaceiris/actions-hugo is a simple integration that builds your Hugo site within Github Actions. The example actions file can be placed into .github/workflows/ and called something like publish-hugo.yml (the name is irrelevant, the .yml and the content matter). It sets up a Ubuntu environment, pulls your Github repository (with your website’s content), sets up Hugo, and builds it into the appropriate local folder (often /public/). The “Deploy” step is if you want to publish to Github pages, but you can swap this out for other hosters too, for example Firebase hosting.
Make sure this works independently before you continue. Edit a page, etc. Check that the hosting works & is updated.
Injecting into an index, now
IndexNow is fairly easy to set up. You get an API key, deposit the key in a file on your site, and then ping the service with new URLs specifying your key. Simple.
Get your key
API key from the service. You just need the API key (long numbers/letters thing). Hit regenerate until you see one you like. (Just kidding, nobody’s going to see it other than you.)
Store your key
In your Github repository, go to “Settings” - “Secrets and variables” - “Actions”. Here you see the names of the secrets attached to your repository. Maybe none. Maybe a hosting key.
Click “New repository secret”, give it the name “INDEXNOW_KEY” (or be creative, but remember it), and copy the API key into the Secret field. Click “Add secret”. You’re done. You don’t need the API key anymore (and also, you can’t look it up if you save it like this).
Github Action to create the local key file
Open your site’s Github action for publishing (from above). After the “build” section and before the deployment / publishing section, add the following lines (indent as needed):
- name: Create IndexNow key file
run: echo "${{ secrets.INDEXNOW_KEY }}" >public/${{ secrets.INDEXNOW_KEY }}.txt
If you have your static content in a different directory other than /public/, adjust as needed. This creates a file with the name of your API key value, with the contents of the API key value, without storing it in your repo.
Try it out, and make sure that you can access your file with https://whateveryoursiteis.com/123abcAPIKEYHERE123.txt (works best with your own domain name and the key value you received; actually, it only works like that.)
Submit the latest pages to processing
I do like me a nice command line. In the section after deployment / publishing, we’ll pull the 5 newest URLs from your sitemap file and submit them to the Index Now service. I found keeping state in GH Actions flakey, so instead of trying to track which pages changed, I just submit the 5 newest ones from the sitemap.
To do that, add the following to the bottom of the yml file, after deploying / publishing (again, indent as needed, adjust “public” if needed, etc.):
- name: Submit to index now
run: >
grep -o '<loc>[^<]*</loc><lastmod>[^<]*</lastmod>' public/sitemap.xml
| sed 's#<loc>\(.*\)</loc><lastmod>\(.*\)</lastmod>#\2 \1#'
| sort -r | head -n 5 | cut -d' ' -f2-
| xargs -I{} sh -c 'echo -n "Submitting {} ... ";
curl -s -o /dev/null -w "%{http_code}\n"
"https://www.bing.com/indexnow?url={}&key=${{ secrets.INDEXNOW_KEY }}"'
Don’t you just love a convoluted command line? Bash is life.
It basically takes your sitemap file, pulls the URLs & dates, sorts by date, and submits the 5 newest entries to Bing’s index-now endpoint using your API key.
When it runs, it shows the URLs it submits and the HTTP status code, which should be “200” if all’s well. If not, double-check your key, make sure the key file is actually created.
Triple-checking submissions
If you verify your site in Bing’s Webmaster Tools, you can check the status of some of the submissions. It doesn’t show them all, or maybe it just doesn’t show me mine. Anyway. Check and be happy.
Done
Well, there you have it. Automated Index Now submission via Github Actions from a static hosted website. No plugins.
Light disclaimer - I currently work for Google. This post (and all here) reflects my personal opinions, and not anything from Google. Google doesn’t use IndexNow. I can’t vouch for how well IndexNow works for other search engines. My knowledge of IndexNow is limited to the public documentation. I like how easily you can submit things there automatically, and it’s fun to try different technologies out. It’s unlikely that your site will rocket to the top of Bing because of this, but ChatGPT says that … ok, I’ll stop.
Comments / questions
There's currently no commenting functionality here. If you'd like to comment, please use Bluesky and mention me there: @johnmu.com. Thanks!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.