Dark Visitors recently published an API to grab updated robots.txt files from. After some stumbles and repeated hair-pulling at Hugo's lovely template language, I created a Hugo module to work with the API.

You can peruse the many curly braces within the module by viewing it on GitHub. Stay tuned though, because I’m about to step through it.

dark-visitors.html

{{- $url := "https://api.darkvisitors.com/robots-txts" -}}
{{- $api_key := getenv "HUGO_DARKVISITORS" -}}
{{- $bearer := printf "Bearer %v" $api_key -}}
{{- $agent_types := slice -}}
{{- if .Site.Params.darkVisitors -}}
	{{- range .Site.Params.darkVisitors -}}
		{{- $agent_types = $agent_types | append . -}}
	{{- end -}}
{{- else -}}
	{{- $agent_types = slice "AI Data Scraper" -}}
{{- end -}}
{{- $agent_types := $agent_types | jsonify -}}

We’re using Hugo’s os.Getenv to grab the API key from the environment variable I set earlier. We do a bunch of variable setup to work around templating limitations and to keep things from getting too hard to read, which is a constant battle. We check for configuration options and use “AI Data Scraper” if there are none. Then we throw it in the JSON blender and set up the request data.

dark-visitors.html

{{- $opts := dict
	"method" "post"
	"headers" (dict "Authorization" (slice $bearer) "Content-Type" "application/json")
	"body" (printf `{"agent_types": %s,"disallow": "/"}` $agent_types)
-}}

The request data is its own little ball of fun because Hugo wants a map for headers and a string for body. We pull in the bearer token from the variable we set earlier and throw it in a dict map for the headers, with slices for nested arrays. The body gets wild with printf—which is Go’s fmt.Sprintf in a trenchcoat—doing some string formatting to pull in the agent types.

dark-visitors.html

{{- with resources.GetRemote $url $opts -}}
	{{- with .Err -}}
		{{- errorf "%s" . -}}
	{{- else -}}
		{{- .Content -}}
	{{- end -}}
{{- else -}}
	{{- errorf "Unable to get remote resource %q" $url -}}
{{- end -}}

Now it’s time for the POST request. We’re using resources.GetRemote and this part is straight out of the docs with error checking.

We’re not done yet! We have configuring to do. The bare minimum config tells Hugo to generate a robots.txt for you:

hugo.yaml

enableRobotsTXT: true

The not-bare-minimum config sets up the API options. Dark Visitors offers three categories of bots:

hugo.yaml

params:
  darkVisitors:
    - AI Assistant
    - AI Data Scraper
    - AI Search Crawler

Respect the API

Don’t use the module if you build your Hugo site super frequently. Hugo can cache the API response when developing locally and when deploying to a server. This requires additional configuration. However, I build my site on CloudFlare Pages, which does not keep Hugo’s cache around between builds. Builds happen once or twice a day though.

What I Learned

I learned more about interacting with APIs and using environment variables. When I emailed Dark Visitors to ask why posting body:{...} as the request body wasn’t working (🤦‍♂️), they helpfully responded with copies of the log and a gentle note about my error. I found cool tools: httpie for pretty API responses in your terminal, and direnv for automatically loading environment variables from the .env file in my project directory. Some seriously great time savers here.

I also relearned the use of curly tie fighters {{-O-}} in Hugo templates to prevent whitespace issues. My robots.txt was full of weird indentation and multiple returns from the templating monster I created.