Using Hugo Modules

Introduction

A little less than a year ago I switched to hugo for my blog and I’m still liking it. Since hugo did everything I wanted out of the box, I didn’t look into how third party components work. Aside from themes.

Recently, I wanted to update the robots.txt file to disallow known AI scrappers. Why is for another post. I started by looking at adding known user agents to the file but while I was looking, I found a very nice hugo module that pulls AI user agents from Dark Visitors automatically.

Unfortunately, I got an error after adding the module import to my hugo.yaml file.

❯ hugo
Total in 7 ms
Error: failed to load modules: module "github.com/lkhrs/hugo-dark-visitors" not found in "/Users/john/hugo/themes/github.com/lkhrs/hugo-dark-visitors"; either add it as a Hugo Module or store it in "/Users/john/hugo/themes".: module does not exist

The error message is a bit confusing because the paths are referring to the themes directory when this isn’t a theme. From the looks of it, modules were originally designed to allow easily importing themes but are used for a lot more now.

Fixing the error wasn’t obvious but it was very simple.

Converting to Modules

The fix is to turn my blog into a module because, interestingly, only modules can import other modules.

❯ hugo mod init nachtimwald.com
go: creating new go.mod: module nachtimwald.com
go: to add module requirements and sums:
	go mod tidy

Oh, I almost forgot, I had to take out the module import that was causing the error before running this. Otherwise I’d get the error again.

This creates two files that technically make my blog a module.

• go.mod
• go.sum

If you have experience with go, you’ll see hugo is using the module system built into go. go.mod declares the directory a module, has a bit of meta data like the name and go version. As well as any modules that are required and should be pulled in. The go.sum file includes check sums for anything defined as a dependency.

You do not need to edit these files or really worry about them.

Installing Modules

Once you’ve converted your blog to be a module you can start using modules. In my hugo.yaml file I added a module import section that lists all of the modules I want to use. Then I added the GitHub repo for the module (or modules) I want to use.

Here is the section with the AI user agent robots.txt generator.

module:
  imports:
    - path: github.com/lkhrs/hugo-dark-visitors

Once the modules are declared run hugo mod get to install them. You’ll see something like this.

❯ hugo mod get
hugo: downloading modules …
go: downloading github.com/lkhrs/hugo-dark-visitors v0.0.0-20240423145051-92ba9eed977f
go: github.com/lkhrs/hugo-dark-visitors@v0.0.0-20240423145051-92ba9eed977f requires go >= 1.22.2; switching to go1.22.3
go: downloading go1.22.3 (darwin/arm64)
go: downloading github.com/lkhrs/hugo-dark-visitors v0.0.0-20240423145051-92ba9eed977f
go: upgraded go 1.22.1 => 1.22.2
go: added toolchain go1.22.3
go: added github.com/lkhrs/hugo-dark-visitors v0.0.0-20240423145051-92ba9eed977f
hugo: collected modules in 45159 ms

Updating is easy too. Run hugo mod get -u.

If you haven’t noticed, this is just hugo calling go underneath.

Using a Theme as a Module

I mentioned earlier that modules are used with themes to make it very easy to add them to a hugo blog. Instead of copying a theme to the themes directory you can declare it as a module and have it managed by hugo.

To use a theme as a module, first make sure the theme is distributed as a module. Easiest way is to check the theme has a go.mod file at the top level.

You’ll define the theme in the config file using the project’s repository. For example with Hugo PaperMod you’d use github.com/adityatelange/hugo-PaperMod This is used with the theme parameter in hugo.yaml. Instead of looking for the theme as the directory name in the themes directory, it will instead pull and install the theme from the project repo. Here is a full example.

theme:
  - github.com/adityatelange/hugo-PaperMod

I found it does not need to be defined in the modules section. Even though I’ve seen multiple places online say it does. I think there is a bit of confusion but when I check the module graph, the theme is being pulled in and referenced properly.

❯ hugo mod graph
nachtimwald.com github.com/lkhrs/hugo-dark-visitors@v0.0.0-20240423145051-92ba9eed977f
nachtimwald.com github.com/adityatelange/hugo-PaperMod@v0.0.0-20240511144135-3f50861a0ced

Dark Visitors

This whole hugo module saga started after finding a lovely hugo module that integrates with the Dark Visitors API to automatically generate a robots.txt that “blocks” known AI scrappers.

The Dark Visitors module I’m using is not affiliated with Dark Visitors the service. The module is a third party project that uses the service’s API to pull the data.

Dark Visitors API

In order to use the module you need to create an account with Dark Visitors. The account is free. Once you have an account, make a project for the website. This will give you an API key in settings that lets you use the API with the hugo module.

You can generate the robots.txt file using CuRL and including the file instead of using the hugo module. However, the module will ensure any new user agents get included automatically in the future.

Module

You will need to set HUGO_DARKVISITORS as an environment variable with the API key from the service after creating an account. If you do not have this set the module will not function.

I didn’t set any parameters for the plugin to tune what’s included in the robots.txt that is generated. The default includes only AI scrapers and that’s all I want. However, Dark Visitors has several categories for things other than AI scrappers.

robots.txt template

Both the Dark Visitors module and PaperMod implement a robots.txt template. However, Hugo won’t stack or automatically combine templates when multiple modules implement the same one. Hence needing a custom template of my own that more or less combines the Dark Visitor module and the theme’s robots.txt output.

While the module does create a list of user agents to disallow, PaperMod adds a few additional things to the robots.txt file which I didn’t want to lose. For example, I really like PaperMod’s inclusion of the Sitemap parameter. Which is helpful for search engines. Unfortunately, the module does not have this entry in the robots.txt it generates.

{{- if and (ne (getenv "HUGO_DARKVISITORS") "") (or hugo.IsProduction (eq site.Params.env "production")) }}
{{ partial "dark-visitors.html" . }}
{{- end }}

User-agent: *
{{- if or hugo.IsProduction (eq site.Params.env "production") }}
Disallow:
{{- else }}
Disallow: /
{{- end }}

Sitemap: {{ "sitemap.xml" | absURL }}

The template starts with an if statement where I’m checking for the environment variable with the API key and whether hugo is building for production. This way the user agent list will only be pulled via the dark-visitors partial when it makes sense. If the API key isn’t defined or if it’s not production there is no reason to pull the user agent’s from the service.

When you run hugo it will automatically build for production. When using hugo server it builds for development. Having the production check will reduce how often it’s pulling the list of user agents. I average one post a month so this is more than sufficient to drastically cut down on unnecessary calls.

Another Production Only Method

While I’m using a custom robots.txt template, I want to call out the solution to respect the API that was contributed to the module’s issue tracker. It cleverly only includes the module when building for production by using a configuration directory instead of a single hugo.yaml file. The directory pattern allows having different configuration options for production vs development vs both. This works great if you’re only planning to use the module generated robots.txt.

Conclusion

One of the reasons I moved away from jekyll was because of issues I was having with plugins and now I’m using the hugo version of plugins (modules). While it seems counter intuitive, the number of modules I’m using is far fewer than what I needed with jekyll.

Also, the one module I’m using for blocking AI scrapers is pulling data from a service. Data I could manually pull from the service and create the robots.txt file myself. If the module breaks or if I need to stop using it, I still have a, relatively, easy fallback solution. The module just makes it easier to keep the list of AI scrapers up to date. So I’m not dependent on this module in any way.

The biggest thing I like about having moved to modules is having the PaperMod theme handled automatically. Previously, I had a git checkout and would need to remember to keep it updated. That’s not hard but it’s not obvious. The module method is cleaner and also makes it easier to try new themes in the future.