I have a fair bit of content on my site. I’ve been writing for over 5 years and at the time of this log have ~326 individual posts. My site is, and has always been, first and foremost a reference for myself. To be useful, to actually find stuff across all these posts, I need some sort of search capability. Here’s a brief history of the search capability on shellsharks.com and what I’d like to see search on this site do in the future…

Classic Search

When I first started the site, I picked a Jekyll theme that had already implemented search (super-search to be specific). Super search worked well enough, but in time I ran into some issues with it (which I honestly don’t remember what they were) and needed to find a new search solution. I liked super search because it displayed results dynamically as I typed my search query and it was decently fast/performant. From here, I went to Lunr.

Modern Search

The modern search capability on my site uses Lunr.js. It was easy enough to get started with, static-site-generator-compatible, and was easily tunable to search across all my different collection types. In practice, it works ok. It’s not “dynamic” in the same way that super search was, and it’s a bit clunkier given that I have it on a dedicated search page and not in-line across the site, but it does a decent job surfacing what I am looking for using a BM25-based scoring system. Lunr supports wildcards, field-level search, search term boosting, fuzzy matches, and logical ANDs. It also has some customization and plugin capabilties, but I haven’t really toyed around with that (yet). All this said, it’s not perfect. There’s still a few things that my perfect vision of search would be able to do…

The only thing I’ve really tinkered with related to Lunr is how I am constructing my page index. Below is the (very ugly) code that I have on my search page for creating the index. Notably, it specifies which pages are excluded from the index and then what front matter and body content is included across each collection type.

Creating the Lunr Search Index


{% assign counter = 0 %}
var documents = [{% for page in site.pages %}{% if page.url contains '.xml' or page.url contains 'assets' or page.url contains '.json' or page.url contains '.xsl' or page.title contains '- page' or page.url contains '/page/' or page.tags contains 'nosearch' %}{% else %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }, {% endif %}{% endfor %}{% for page in site.without-plugin %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }, {% endfor %}{% for page in site.posts %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.date | date: "%Y/%m/%d" }} - {{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }, {% endfor %}{% for page in site.notes %}{% if page.tags contains 'nosearch' %}{% else %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.date | date: "%Y/%m/%d" }} - {{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }, {% endif %}{% endfor %}{% for page in site.captain_logs %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.date | date: "%Y/%m/%d" }} - {{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }, {% endfor %}{% for page in site.scrolls %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.date | date: "%Y/%m/%d" }} - {{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
}, {% endfor %}{% for page in site.links %}{% if page.tags contains 'nosearch' %}{% else %}{
    "id": {{ counter }},
    "url": "{{ page.link }}",
    "title": "{{ page.title }}",
    "body": "{{ page.date | date: "%Y/%m/%d" }} - {{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }, {% endif %}{% endfor %}{% for page in site.devlogs %}{
    "id": {{ counter }},
    "url": "{{ site.url }}{{ page.url }}",
    "title": "{{ page.title }}",
    "body": "{{ page.date | date: "%Y/%m/%d" }} - {{ page.content | markdownify | replace: '.', '. ' | replace: '</h2>', ': ' | replace: '</h3>', ': ' | replace: '</h4>', ': ' | replace: '</p>', ' ' | strip_html | strip_newlines | replace: '  ', ' ' | replace: '"', ' ' }}"{% assign counter = counter | plus: 1 %}
    }{% if forloop.last %}{% else %}, {% endif %}{% endfor %}];

I think having a native search capability on your site is very important, but I can’t dismiss the potential value of a traditional web search. Look, Google and other web search providers aren’t what they used to be, that’s undeniable. But, you can go on Google right now and do a good ol’ fashioned “site:shellsharks.com” search and pull back actual results from my site. So yeah, in a pinch, if your site is indexed by Google, or other search engines, you can fall back on that for search.

Next-Generation Search

Alright, let me chat a bit about what I’d like to see in future versions of my sites search capability. I’ve got a wishlist below actually…

Search Wish List
  • Searches across all pages of my site, but customizable so I can limit based on front matter (e.g. tag), or collection type (e.g. link)
  • Performant
  • Regex-capable
  • Supports boolean logic (i.e. AND, OR, NOT, etc…)
  • Can be put in-line in the header, rather than having to load into the separate /search page
  • Dynamically loads results (you don’t have to “submit” the queries) as you enter search terms
  • Search results return not only the page which matches the search term, but also a snippet from that page with the terms highlighted
  • The ability to filter results based on a number of criteria (e.g. date, collection, tag, etc…)
  • No server-side requirement (preferably)

Yep, that’s it. Put search on your site!!