Recent content on Ben E. C. Boyter
I recently did a brown bag at work, and as a change covered a non technical topic the subject of fortified wine. This was partly to encourage the team to realize they don’t have to discuss in depth how a data structure works, why the hip new programming language is worth looking at or even how to sandbox your applications using docker containers. Any topic shared with the team is acceptable.
One of the things I added to my code counter scc back when developing it was the COCOMO calculation. It’s not a reference to a song by the beach boys but instead the Constructive Cost Model , which is a model for estimating the cost of software developed by Barry Boehm in the late 70’s. 
 
 All models are wrong, but some are useful - George Box 
 
 Regardless if you…
The last few years have been pretty wild in the software development space. My passion project searchcode.com was deeply affected and as a result I had to shut it down and figure out what to do with it. 
 The reality is, in 2026 the average user of your site isn’t “Dave” filling in a form and clicking a mouse anymore. Half the internet is bot traffic and with LLM’s…
TL/DR; Play on itch.io or Plan or download from lexaloffle 
 I previously wrote about making a game using pico-8 . Once done I more or less instantly wanted to make another one and started sketching out some ideas. 
 Years ago I had watched a Youtube video https://www.youtube.com/watch?v=slDAvewWfrA about how Britain’s peak power demand spikes when the closing credits of a popular…
What felt like yesterday but is actually closer to 4 years ago I released the first version of code spelunker (cs) and wrote about it here . 
 I then promptly forgot about everything in it as I was focussed on everything else I had been working on, scc, searchcode, bonzamate, work, children. 
 Since a few of those are now not something that need my constant focus I decided to revisit cs…
I was looking through Google Scholar the other day looking for references to scc since I had previously had people reach out to me about how they were using it for their thesis (It’s not narcissism, its research I swear!). I was curious to see how it was being used. One thing that I noticed was how researchers were using it to measure size of projects. A valid use case IMHO. However I was…

 I am hoping to get another 5 years out of the machine as it is now. 
 
 My previous post about my desktop ended with the above. It has been about 5 years and while I was happy with my old desktop I had started looking into agentic workflows using LLM’s. After burning $50 in an afternoon just debugging some flows against OpenAI I decided I needed a better solution. 
 A…
searchcode HAS been rebooted see https://boyter.org/posts/searchcode-has-been-rebooted/ for details. 
 By the time you are reading this, searchcode.com’s DNS should have been updated to point at a placeholder index.html file I created. 
 In short I am shutting down the current iteration of searchcode.com, unless something drastic happens in the next few days. 
 The reasons for…
With the rise of AI LLM technology, I naturally wanted to learn more about using it. Both from a what can it do but also where can I use it in various tasks. Since I am also investing more time in searchcode.com these days I wanted to find a way I could use these tools to improve it. 
 One of the easiest things I could think of was by creating summaries of each file. This could then be used in…
Previously I had covered the worst program I had ever worked on (although that probably needs updating now) and the worst individual and the most incompetent individual I ever worked with. 
 I thought it was time to lay out one of my own failures, of which resulted in me adopting trust by verify for a all future interactions… at least until I am more familiar with the individual. 
…
Back in August 2021 during the covid madness I updated my rather aged desktop to have new CPU, GPU, PSU and fans. The goal being to make it quieter and get more life out of the system. I also kept it running Windows at the time, since I already have a stable Windows 10 install on it. However it is now 2025 and we are getting close to the end of life time for it in October. I never had any major…
At GopherCon2023 I presented a talk about how I built a custom index for searchcode . In there somewhere I mentioned that having a visualization of how the search works is something I would like to add in. 
 With some free time on a weekend rather than “wasting” time playing a game I decided to actually build it, and as is the current trend on the internet attempted to vide code…
I was a speaker at the first Go meetup in Sydney 2025… the first speaker in fact. As is my habit I am posting my notes for everyone who attended to view them. 
 I believe it was recorded and will post the link to that if I ever get the link. 
 You can find the link to the talk based on the blog post searchcode.com’s SQLite database is probably 6 terabytes bigger than yours .
I have been working on a secret project within https://searchcode.com/ and ran into a blocker where the UI had become a road-block. I will be the first to admit that I do not like styling things with CSS very much. In fact working with HTML and CSS is probably my least favorite part of modern web development. 
 I have never discussed any use of “AI” on this blog. Not for lack of…
searchcode.com’s SQLite database is probably one of the largest in the world, at least for a public facing website. It’s actual size is 6.4 TB. Which is probably 6 terabytes bigger than yours. 
 -rw-r--r-- 1 searchcode searchcode 6.4T Feb 17 04:30 searchcode.db
 
 At least, I think its bigger. I have no evidence to the contrary, being by far the largest I have ever heard of.…
Happy new year! 
 I am in the middle of converting searchcode.com over to SQLite, which is a far more interesting future post, and as a result had to rewrite some of the core service logic. The previously implemented code had grown over time so I decided to start fresh. Of course the catch of this is introducing regressions, which I did by stripping out some of the caching logic I had…
As much as I like ZeroLog for Go logging Slog being added to the standard library fits in with my rule of avoiding 3rd party packages when they are not entirely required hence I have started moving anything I work on over to it. 
 One thing I find especially annoying however is log messages inside tests. As such here is a quick code snippet which turns off all slog outputs, but only for tests.…
I was talking to someone recently about how searchcode.com works with regards to its bloom filter. The question came up about why it stores the index in memory and not on disk considering how fast disks are these days. 
 Now the answer is firstly performance, but the second is that writing all the code to store the index on disk is a lot of work and I didn’t feel like working on it.…
A work colleague (Dazza!) messaged me this morning about a new code counter with some pretty serious performance claims. Not only was it claiming to be accurate it was claiming to have a 38x speed improvement over tools such as tokei! 
 
 Code counters are something I am interested in so I did some digging to find the repository https://github.com/mainak55512/stto Interesting it is written…
Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my ServerlessDaysANZ 2024 talk about “Abusing Go, AWS Lambda and bloom filters to make a true Australian serverless search engine”. Whatever the case may be, welcome! 
 For the slides visit https://boyter.org/static/serverlessdaysanz2024/ then just use the space bar to…
I recently pushed out a new release of Sloc Cloc and Code (scc) with two main pieces of functionality. The first being accurate .gitignore support. The latter being the first new feature to hit the codebase in a long time in the form of a new metric you can access Unique Lines of Code or ULOC. 
 A few years ago exwhyz (who appears to have dropped off the internet) posted a feature request to…
I recently X’ed/Tweeted about how my searchcode.com since Nov 2022 using its custom index https://boyter.org/posts/how-i-built-my-own-index-for-searchcode/ had processed over 50 million queries with a rolling average time around 80ms to process them. 
 That was a low-ball estimate though. I actually checked and its actually just over 100,000,000 queries. Which I find astonishing…
I have always been a big fan of SQLite and recently was using it with Go/Golang. However due to fact that goroutines are very convenient I quickly ran into the “database is locked” issue where multiple writes would try to update the database at any one time. Multiple reads worked without issue. 
 I could have solved this by throwing a mutex lock around the database writes, but…
One of the things I see commonly on any mockup is a search box, usually with vaguely defined requirements as to how this is meant to actually work. However most people tend to assume its an instant update as you type with a debounce option of 100ms or something. 
 The fastest way to implement this with a database is a horrible concatenated like query. However where you can store things in…
Having had to install direnv multiple times over the last few days to get a team up and running im writing the method down because I keep having to look it up. 
 Install based on the following instructions https://github.com/direnv/direnv/blob/master/docs/installation.md 
Then hook it up to your shell via https://github.com/direnv/direnv/blob/master/docs/hook.md 
 Lastly to have it…
Sometime last year Justin Vincent of the techzing podcast suggests that I should add a license to
Quote, 
 
 I “strongly” suggest that you wrap each product into a paid version - where you sell a “license” for usage with priority support. 
 
 New year means new energy and so I got around to do this. All the tools will remain free software, so there is no…
In no particular order a list of Go libraries/packages I really like and some reasons why. 
 
 zerolog I know that the changes to slog make it far closer to being what zero log is now, but I still prefer it for my own use. 
 requests Anyone who works with python for a while learns about the requests library for making HTTP calls. This is similar but for Go. It has a few sharp edges…
So on occasion, I find myself wanting to run SQL against customer databases. Where this gets annoying is that because we are often in AWS using lambda I have no ability to connect to them. Occasionally im able to bully the friendly devops’ and security people to allow me access to a bastion host, which I can then connect to the SQL database. 
 However this is often a painful process if…
Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my GopherConSyd 2023 talk about “Bloom Filters: Building a Cutting Edge Go Search Engine to Explore the World’s Source Code”. Whatever the case may be, welcome! 
 For the slides visit https://boyter.org/static/gophercon-syd-presentation-2023/ then just use the…
TL/DR; Play/Download a copy from itch.io 
 I had always wanted to create games. In fact its one of the reasons I went down the development path. For various reasons I never invested enough time (probably too much time playing them) and went into writing code for businesses. 
 In hindsight that was probably the right move from a financial perspective, since it seems most game developers…
I was recently selected to present at DataEngBytes 2023 after submitting an idea for a talk based on my blog post Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100 . 
 I think it went well, other than not having the slides able to present for the first 10 minutes. After 5 mins or so I decided to just plough on since it was less of a visual talk anyway.…
Since I maintain a few projects that deal with source code one of the things I needed badly was a way to parse and understand gits .gitignore and .ignore files in order to get as much accuracy as possible. 
 I had previously tried using code I lifted from The Platinum Searcher albeit with some fixes to avoid crashes. Annoyingly however it never implemented glob’s correctly. I tried…
I released code spelunker a few days ago https://github.com/boyter/cs and literally one person asked for details on how it worked. Fitting in with my habit of putting any content I produce into my blog what follows is a built out version of it. 
 So code spelunker started when I noticed someone using Visual Studio to search files recursively in a directory and my own use of ripgrep with fzf.…
Code spelunker (cs) or code search is a new command line tool I have been working on and off over the last few years. It allows you to search over code or text files in the current directory either on the console, via a TUI or HTTP server, using some boolean queries or regular expressions. I just recently pushed out a stable first version release and thought I would write a quick post about it.
I previously wrote about starting a Go project in 2018 . A lot has changed since I wrote that and I had been wanting to write an updated version. What follows should be enough for anyone new to Go to get started and ideally start them being productive. 
 Quicklinks 
 
 Install / Setup 
 Starting a Project 
 Learning Go 
 Searching 
 Building / Installing 
 Linting /…
Another progress update on https://searchcode.com where I was trying to reduce the time it takes to index source code. Since the this was a real world example of profiling code trying to reduce its cost I thought I would document how it went. 
 The first step is to add code allowing the collection of a profile. 
 I don’t do this often enough in searchcode to have a want to…
TL;DR: If you don’t need to preserve order of the elements, sort the slice first and dedupe. 
 I have been working on https://searchcode.com a fair bit recently, having been dealing with memory issues which were causing it to crash with out of memory exceptions. The cause of that was due to backtracking regular expressions in the syntax highlighter, which will be the subject of another…
I have been doing interviews recently, and whenever someone mentions knowing a lot of Go and Java one of the questions I put to them is how to implement Go channels in Java. Since there isn’t anything online to suggest how to do this that I liked reading (that I could find) I thought I would write one, and do it in native Java without needing to import a library.
I have been investigating how ActivityPub is meant to work. What follows is how a boost/announce/retweet works in Mastodon and any other system that implements against ActivityPub. 
 You can find the full collection on github https://github.com/boyter/activitypub and contribute your own details. 
 ActivityPub -> Announce -> Post 
 When a user clicks on bonk/boost/retweet the following…
I didn’t really expect to have a post this late in the year but here we go. Lets go through some assumptions that were broken for me recently. Not just my own through perhaps some other peoples assumptions too. 
 I have been working with ActivityPub for a bit. I want to run my own server, beyond installing some 3rd party. I want to build my own from scratch. However this rapidly exposed…
Abstract TL/DR 
 I present what I belive is a unique index for indexing and searching source code. It copies ideas from Bing bitfunnel implementation to create a very fast, memory efficient trigram index over source code. 
 
 searchcode.com is now using a custom built index written by yours truly 
 It indexes 180-200 million documents and 75 billions lines of code 
 The index…
Some time ago I wrote about creating leet speak guid’s as a way of having fun while working with sitecore. https://boyter.org/2014/05/unique-guid/ 
 I recently realised I wanted to have this around so I could create more interesting guids for one reason or another, but couldn’t find the script I was using back then. So I created it again, and its now on github as a gist for myself…
Recently on my M1 Macbook I ran into the following issue when trying to compile some Go code with C dependencies. 
 
 
 The “clang” command requires the command line developer tools. 
 
 Being a Mac this also prompted me with a button I could click to install them. Naturally I clicked it, only to be prompted with the following. 

Working for a large corporate company as a mid level developer is generally rather boring. Churn out some CRUD applications. Write ETL as businesses live on data (usually processed in excel). Meetings that are meant to resolve things but just result in more meetings. 
 So in short, it is usually boring. However occasionally you get some sort of interesting problem, especially if you gain a…
Thought I would write done some of the more annoying/interesting software bugs I have had to deal with recently. 
 Both were dealing with an archive system where large amounts of video/audio/images are uploaded into, then searchable. It allows the editing of metadata for each type of content, as well sending to external systems. 
 Problem #1 
 When a record is being edited a flag…
Recently have been doing interviews at work for Go developers. 
 The filter for this is a simple review exercise. We present a small chunk of code and ask them to review it over 15 minutes pointing out any issues they see. The idea is to respect their and our time. It works pretty well and we can determine how much experience someone has by their ability to pick up the obvious vs subtle bugs.

 If you succeed, you will fail. 
 
 Something said at a client engagement I was at some time ago. I figure enough time has passed that it is worth sharing this story. 
 One of the early engagements was to deliver for a very large company an iOS Application powered by a RESTful API. This was problematic due to what is common for every large organisation, a collection of older…
Some time ago I wrote about building an Australian search engine by abusing how AWS Lambdas work. You can view it at bonzamate.com.au , and read the post about it here . 
 Recently I noticed that Bonzamate.com.au was in turn being abused by a series of bots repeatedly making queries against it. 
 Now normally I don’t worry about bots. There is little you can do to stamp them out…
A living document where I will be putting information about search index implementations as I learn about them. 
 So there are a few talked about ways to search across content or build an index that I am aware of. Lets discuss each in term with its advantages and disadvantages 
 Brute Force 
 Brute Force, note that this isn’t an index strategy per say, but worth discussing…
I have been working with AWS API Gateway and Lambda using Go a lot recently. One of the more annoying things about API Gateway and Lambda is the inability to run things locally. As a result I tend to write a lot of unit tests to compensate. This works up till you start doing things like SQL queries as mocking away the database is a less than ideal situation if you are working with raw SQL, which I…