Until this week, search.feep.dev was served via Cloudflare and Cloudflare Tunnel . The only reason for this was that Feep! runs on a server in my living room, and I didn’t want to expose my home’s IP address to the entire world.
Because Feep! Search runs on a server with a fairly small hard disk (and also to making huge copies all the time), I need a way to share files between my production and development environments, ideally without causing dev changes to accidentally mangle production data. I’m currently using hardlinks to read-only files, which is a strategy that can sort of emulate copy-on-write data but has some…
In my last post, I wrote about cleaning up URL processing . Those URLs are used to process information about pages and the links between them (for computing things like PageRank metrics), and all that data needs to get stored somewhere. Until recently I was using a collection of ad-hoc file formats, which meant that pulling any data out required writing a bunch of bespoke code. This was tedious at…
Over time, the indexing and ranking code has gotten hard to follow, in part because I was using the term “ URL ” for what turned out to be several different things. I recently cleaned that up, which not only made the code easier to work with but also gave me a good opportunity explain how the system thinks about links.
The branding for Feep! search has tended to be minimal at best. This lack of branding is part of what made adding dark mode so easy: there’s hardly any styling to begin with, a sort of Béton brut aesthetic with only a thin layer of paint over the underlying code. Still, branding isn’t a terrible thing to have; so today I’d like to introduce you to the new mascot for my search engine.
I said in my last post that I was going to try out OpenHands; but then somebody showed me Aider Chat and I decided to try that instead, on the strength of a personal recommendation. The results were very good, and confirmed my suspicions that having the AI agent as a collaborator rather than attempting to one-shot it seems like a good idea. I got a fix I’m happy with in one go, with only a little…
Like almost everybody, I’ve been using LLMs more and more recently. I’ve used ChatGPT to generate quick scripts , and I use GitHub Copilot autocompletions in my IDE; but I haven’t yet tried going as far as getting an LLM to autonomously make a complicated change in an existing codebase. So, in the spirit of my post on 20 minutes with Alpaca.cpp , here's a post about a morning with SWE-agent .
Feep! now has a dark mode theme! The stylesheet for the site is a fairly minimal set of overrides on top of the browser default stylesheet, so adding this support was actually pretty straightforward, though there were a few complications.
I run a personal GitLab instance on a local machine. Since there’s no real reason to expose it to the internet, and I don’t trust myself to stay on top of security patches, it’s only accessible inside my network. However, I still want it to have an HTTPS certificate, mostly because more and more things (correctly) insist on using an encrypted connection. If the instance were publicly accessible,…
Since I just updated to a newer copy of the Stack Exchange data dumps , I’ve been shepherding that process through my CI pipelines to get the updated data into production. Even when everything goes well this takes a couple of days to finish (my server is not very fast), but invariably it doesn’t go well and I have to make tweaks and restart it a couple of times. Since it can fail unexpectedly at…
My search engine got its start from StackExchange data dumps , and they’re still one of the major data sources I use. But the official source I was using stopped updating in April 2024, so my data has been stuck in time for a while. I finally got around to investigating why I hadn’t gotten updates, and it turns out that the interesting part of the issue is not some technical problem on my end, but…
The title kind of spoils the story, but I ran into a confusing bug recently and it seemed interesting enough to write about.
Someone emailed me a few weeks ago and asked me to add an RSS feed to this blog. That’s done (and while I was at it I added h-entry markup to the index page in case anyone has a use for it); and since I’m here I figured I may as well talk about the underlying code, which has changed quite a lot since my last post on the topic .
There’s been a sudden explosion of advancements in Large Language Models in the past couple of weeks. Lots of the discussion has been around how they can be used to write code, so I’ve been thinking of how this might be useful for Feep! search. The release of llama.cpp and Alpaca seems to have made it practical for me to play around with this locally, so I spent a few minutes this afternoon trying…
As I mentioned briefly in my post on the CDB file format , I had some trouble understanding how the JavaScript bitwise operators worked. I couldn’t find a good resource that clearly explained this, so now that I’ve spent a bunch of time figuring this out, here are some diagrams to help clarify the situation.
For a new feature I’m building (“quick results”, coming Soon™), I needed a simple key-value file format, ideally one that was fast and easy work with. The “CDB” format seemed to fit the bill, but there weren’t any JavaScript libraries for it that I liked, so I found myself digging in to the internals to undertand how it worked; and this blog post is a summary of my investigation into the details…
In my previous post , I talked about crawling the web and mentioned that I planned to start with the ability to download a list of URLs and write them to a WARC file. At the time, I was imagining that I’d have to write some code to do this, but while I was doing some research I discovered that wget added some basic support for WARC output a few years ago.
I’m currently relying on various bulk exports ( Stack Exchange XML dumps , Zim files from Wikipedia, and so on) to populate the search engine. This is a convenient way to get a lot of pages in very quickly—a single export file contains hundreds or thousands of pages, with generally well-structured HTML—but it’s also limiting: most of the web doesn‘t come in nicely structured dump files, but I’d…
A 2004 article on Why Writing Your Own Search Engine is Hard was posted on Hacker News today, and I wrote a comment comparing it with my experiences writing my own search engine 18 years later. This post is that comment, cleaned up and expanded a bit. (Go read the article first, I quote a bit from it here but this post won’t make as much sense without context.)
To prepare for setting up a production environment, I just added a some code to handle the situation when the server gets overloaded. The actual implementation is simple, but the theory behind it is quite interesting.
I’m using docker-compose to deploy Feep! Search to production. I’ve used it in similar capacity for a few other projects, too. This feels like the wrong way to do it (seems like there really ought to be systemd units involved somehow), but in practice it works great.
Despite my previous enthusiasm about redo , I’ve just finished tearing the last of my .do files out and replacing them with shell scripts again. I don’t think the problems I had were redo ’s fault, really; I was just asking it to do things it wasn’t really designed for.
I’ve spent the last week or so writing down all the ideas I have for Feep! Search, to get them out of my head. (I also wrote my own bug tracker to write them in, but that’s a topic for another time.) This has given me some thoughts on project management, which seemed like a good topic for a mildly interesting (if ironically somewhat disorganized) blog post.
Almost every time I update StackOverflow's data files (by far the largest single data source I have, at 40 GB), I find out halfway through the process that I don’t have enough disk space and need to clear out some old data. This is incredibly annoying when it happens, so I took the opportunity today to make the failure a bit more graceful and prompt.
I've been writing blog posts, but so far they've just been markdown files in a folder and there hasn't been any way to read them. Today I spent a few minutes working on adding them to the web interface.
Today I added favicons the search results, to make them more visually interesting.
Previously , I introduced the concept of "chunks" for use in generating link data. This time, I'd like to apply the same principle to the process of importing pages into ElasticSearch. On the surface, this looks the same as for the link data; however, there are some complications because I'm now working in ElasticSearch instead of with files on a disk.
In my last post I showed a big diagram of all the steps to get a StackExchange post from a dump file into the search index, and drew some boxes around different parts. This time I'm looking at the .link.dat generation, which is shown in the diagram as part of pagerank , but was actually managed by the datasource.
I've gone through several different schemes for coordinating everything that needs to be done to get pages into the index. Today I had another go at it, and came up with a scheme that seems to be a good foundation, if a bit rocky in places.
I've been managing the process of importing StackExchange posts with an increasingly hacky and annoying shell script. Today I built out what I hope will be a useful replacement, though I'm still not entirely happy with it.
I've built a data processing system for working with streams of pages; it seems like it's pretty solid now so let's take a look at how it works.
For a search engine to be of any use, it has to have a collection of data to search over. I want to start off with a reasonably large dataset so I can start getting a feel for what tuning is actually needed, but I don’t particularly want to spend a lot of time in the beginning building my own crawler and dealing with all of the vagarities of HTML out in the wild, so I want to start with a large,…