marginalia.nu RSS feed
I currently have 17 GB free space on my root drive, out of 0.47 TB usable space, that’s 3% free space. I also have an additional harddrive installed, 12 TB, it has 140 GB free, which may seem a lot but it’s actually only about 1%. Your harddrive is probably pretty full too. Maybe not as crammed, but odds are it’s probably closer to full than empty. I ran a poll on mastodon and…
There’s been some changes to Marginalia Search: Migrated to systemd from docker New unranked query endpoint Wide domains got their own index for faster crawls In brief, this has removed a lot of operational headaches, reduced expensive queries leaving more computational power for the rest, and cut crawl times in half. Let’s tackle them in some order. We have docker at home The system…
I was on the bus on my way to an important business meeting about the exponential growth of my MRR, when suddenly I noticed some troublemakers were hassling an old lady. Their caps were on backwards, their clothes were baggy, and probably thought the grindset was about skateboarding. I had to act. I opened my macbook and showed Claude what was happening. “Claude, we gotta do…
You can learn new things. Pixel art, touch typing, 3d modelling, music, calligraphy, wood working, knitting, a language. Whatever is practical and calls to you, you can learn. In the long term, learning new things is fun and makes life richer in ways you can’t even imagine, and it’s a time investment that will pay dividends for life as these skills never really go away. There are even…
… optional, that is. I’ve been working on an NSFW filter for Marginalia Search, as that is something some people have asked for, primarily API consumers. The search engine has had some domain based filtering for a while, based on the UT1 lists, but that isn’t a very comprehensive approach. We’ll land on a single hidden layer neural network approach, implemented from…
I’ve had this sense that HN has gotten absolutely innundated with bots last few months. First most obvious giveaway is the frequency with which you see accounts posting brilliant insights like 13 60 well and t6ctctfuvuh7hguhuig8h88gd to f6gug7h8j8h6fzbuvubt GB I be cugttc fav uhz cb ibub8vgxgvzdrc to bubuvtxfh tf d xxx h z j gj uxomoxtububonjbk P.l.kvh cb hug tf 6 go k7gtcv8j9j7gimpiiuh7i…
This post is an elaboration on a comment I made on Hacker News recently, on a blog post that showed an increase in volume and decline in quality among the “Show HN” submissons. I don't actually mind AI-aided development, a tool is a tool and should be used if you find it useful, but I think the vibe coded Show HN projects are overall pretty boring. They generally don't have a lot of…
The Marginalia Search index has recently seen some design tweaks to make it perform better, primarily the introduction of postings list compression. Last year, the index was partially re-implemented with SSDs in mind . This was largely a success, but left some lingering issues with tail latencies that sometimes weren’t what they needed to be. To ensure predictable execution times, the query…
The Marginalia Search default ranking algorithm recently saw a fairly radical improvement, due to a new domain trust system that drastically reduces the number of content farm results, as long as there are human results it usually finds them across all the usual test queries. Recently fixing a few bugs that made the search engine work more correctly had the unexpected and undesired side-effect of…
Being clear about what your blog posts are about lets people who are interested in what you have to say find your writing more easily. The more paragraphs you spend getting to the point, the bigger the odds they’ll lose patience and click on something else before you’ve presented your thesis. When publishing articles online, no matter how obscure the subject matter, there is almost…
The search engine recently exposed a fair number of new tools for custom filtering to the API consumers and users of the new UI. This was originally going to be an incredibly chaotic update, both annuncing the new features and doing a technical walkthrough of the changes but that ambition turned out a bit too chaotic, so let’s split them up and focus on the feature announcement bit today.…
One of the big ambitions for the search engine this year has been to enable searching in more languages than English, and a pilot project for this has just been completed, allowing experimental support for German, French and Swedish. These changes are now live for testing, but with an extremely small corpus of documents. As the search engine has been up to this point built with English in mind,…
I’ve been using the CoPilot plugin for IntelliJ on and off for the last few years, and while initially pretty enthusiastic, I’ve come to first disable it and then delete it altogether along with JetBrains’ local AI-completions, and generally felt this has been an improvement in productivity and a reduction of frustration. CoPilot is pretty good at taking things that are already…
How you engage with the world changes how you experience the world, and how the world experiences you. A snarky and cynical approach, by its default assumption that things are shit, or if they are not yet shit will inevitably turn to; such an approach will give your world a malodorous brownish tint. Granted, snark gives you plausible deniability, a motte-and-bailey that protects you from direct…
The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes. The index is already fairly large, but can sometimes feel smaller than it…
As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change. This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and…
The most recent change to the search engine is a system that profiles websites based on their rendered DOM. The goal is identifying advertisements, trackers, nuisance popovers, and similar elements. The search engine already tries to do this, but isn’t very good at it because it’s only looking at static code. It turns out to be somewhat difficult to determine what a website that has…
You wake at 05:30 in the morning, feeling somewhat groggy. Instead of the alarm clock ringing like it normally does, a cheerful hologram appears: “Hi! I’m Kyle, your new alarm clock assistant!” You get dressed as Kyle explains all of the fantastic things he is capable of. You head over to the coffee machine. “Hey there! I’m Evan! Are you ready for AI in your coffee?…
The search engine has recently gained the ability to index the PDF file format. The change will deploy over a few months. Extracting text information from PDFs is a significantly bigger challenge than it might seem. The crux of the problem is that the file format isn’t a text format at all, but a graphical format. It doesn’t have text in the way you might think of it, but more of a…
Some time ago, I migrated the crawler off the okhttp library, to use Java’s builtin HTTP client. This seemed like a good idea at the time, but has led to a fair number of headaches. Java’s HttpClient has one damning flaw, and that that it doesn’t support socket timeouts. Its only supported timeout values are time to connect, and time until first byte of the response. This means…
A problem the search engine’s crawler has struggled with for some time is that it takes a fairly long time to finish up, usually spending several days wrapping up the final few domains. This has been actualized recently, since the migration to slop crawl data has dropped memory requirements of the crawler by something like 80%, and as such I’ve been able to increase the number of…
I’m happy and grateful to announce that the Marginalia Search project has been accepted for a second nlnet grant. All the details are not yet finalized, but tentatively the grant will go toward addressing most of the items in the project roadmap for 2025 . I’ve already been working full time on the project since summer 2023 , and this grant secures additional development time, and…
This text is satirical in nature. Tech news is abuzz with rude AI crawlers that forge their user-agent and ignore robots.txt . In my opinion, if this is all the AI startups can muster, they’re losing their touch. wget can do this. You need to up your game, get that crawler really rolling coal. Flagrant disregard for externalities is an important signal to the investors that your AI startup…
This update is a few days late, the canonical birth date of the project is Feb 26. It has been another year of Marginalia Search. The project is still ongoing, still my full time job, although the project is entering a somewhat more mature phase of development, most of the big pieces are in place and do a decent job at what they do. The roadmap for the project is available on GitHub.
Want a dumb phone but live somewhere that requires a smartphone app to park your car or pay your bills, or have friends that are only reachable via Messenger or some other app? Googles hate this one weird trick: Simply uninstall all irrelevant apps and disable everything that can’t be uninstalled, including chrome. This can be done by long pressing the icons in the launcher menu, and going…
A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days. Conditions being if the website has an RSS or Atom feed. The big crawl job takes about two months, and is run partition by partition, meaning there’s typically a slice of the index that is two months stale at any…
I recently put together a small library called Slop , for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before. This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely,…
This is a new major release of marginalia search, mostly leaning toward the technical side. Emphasis has been on ensuring the search engine has the technical capabilities to serve more types of queries, especially longer queries which it previously did not handle very well. Effort has also been put toward making sure it’s possible to install and run outside of docker. There is still some…
Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query. This is a write-up about implementing this change. This is going to be a relatively long post, as it represents about 4 months of work.…
I’ve been on a bit of a frustration-driven quest to solve a problem I frequently encounter working on the search engine, that is, reading data from disk. You’d think this would be a pretty basic thing, but doing this in a way that is half-way performant is surprisingly hard and requires avoiding basically all the high level tools at your disposal. There’s a common sentiment that…
As an experiment, I’ve reduced my coffee-intake to a single cup a day for about a week now. It’s made an enormous difference in sleep, mood and energy. I get tired at night, fall asleep quickly, and wake up refreshed. As mentioned previously in the context of morning sunlight exposure —another thing that’s aided my sleeping habits, but is somewhat less practical to sustain…
A year ago I walked out of the office for the last time. I handed in my corpo laptop, said some good-byes, and since then I have been my own boss. This first year has been funded by an NLnet grant, which I’m in the midst of wrapping up. As of now, the work is all done, the final request for payment has been sent. There’s a similar last-day-of-school levity to both these events.
The best description of my problem solving process is the Feynman algorithm, which is sometimes presented as a joke where the hidden subtext is “be smart”, but I disagree. The “algorithm” is a surprisingly lucid description of how thinking works in the context of hard problems where the answer can’t simply be looked up or trivially broken down, iterated upon in a…
I’ve experimentally replaced some of the Java implementations of quicksort and binary search with calls to C++ code, and saw huge benefits for the sorting code but the same or worse performance for binary search. The Marginalia Search engine is mainly written in Java, which is language that is good at many things, but not particularly pleasant to work with when it comes to low level systems…
A neat property of the parquet file format is that it’s designed with block I/O in mind, so that when you are interested in only parts of the contents of a file, it’s possible to some extent to only read that data. Many tools are aware of this property, and DuckDB is one of them. Depending on which circles you run in, a lesser known aspect of HTTP is range requests, where you specify…
Been working on improving Marginalia Search query parsing and understanding. This is going to be a pretty long update, as it’s a few months’ work. Apart from cleaning up the somewhat messy query parsing code, a problem I’m trying to address is that the search engine is currently only good at dealing with fairly focused queries, they don’t need to be short, but if you try to…
The project has been haunted by a mysterious bug since sometime February. It relates to the code that constructs the index, particularly the code that merges partial indices. In short the search engine constucts the reverse index through successive merging of smaller indices, which reduces the overall memory requirement. You can conceptualize the revese index itself as two files, one with offset…
It is a pretty well known fact that if you give kids one-on-one tutoring from experts, they outperform classroom-educated students by an absurd two standard deviations . The effect size is absolutely mind-boggling, and a damning brand of indictment against the entire modern education system. The problem is that this stuff clearly doesn’t scale, for two reasons; first of all there isn’t…
I set out a little over a week ago to add a service registry to Marginalia Search, primarily to reduce its dependence on docker. I would like it to be able to run on bare metal as well, which poses a problem since configuring the application manually is a bit of a headache with dozens of ports that need to be set up. It would also be desirable to be able to run multiple instances of important…
It’s been three years since the inception of Marginalia Search, then a dinky experiment to find where the heck the cool Internet has gone, now my full time job. While there’s always things that can be improved, it’s fair to say the search engine has never worked as well as it does right now. A great number of milestones have been reached, perhaps biggest of all the search engine…
One of the great joys of working on a search engine is that you get to reverse engineer SEO spam, and overall study how it evolves over time. I’ve been noticing the search engine spam strategy of adding ‘reddit’ to page titles for a few years now, but it feels like it’s been growing a lot recently. I don’t think it’s actually working , but it’s so cute…
I get significantly more work done when I unplug my computer from the Internet. It’s not that my productive output drops to zero when I’m plugged in, but more like 70%. Despite many of the tools that I use requiring a connection, and certainly the Internet containing a wealth of information that might expedite my work, these benefits are drastically outweighed by the wealth of…
You have a hobby you’ve been into for a decade or more. You like talking about your hobby, and your friends and family, after listening to these things for as long as you’ve been into them, maybe aren’t as excited to always hear about it as you are about discussing them, so in an act of compassion you create a youtube channel where you can monologue about your passion instead.
This is a major new release of the search engine software, corresponding to nearly four months of changes. In these months, the state of the code hasn’t been stable enough for a new release, but it’s now been brought to a stable point. Release Highlights: The installation procedure has been cleaned up. It’s now possible to run the search engine in a white label/bare-bones mode,…
This is a bit of a retrospective of every project I’ve worked on, as far as I remember them. I’ve tried to unearth any artifacts that remain. Far from everything is flattering and resounding success, but then again, maybe that’s good. There are definitely patterns in the things that didn’t pan out. Earliest Traces I was definitely programming stuff, but I don’t think…
Marginalia Search very recently gained the ability to filter results by Autonomous System, not only searching by ASN but by the organization information for that AS. At a glance this seems like a somewhat frivolous feature, but it has interesting effects. Autonomous Systems are part of the Internet’s routing infrastructure. If your mental model of an IP number is that they are the phone…
A simple guide to reading in 9 simple steps Navigate to the desired article. Dismiss the GDPR banner It may seem safe to start reading, but you need to wait about 10 seconds as the various ad auctions resolve and scripts load in Wait while the article is populated with ads. While the article is in front of you, there is no point to starting to read yet, as the minute’s worth of layout shift…
The Marginalia Crawler has seen improvements! A long term problem with the crawler design is that if for whatever reason the crawler shuts down, then it needs to re-start fetching whatever domains it was currently traversing during the termination from zero. This isn’t fantastic, since not only does crawling a website take a fair bit of time, it’s a nuisance for the server admins to…
I’ve been working on getting anchor tag keywords into the search engine, basically using link texts to complement the keywords on a webpage. The problem I’m attempting to address is that many websites don’t really describe themselves particularly well. As Steve Ballmer’s stage performance once illustrated, merely repeating a word doesn’t on its own make what…
So a bit of an update on what I’ve been working on. This will be adapted into release notes in a while, but I haven’t quite wrapped a bow on the change set yet. Still, it has certainly been a few weeks. Didn’t quite land how busy I’ve been until I set down to draft this post. Them’s some changes, and I’m skipping a few to keep this meandering post at a sane…