My projects, my thoughts and other things.
In the part 2 and part 3 , we implemented the way to store the indexed data and to split them in small enough pieces for the search engine to remain efficient, due to the encryption bottleneck. Now, it's time to implement read access and search functionality for our indexed entries. Query Definition In order to execute a search, the user first needs to define its query. Considering the…
In the previous article , we implemented the core document structure and indexes for our search engine. Now, let's tackle how we'll make our search engine scalable through sharding and reliable through transactions. Shard Definition Here's how our sharding architecture organizes data across multiple shards: Manifest +--------+ | shards | +--------+ | +----------+-----------+ v v v…
In the previous article , I explained how we'll write on disk and how we'll implement an abstraction so that it works on any device as well as in the browser. Now, it's time to think about what we'll store and in what format. Defining A Document In the majority of time, the use of a search engine is coupled with a database, because it's rarely the search-engine…
In the previous article , I introduced what project we're going to address in the following weeks: how to build a cross-platform search engine with encryption capabilities. Today, we'll have a look at the first technical challenge: how to store things on disk. You might be thinking that we start with a simple topic, to warm up and get ready for the serious parts. That's both right…
Have you ever wondered how search engines work under the hood? I'm not talking about vector search or anything fancy, but just the good old search engines that could really find what we would look for. Even though we can already find some articles about how search engine work from a high level, it could be interesting to implement our own and add some constraints to make it fun (and maybe…
Monitoring metrics for a project can be challenging. Deciding what to track, determining the cardinality of each tag, naming the metrics, and choosing what should be split into a tag or kept as a metric are all complex tasks that require experience. But monitoring the run of your project is not the only thing we need to monitor; the development phase is also important. Code coverage is a good…
First, I have nothing against conventional commits. Enforcing conventions when working on a project or codebase is the best way to maintain a clean and understandable project. While it may take time for newcomers to join and adopt these conventions, once they do, it becomes easier for them to contribute. I have been accustomed to using conventional commits in most of my personal projects or…