For those who came from reddit and are not familiar with tantivy. tantivy is a search engine library for Rust. It is strongly inspired by lucene. This blog post might interest three type of readers. people interested in tantivy : You’ll learn how tantivy uses SIMD instructions to decode posting lists, and what happens on platform where the relevant instruction set is not available. rustaceans who…
For those who came from reddit and are not familiar with tantivy. tantivy is a search engine library for Rust. It is strongly inspired by lucene. Things tantivy does differently Some developers have a strange aversion to reading other project source code. I am not sure if this is the sequel of the school system, where this would be called cheating, or if it is a side-effect of people taking pride…
Of storing files in url minifiers Today I had an epiphany while staring at a very long url. I thought: “Url minifiers are really nice to store all of this data for free”. And then it stroke me… One can really store 4KB of arbitrary data with a url minifier system and share it for free. Storing files as a programming puzzle So here is a programming puzzle. You are given a service that has two…
Family feud meets Big Data When I was working at Exalead, I had the chance to have access to a 16 billions pages search engine to play with. During a hackathon, I plugged together Exalead’s search engine with a nifty python package called pattern , and a word cloud generator. Pattern allows you to define phrase patterns and extract the text matching a specific placeholders. I packaged it with a…
This post is the second post of a series describing the inner workings of a rust search engine library called tantivy . Foreword In my last blog post , I talked about the data-structures that are used in a tantivy index, but I did not explain how indexes are actually built. In other words, how do you get from a file containing documents (possibly too big to fit in RAM) to the index described in my…
Foreword. Search. Rust. I have been working more or less with search engines since 2010. Since then, I entertained the idea to try and code my own search engine. I ended up never starting this project, but accumulated more and more information over the year about how to implement a search engine, mostly by learning from coworkers, going through Lucene’s code, and reading academic papers and blogs.…
Caret Awareness : A neat feature for autocomplete Around 8 years ago, I read the description of a nice UI improvement to the traditional autocomplete search box. I cannot recall the name the author used for the feature, but I like to call it caret awareness . (caret is just another fancy name for text cursor). Here is the problem it was addressing. When I search for something, and the results do…
A nice little puzzle So my next post was supposed to be about an extension to levenshtein automata, but I had some good reason to want to delay that for a bit. Anyway stay tuned ! In the meanwhile, as I was talking about dices at work, I thought about a very cool puzzle. He it goes : ** Find all ways to relabel the faces of two dices with non-negative integers in such a way that if you roll them,…
Thanks to Ken Hironaka for kindly taking a lot of time to read and fix countless errors in this blog post! Back to Tokyo It’s been such a long time since my last post, and so much have happened. I moved to Tokyo in November 2014 and started working for Indeed Japan. I’m still kind of foreign to the dev community in Japan, so if you are also in Tokyo and you have some good tips about tech/startup…
Fattable Quite recently, I released under MIT license a javascript library to display large tables called fattable . The project got an unexpected amount of good publicity, got many tweets and as of this day 270 github stars , which is very rewarding ! Everything started with a problem we needed to address at Dataiku : our product gives datascientists a nice view of their dataset as they go…
What’s transient anyway ? Java programmers are probably familiar with the concept of transient as it is a keyword in this language. By marking an object property as transient, you tell Java that this property should be skipped in serialization. While this kind of functionality should arguably not be part of a programming language, but live in its standard library (as a decorator maybe), last week…
New job, new life I changed job ! I recently joined Dataiku . We’re creating the perfect Data Science Platform. And so far, it has been pretty awesome… By the way we are still recruiting, so if you are looking for a job in a top notch tech startup in Paris, drop me an email : paul.masurel at dataiku.com. Back to today’s subject. Last week I was discussing with a colleague at work about the painful…
Internet is full of solution for the rubik’s cube. However, it is seldom described how these solutions were discovered. In this post I’ll try to detail how one can solve the rubik’s cube from scratch. **Disclaimer : ** The solution presented here is by no mean the fastest… It is actually very long to solve the Rubix Cube using this algorithm. It is just the one I came up with, so I guess it is…
Memory my friend ! Nowadays RAM is so cheap, you might be tempted to just rely on his database being in RAM to get the wanted performance. Disk is just there for persistence. Many people talk on the web about their production setup bein in TmpFs, or using the RAMDirectory. But isn’t your OS supposed to make sure that the stuff your accessing is page cache? Let’s see how we can measure how much of…
A post about Solr. This post is about the innerworkings of one of the two most popular open source search engines : Solr . I noticed that many questions (one or two everyday) on solr-user’s mailing list were about Solr’s collapsing functionality. I thought it would be a good idea to explain how Solr’s collapsing is working. Because its documentation is very sparse, and because a search engine is…
E-Commerce (sometimes) doing it wrong Most e-commerce websites are offering you to sort your search results by customer ratings… and quite a lot are doing it wrong. Let’s assume here I’m looking for a book about CSS. I want to get the best book money can buy, so I will definitely hit the sort by rating button. The website is offering two options book A : 1 rating of 5. Average rating of 5. book B…
Demo Wiggle Stereoscopy Wiggle stereoscopy is probably the cheapest trick to give your brain a 3D feeling. While most technics try to take advantage of binocular vision by giving your left eye and your right eye different pictures, wiggle stereoscopy is just about looping on a couple of pictures with a slight shift of point of view. Your brain will interpret the parallax and give you a sense of…
Comparison sort complexity In my first post I dropped a line about 525 being the theoretical minimal number of comparison required to sort a list of 100 elements without explaining it. I will do so here, and show how the same thought process can help solving the 12 pearls puzzle. Let’s got through a thought experiment . Let’s imagine you are playing a game with a friend. You are locked in a room…
What’s lazy evaluation about ? Some functional programming languages (like Haskell) offers a functionality called lazy evaluation by default. It consists of defering evaluation of functions to the moment their results are actually used. Instead of results, everything works as if your function call are returning the recipe to compute the actual result. In python, it is actually pretty…