Common wisdom is that databases should make decisions at the query planning stage and then hand over the plan to a dumb executor. This has several nice properties. Separation of concerns helps each tool focus on one thing and do it right: computing the cheapest possible plan in the planner, and squeezing the best efficiency out of the hardware in the executor. It’s easy to introspect how the…
I was recently chatting with a friend about database architecture, and why, for instance, OLTP databases and search engines are implemented very differently. This friend is familiar with computer science concepts, but not with database internals, so I tried to make the explanation as simple as I could. This blog is an expanded version of the discussion we had.
In a previous blog post, I explained how Lucene significantly improved query evaluation efficiency by migrating to a vectorized execution model, and described the algorithm that Lucene uses to evaluate conjunctive queries. In this blog post, I’ll describe the algorithm that Lucene uses to evaluate disjunctive queries, which is a bit more sophisticated.
Virtual function calls are quite expensive, which is why database systems have been looking into ways to avoid performing one or more virtual function calls per record when processing a query. Two main approaches have proved successful: compilation and vectorization. With compilation, the database engine dynamically generates code specifically for the query that needs to be processed. With…
Some time ago, I wrote that “if you do not configure an index sort on your Lucene indexes, you are missing search-time efficiency benefits that are almost certainly worth the (low) index-time overhead”.
In a previous post, I took a look at the Vespa vs. Elasticsearch benchmark that the Vespa people run. The results made me want to dig a little deeper to see how Vespa and Lucene/Elasticsearch differ implemetation-wise. I took advantage of Vespa being open-source under the Apache License 2.0 to take a look at the source code. I found some similarities and differences that I expected, but also some…
I was attending Berlin Buzzwords today and someone asked me about the Elasticsearch vs. Vespa comparison produced by the Vespa people, so I thought I’d publish my thoughts.
“Search Benchmark, the Game” is maintained at https://github.com/quickwit-oss/search-benchmark-game by the Tantivy folks and published at https://tantivy-search.github.io/bench/. I don’t know the full history behind this benchmark, GitHub says that this repository was started by Jason Wolfe in 2018 as a comparison between Lucene and (early versions of!) Tantivy before getting forked and more…
EDIT (2025-11-12): Thanks to learnings from the Tantivy project, as of version 10.3, Apache Lucene is now on par with Tantivy on most queries and even faster on some queries. This is likely a temporary win though, until the Tantivy project replicates performance improvements from the Lucene project, or even finds new opportunities for optimizations!