Dormant Last read · last published · next check
Read 4 days ago and current, but nothing has been published for 5 years.
The open-source software movement is clearly a powerful phenomenon. A diverse (in time, geography, interests, gender ( hmm not really, not yet, hrmph ), race, skills, use-cases, age, corporate employer, motivation, IDEs (or, Emacs (with all of its recursive parens)), operating system, ...) group of passionate developers work together, using surprisingly primitive digital tooling and asynchronous…
Almost a decade ago, I started running Lucene's nightly benchmarks , and have been trying with mixed success to keep them running every night, through the numerous amazing changes relentlessly developed by the passionate Lucene community. The benchmarks run on the tip of Lucene's mainline branch each night, which is understandably a volatile and high velocity code base. Sure, Lucene's wonderful…
Apache Lucene is a wonderfully concurrent pure Java search engine, easily able to saturate the available CPU or IO resources on your server, if you ask it to. The concurrency model for a "typical" Lucene application is one thread per query at search time, but did you know Lucene can also execute a single query concurrently using multiple threads to greatly reduce how long your slowest queries…
[ TL;DR : Apache Lucene 6.0 quietly introduced a powerful new feature called near-real-time (NRT) segment replication , for efficiently and reliably replicating indices from one server to another, and taking advantage of ever faster and cheaper local area networking technologies. Neither of the popular search servers ( Elasticsearch , Solr ) are using it yet, but it should bring a big increase in…
Long ago, Lucene could only use a single thread to write new segments to disk. The actual indexing of documents, which is the costly process of inverting incoming documents into in-memory segment data structures, could run with multiple threads, but back then, the process of writing those in-memory indices to Lucene segments was single threaded. We fixed that, more than 6 years ago now , yielding…
The Apache Lucene project will likely release its next major release, 7.0, in a few months ! Remember that Lucene developers generally try hard to backport new features for the next non-major (feature) release, and the upcoming 6.5 already has many great changes , so a new major release is exciting because it means the 7.0-only features, which I now describe, are the particularly big ones that we…
A few years ago I first built and released Jirasearch as a fun dog-food test case for the thin-wrapper Lucene server , to expose a powerful search UI over our Jira issues. This is a great showcase of a number of Lucene's important features: Using block join queries to model parent (the original Jira issue) and children (each comment) documents. This basic relational structure is also common in…
Some of you have noticed that I'm not writing much in this blog lately. But fear not: exciting changes are still happening in Lucene , and I am still writing about them! It's just that most of what I write is now appearing at either the Elastic blogs or on my Google+ feed , so please head over to those two sources to keep reading about the fun changes in Apache Lucene and Elasticsearch .
At long last, after a strong series of 4.x feature releases, most recently 4.10.2 , we are finally working towards another major Apache Lucene release! There are no promises for the exact timing (it's done when it's done!), but we already have a volunteer release manager (thank you Anshum!). A major release in Lucene means all deprecated APIs (as of 4.10.x) are dropped, support for 3.x indices is…
For some reason having to do with the medieval French , the scoring system for tennis is very strange . In actuality, the game is easy to explain: to win, you must score at least 4 points and win by at least 2. Yet in practice, you are supposed to use strange labels like "love" (0 points), "15" (1 point), "30" (2 points), "40" (3 points), "deuce" (3 or more points each, and the players are tied),…
The simplest Apache Lucene query, TermQuery , matches any document that contains the specified term, regardless of where the term occurs inside each document. Using BooleanQuery you can combine multiple TermQuery s, with full control over which terms are optional ( SHOULD ) and which are required ( MUST ) or required not to be present ( MUST_NOT ), but still the matching ignores the relative…
Most search applications using Apache Lucene assign a unique id, or primary key, to each indexed document. While Lucene itself does not require this (it could care less!), the application usually needs it to later replace, delete or retrieve that one document by its external id. Most servers built on top of Lucene, such as Elasticsearch and Solr , require a unique id and can auto-generate one if…
One of Lucene's useful transactional features is index durability which ensures that, once you successfully call IndexWriter.commit , even if the OS or JVM crashes or power is lost, or you kill -KILL your JVM process, after rebooting, the index will be intact (not corrupt) and will reflect the last successful commit before the crash. Of course, this only works if your hardware is healthy and your…
You may remember my first blog post describing how the Lucene developers eat our own dog food by using a Lucene search application to find our Jira issues . That application has become a powerful showcase of a number of modern Lucene features such as drill sideways and dynamic range faceting, a new suggester based on infix matches , postings highlighter , block-join queries so you can jump to a…
Lucene's suggest module offers a number of fun auto-suggest implementations to give a user live search suggestions as they type each character into a search box. For example, WFSTCompletionLookup compiles all suggestions and their weights into a compact Finite State Transducer , enabling fast prefix lookup for basic suggestions. AnalyzingSuggester improves on this by using an Analyzer to normalize…
There have been several recent, quiet improvements to Lucene that, taken together, have made it surprisingly simple to add geospatial distance faceting to any Lucene search application, for example: < 1 km (147) < 2 km (579) < 5 km (2775) Such distance facets, which allow the user to quickly filter their search results to those that are close to their location, has become especially important…
In Lucene's facet module we recently added support for dynamic range faceting , to show how many hits match each of a dynamic set of ranges. For example, the Updated drill-down in the Lucene/Solr issue search application uses range facets. Another example is distance facets ( To implement range faceting, for each hit, we first calculate the value (the distance, the age, the price) to be…
IP cameras have come a long ways, and recently I upgraded some old cameras to these new Lorex cameras (model LNB2151/LNB2153) and I'm very impressed. These cameras record 1080p wide-angle video at 30 frames per second, use power over ethernet (PoE) , can see when it's dark using builtin infrared LED s and are weather-proof. The video quality is impressive and they are surprisingly inexpensive. The…
Sometimes you need to play sounds or music (digitized samples) from Python, which really ought to be a simple task. Yet it took me a little while to work out, and the resulting source code is quite simple, so I figured I'd share it here in case anybody else is struggling with it. The Python wiki lists quite a few packages for working with audio, but most of them are overkill for basic audio…
Last year, Han Jiang's Google Summer of Code project was a big success : he created a new (now, default) postings format for substantially faster searches, along with smaller indices. This summer, Han was at it again, with a new Google Summer of Code project with Lucene: he created a new terms dictionary holding all terms and their metadata in memory as an FST. In fact, he created two new terms…
Three exciting Lucene features in one day Yesterday was a productive day: suddenly, there are three exciting new features coming to Lucene. Expressions module The first feature, committed yesterday, is the new expressions module . This allows you to define a dynamic field for sorting, using an arbitrary String expression. There is builtin support for parsing JavaScript , but the parser is…
Lucene now has a nice set of suggesters that use an analyzer to tokenize the suggestions: AnalyzingSuggester , FuzzySuggester and AnalyzingInfixSuggester . Using an analyzer is powerful because it lets you customize exactly how suggestions are matched: you can normalize case, apply stemming, match across different synonym forms, etc. One of the most common things you'll do with your analyzer is to…
It's been almost two years since I originally factored out the fast and accurate Compact Language Detector from the Chromium project , and the effort was clearly worthwhile: the project is popular and others have created additional bindings for languages including at least Perl, Ruby, R, JavaScript, PHP and C#/.NET. Eric Fischer used CLD to create the colorful Twitter language map , and since then…
I recently described the new lucene-c-boost github project , which provides amazing speedups (up to 7.8X faster) for common Lucene query types using specialized C++ implementations via JNI. The code works with a stock Lucene 4.3.0 JAR and default codec, and has a trivial API: just call NativeSearch.search instead of IndexSearcher.search . Now, a quick update: I've optimized PhraseQuery now as…
Suggest, sometimes called auto-suggest, type-ahead search or auto-complete, is now an essential search feature ever since Google added it almost 5 years ago . Lucene has a number of implementations; I previously described AnalyzingSuggester . Since then, FuzzySuggester was also added, which extends AnalyzingSuggester by also accepting mis-spelled inputs. Here I describe our newest suggester:…
