RPT refers to either SpatialRecursivePrefixTreeFieldType (aka simply RPT) and an extended version: RptWithGeometrySpatialField (aka RPT with Geometry). RPT offers several functional improvements over LatLonPointSpatialField:
-
Non-geodetic – geo=false general x & y (not latitude and longitude)
-
Query by polygons and other complex shapes, in addition to circles & rectangles
-
Ability to index non-point shapes (e.g. polygons) as well as points – see RptWithGeometrySpatialField
-
Heatmap grid faceting
RPT shares various features in common with LatLonPointSpatialField. Some are listed here:
-
Latitude/Longitude indexed point data; possibly multi-valued
-
Fast filtering with
geofilt,bboxfilters, and range query syntax (dateline crossing is supported) -
Sort/boost via
geodist -
Well-Known-Text (WKT) shape syntax (required for specifying polygons & other complex shapes), and GeoJSON too. In addition to indexing and searching, this works with the
wt=geojson(GeoJSON Solr response-writer) and[geo f=myfield](geo Solr document-transformer).
Schema Configuration
To use RPT, the field type must be registered and configured in schema.xml. There are many options for this field type.
| Setting | Description |
|---|---|
name |
The name of the field type. |
class |
This should be |
spatialContextFactory |
This is a Java class name to an internal extension point governing support for shape definitions & parsing. If you require polygon support, set this to |
geo |
If true, the default, latitude and longitude coordinates will be used and the mathematical model will generally be a sphere. If false, the coordinates will be generic X & Y on a 2D plane using Euclidean/Cartesian geometry. |
format |
Defines the shape syntax/format to be used. Defaults to |
distanceUnits |
This is used to specify the units for distance measurements used throughout the use of this field. This can be
|
distErrPct |
Defines the default precision of non-point shapes (both index & query), as a fraction between 0.0 (fully precise) to 0.5. The closer this number is to zero, the more accurate the shape will be. However, more precise indexed shapes use more disk space and take longer to index. Bigger distErrPct values will make queries faster but less accurate. At query time this can be overridden in the query syntax, such as to 0.0 so as to not approximate the search shape. The default for the RPT field is 0.025. Note: For RPTWithGeometrySpatialField (see below), there’s always complete accuracy with the serialized geometry and so this doesn’t control accuracy so much as it controls the trade-off of how big the index should be. distErrPct defaults to 0.15 for that field. |
maxDistErr |
Defines the highest level of detail required for indexed data. If left blank, the default is one meter – just a bit less than 0.000009 degrees. This setting is used internally to compute an appropriate maxLevels (see below). |
worldBounds |
Defines the valid numerical ranges for x and y, in the format of |
distCalculator |
Defines the distance calculation algorithm. If |
prefixTree |
Defines the spatial grid implementation. Since a PrefixTree (such as RecursivePrefixTree) maps the world as a grid, each grid cell is decomposed to another set of grid cells at the next level. If |
maxLevels |
Sets the maximum grid depth for indexed data. Instead, it’s usually more intuitive to compute an appropriate maxLevels by specifying |
And there are others: normWrapLongitude , datelineRule, validationRule, autoIndex, allowMultiOverlap, precisionModel. For further info, see notes below about spatialContextFactory implementations referenced above, especially the link to the JTS based one.
JTS and Polygons
As indicated above, spatialContextFactory must be set to JTS for polygon support, including multi-polygon.
All other shapes, including even line-strings, are supported without JTS. JTS stands for JTS Topology Suite, which does not come with Solr due to its LGPL license. You must download it (a JAR file) and put that in a special location internal to Solr: SOLR_INSTALL/server/solr-webapp/webapp/WEB-INF/lib/. You can readily download it here: https://repo1.maven.org/maven2/com/vividsolutions/jts-core/. It will not work if placed in other more typical Solr lib directories, unfortunately.
When activated, there are additional configuration attributes available; see org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory for the Javadocs, and remember to look at the superclass’s options in as well. One option in particular you should most likely enable is autoIndex (i.e., use JTS’s PreparedGeometry) as it’s been shown to be a major performance boost for non-trivial polygons.
<fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
spatialContextFactory="org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory"
autoIndex="true"
validationRule="repairBuffer0"
distErrPct="0.025"
maxDistErr="0.001"
distanceUnits="kilometers" />
Once the field type has been defined, define a field that uses it.
Here’s an example polygon query for a field "geo" that can be either solr.SpatialRecursivePrefixTreeFieldType or RptWithGeometrySpatialField:
&q=*:*&fq={!field f=geo}Intersects(POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30)))
Inside the parenthesis following the search predicate is the shape definition. The format of that shape is governed by the format attribute on the field type, defaulting to WKT. If you prefer GeoJSON, you can specify that instead.
RptWithGeometrySpatialField
The RptWithGeometrySpatialField field type is a derivative of SpatialRecursivePrefixTreeFieldType that also stores the original geometry internally in Lucene DocValues, which it uses to achieve accurate search. It can also be used for indexed point fields. The Intersects predicate (the default) is particularly fast, since many search results can be returned as an accurate hit without requiring a geometry check. This field type is configured just like RPT except that the default distErrPct is 0.15 (higher than 0.025) because the grid squares are purely for performance and not to fundamentally represent the shape.
An optional in-memory cache can be defined in solrconfig.xml, which should be done when the data tends to have shapes with many vertices. Assuming you name your field "geom", you can configure an optional cache in solrconfig.xml by adding the following – notice the suffix of the cache name:
<cache name="perSegSpatialFieldCache_geom"
class="solr.LRUCache"
size="256"
initialSize="0"
autowarmCount="100%"
regenerator="solr.NoOpRegenerator"/>
When using this field type, you will likely not want to mark the field as stored because it’s redundant with the DocValues data and surely larger because of the formatting (be it WKT or GeoJSON). To retrieve the spatial data in search results from DocValues, use the [geo] transformer — Transforming Result Documents.
Heatmap Faceting
The RPT field supports generating a 2D grid of facet counts for documents having spatial data in each grid cell. For high-detail grids, this can be used to plot points, and for lesser detail it can be used for heatmap generation. The grid cells are determined at index-time based on RPT’s configuration. At facet counting time, the indexed cells in the region of interest are traversed and a grid of counters corresponding to each cell are incremented. Solr can return the data in a straight-forward 2D array of integers or in a PNG which compresses better for larger data sets but must be decoded.
The heatmap feature is accessed from Solr’s faceting feature. As a part of faceting, it supports the key local parameter as well as excluding tagged filter queries, just like other types of faceting do. This allows multiple heatmaps to be returned on the same field with different filters.
| Parameter | Description |
|---|---|
facet |
Set to |
facet.heatmap |
The field name of type RPT |
facet.heatmap.geom |
The region to compute the heatmap on, specified using the rectangle-range syntax or WKT. It defaults to the world. ex: |
facet.heatmap.gridLevel |
A specific grid level, which determines how big each grid cell is. Defaults to being computed via distErrPct (or distErr) |
facet.heatmap.distErrPct |
A fraction of the size of geom used to compute gridLevel. Defaults to 0.15. It’s computed the same as a similarly named parameter for RPT. |
facet.heatmap.distErr |
A cell error distance used to pick the grid level indirectly. It’s computed the same as a similarly named parameter for RPT. |
facet.heatmap.format |
The format, either |
|
You’ll experiment with different distErrPct values (probably 0.10 - 0.20) with various input geometries till the default size is what you’re looking for. The specific details of how it’s computed isn’t important. For high-detail grids used in point-plotting (loosely one cell per pixel), set distErr to be the number of decimal-degrees of several pixels or so of the map being displayed. Also, you probably don’t want to use a geohash based grid because the cell orientation between grid levels flip-flops between being square and rectangle. Quad is consistent and has more levels, albeit at the expense of a larger index. |
Here’s some sample output in JSON (with "…" inserted for brevity):
{gridLevel=6,columns=64,rows=64,minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0,
counts_ints2D=[[0, 0, 2, 1, ....],[1, 1, 3, 2, ...],...]}
The output shows the gridLevel which is interesting since it’s often computed from other parameters. If an interface being developed allows an explicit resolution increase/decrease feature then subsequent requests can specify the gridLevel explicitly.
The minX, maxX, minY, maxY reports the region where the counts are. This is the minimally enclosing bounding rectangle of the input geom at the target grid level. This may wrap the dateline. The columns and rows values are how many columns and rows that the output rectangle is to be divided by evenly. Note: Don’t divide an on-screen projected map rectangle evenly to plot these rectangles/points since the cell data is in the coordinate space of decimal degrees if geo=true or whatever units were given if geo=false. This could be arranged to be the same as an on-screen map but won’t necessarily be.
The counts_ints2D key has a 2D array of integers. The initial outer level is in row order (top-down), then the inner arrays are the columns (left-right). If any array would be all zeros, a null is returned instead for efficiency reasons. The entire value is null if there is no matching spatial data.
If format=png then the output key is counts_png. It’s a base-64 encoded string of a 4-byte PNG. The PNG logically holds exactly the same data that the ints2D format does. Note that the alpha channel byte is flipped to make it easier to view the PNG for diagnostic purposes, since otherwise counts would have to exceed 2^24 before it becomes non-opague. Thus counts greater than this value will become opaque.