Dewey Dunnington
About a year and a half ago I wrote a post on “wrangling” and joining 130 million points using various open source tools . Shortly thereafter I joined Wherobots where I get to spend most of my time building SedonaDB , a tool purpose-built to make exactly that sort of wranging and joining effortless. Until today I had never checked back to the query that got me thinking about all of…
With apologies for skipping the last 18 days…it’s day 24 of the 30 day map challenege and I’m back! In my first post about SedonaDB I did a brief investigation into the least creatively named waterbodies in Nova Scotia based on the water segments data in Nova Scotia. Anecdotally I’m also aware that there are many “Long Lake"s and “Little Rivers” out there…
Day 6 of the 30 day map challenege is Dimensions, and when I read the prompt I immediately thought: I have to do something with M values. M values, you say? It’s true that these don’t come up as often as Z, but many spatial specifications out there allow combinations of XY, XYZ, XYM, or XYZM values. The M stands for “measure” because I think the original motivation was that…
It’s day 5…earth! I was particular excited about this day in the 30 day map challenege because earth is my thing…I did a M.Sc. in Geology and taught Geomorphology at Acadia University for several years. While my first thought was mapping bedrock or surficial geology in Nova Scotia, I was foiled by the fact that Nova Scotia DNR distributes its files as self-extracting executable…
Day two of the 30 day map challenege is lines! A few weeks ago I wrote a post about stream traversal with SedonaDB . For day two I’ll do this with the river system for Gaspereau Lake (starting at the mouth of the beautiful Gaspereau River!). For anybody not familiar with the Gaspereau River, it features excellent wineries and the best tubing this side of any line you draw. Let’s get to…
Day one of the 30 day map challenege is points! One of the datasets I use all the time in testing is named ns-water_point from GeoArrow Data . When I made this test data I didn’t think very hard about it…the Nova Scotia Geospatial Data Directory has a hydrological section and I spent the better part of a decade there working on lakes…it seemed like fun! Other than a few sanity…
I’m an obvious fan of GeoParquet and SedonaDB and it should be no surprise that I worked (still working!) pretty hard making sure SedonaDB could take advantage of all GeoParquet had to offer. This post talks about one of those things: lazy reads (or if you’re a databse nut, “pruning”). Basically, GeoParquet was designed to allow traditional GIS software to take advantage of…
Last week I had the pleasure of being a guest on a Cloud Native Geospatial webinar with my colleagues Jia Yu and Matt Forrest where we talked SedonaDB . In the webinar Matt demoed a fantastic blog post comparing a real-world workflow on SedonaDB, DuckDB, and PostGIS . In preparation, I came up with a few examples and only a small bit ended up making it to the live demo. Here’s the full…
Last Wednesday the Apache Sedona project announced SedonaDB . There’s also a great post on the Whereobots blog that has a bit more context, or if you’re like me and you just want to see the code you can do that too . I’ve been a sparse blog poster ever since I (1) got a job and (2) had kids, but if you’ve been vaguely following over the past few years you’ll notice…
One of the most fun parts of being involved in GeoParquet is that I get to hear about all the amazing tools that people are writing to scale geospatial analysis above and beyond what the stack I’m familiar with can handle. GeoPandas (Python) and sf (R) are great tools but the struggle to scale as you approach the available memory on your laptop. On the other hand, Arrow, DuckDB, Polars,…
There’s a great discussion going on in the GeoParquet repo about how exactly to split up GeoParquet datasets for optimal querying. This post is my adventure giving the partition strategies a shot for practicality. All things are possible with enough software engineering effort, of course, but the ability to get these things done today on a bigger-than-memory dataset is what I’m after…
A little over a two years ago I wrote about point-in-polygon joins in R using an example of parking violations in Philedelphia. This example has been written about a few times involving a GPU example , using it as example for tuning PostGIS , a GeoPandas version for comparison , and a version comparing ArcPro, QGIS, PostGIS, and Manifold . I don’t want to downplay the parking example because…
GDAL’s 3.6 release is a huge deal. As with every GDAL release, there are a lot of bugfixes and incremental updates that represent a huge amount of maintainer time and effort. The feature I’m going to sing the praises of, though, is a shiny new feature first proposed by Even Rouault in RFC 86 and whose implementation could make reading vector layers (i.e., sf::read_sf() /…
Another release of the wk package is upon us! This release is a mix of exciting new features and bugfixes for some corner cases I encountered when working on the geoarrow package. Let’s get to it! Sit back, relax, and enjoy the…well, let just do this thing. library ( wk ) packageVersion ( 'wk' ) ## [1] '0.9.0' Grids support (i.e., raster, sort of) A few months ago I spent some time…
The first post I wrote about GeoArrow builds on a lot of great work in the GeoArrow specifcation repo that’s accumulated a lot of really exciting discussion over the past year or so. For that post I prototyped a low-level implementation , an R implementation that uses it , and made a bunch of data available in that form for testing . There have been more great blog posts like the one by Kyle…
A new version of the geos package for R is upon us! Buckle up and, if you so choose, get some of the example data to follow along! library ( sf ) library ( geos ) curl :: curl_download ( 'https://github.com/paleolimbot/geoarrow-data/releases/download/v0.0.1/nshn_water_line.gpkg' , 'nshn_water_line.gpkg' ) This example data is ~800 MB to highlight some of the features of this release designed to…
This release of the s2 package for R is an exciting one! Most users will use s2 implicity through its use in sf ; however, the s2 package can be used on its own and these improvements will likely be available in future releases of sf. Most of the hours spent on this release went into a massive refactor of the initial implementation so that the internals are more useful elsewhere (e.g., maybe a…
There’s lots of cool chatter about the potential for Apache Arrow in geospatial and for geospatial in Apache Arrow. Some recent blog posts have touched on some of the opportunities that are unlocked by storing geospatial vector data in Parquet files (as opposed to something like a Shapefile or a GeoPackage), like the ability to read directly from JavaScript on the web , the ability to…
Inspired by Martin Fleischmann’s post on spaital joins using Dask and geopandas , I thought I’d give it a go in R! I’ve been playing with bigish point datasets for a while and this particular bit of profiling seemed like a good opportunity to illustrate a few things to think about when dealing with them in the r-spatial universe. Selfishly, I also want to see if I can make R beat…
It’s time for another release of the wk package! The wk package is a small R package that powers spatial data interchange in packages like geos , s2 , and some exciting other packages still in development. It provides the things that other packages need to be really good at what they do and work seamlessly with other spatial packages: vector geometry classes, generics, and a stable C ABI…
After spending a few kiddo nap times experimenting with low-level Apache Arrow representations of vectors in R , I remembered a lively GitHub discussion of Arrow representations of geometry in R . The arrow R package can already round-trip sf objects to and from feather and parquet formats, and the sfarrow package does so as well with a slightly different encoding. Why bother? Why do we need…
A new version of wk is fresh on CRAN! Version 0.5 introduces some new features to the framework, incorporates most of the functionality that was previously in the wkutils package, and fixes a number of bugs that popped up in the development of s2 and geos . To showcase some of the new features I’ll use the Vermont counties data set from the VT Open Geodata Portal . library ( wk ) library (…
Almost exactly a year ago I wrote a post on using R and sf to work with stream networks . The post was about low-level analysis with the Nova Scotia stream network to find upstream networks of various lakes so that I could approximate the catchments without a province-wide DEM analysis. If you read the post, you’ll realize that it’s not just you, it’s a tiny bit awkward to work…
About the time the COVID-19 pandemic began, I started getting interested in some low-level geometry programming in R. Around that time the vctrs package was starting to mature and dplyr version 1.0.0 had just been released. In particular, vctrs provided a template for how a minimal but carefully-designed nuts-and-bolts framework can inspire an extensible ecosystem of packages enabling dplyr to…
In some recent adventures parsing text embedded within binary files, I came across the need to correctly interpret input bytes from a file representing a character string. As a developer, you want to write software that protects users from ever having to deal with an encoding issue! I do a fair amount of (and like the challenge of) parsing old file formats that assumed system encoding, and in some…
Avid readers of this blog, if there are any, would have figured out that I really love low-level data IO and hacking proprietary file formats so that I can use the data in the open-source (mostly) software that I know and love. At my new position I was assigned to attend Seabird University , an excellent six-session training course on how to acquire and process oceanographic data using Seabird…
Today the pivotal and final chapter of my Ph.D. thesis was published in Science of the Total Environment ! I’ve been excited about Pb deposition ever since my honours thesis , when I measured Pb (and other elements) in two lakes near the Nova Scotia-New Brunswick border. When I returned to Nova Scotia in 2015 for my M.Sc. at Acadia, I started collecting records of Pb deposition in dated lake…
I have to start with saying that there’s a lot of ways to write scientific articles with all of these tools. In particular, there’s some great guides from rOpenSci , packages with LaTeX templates for major publishers , and R packages to help with reproducible documents . For all of these, I think there’s some degree of familiarity with programming or LaTeX that’s needed to…
This is a preview of the wk package for R , which is currently undergoing the CRAN submission process. I’m just so excited about it that I can’t wait to start talking about what it can do! For now, you’ll have to install from GitHub to follow along: # remotes::install_github('paleolimbot/wk') library ( wk ) I’ll be using the North Carolina dataset from the sf package as…
If you can’t tell from my last two posts , I’m trying to delineate watersheds for about 650 lakes. We’ve gotten to the point where we can calculate an approximate catchment using the upstream network, and call the QGIS processing algorithms we need to delineate the catchment from the outlet. The problem is, we don’t know which outlet point will give us the catchment…
If you can’t tell from my last post , I’m trying to delineate watersheds for about 650 lakes. It’s not possible (er, I am unwilling) to do this by hand, so I need a way to do this programmatically. R is my weapon of choice, but unfortunately the watershed delineation tools are based in GRASS and SAGA . It’s possible to run GRASS and SAGA directly (via rgrass7 , and RSAGA ),…
I have a paper in the works that involves data for a few hundred lakes in Nova Scotia, for which I need catchment parameters. That’s a tall order! Automated catchment extraction is possible, but it helps to have a ballpark area to work with. Actually, it’s mission critical to have a ballpark area to work with…650 is just too many lakes to catch all the errors by hand. One of the…
This week in random R tutorials…how to program a deck of cards as a vctrs class, using R6 to keep track of multiple player hands! It was inspired by the deck of cards tutorial in Hands On Programming with R, except this version is more about creating interfaces than about learning R programming. It was also inspired by a game of cribbage with my wife, in which there were many exclamations…
Before we start, I have to warn you that this is probably the nerdiest thing you’re going to do all week. In fact, the only thing that is nerdier than you reading this is me writing it. I got started on GEOS in Rcpp because I wanted to do some complex vector processing, and the overhead of converting to and from sf was inefficient (in my case, I’d be literally paying for the…
Ever since I got a primer on Bayesian statistics from Aaron McNeil in my Analysis of Biological Data course at Dalhousie, I’ve been Bayesian-curious. As an avid R user, the way to do Bayesian statistics (as far as I can tell) is Stan , “a state-of-the-art platform for statistical modeling and high-performance statistical computation”. Other packages like brms build on Stan for…