Data, Software, Systems
I started playing with Clojure in 2013 and was immediately impressed. It had the killer combination of immutable data structures , functional programming , an interactive REPL , and structural editing . I dabbled a bit, created a few demos, fell in love with the language, evangelized everywhere I could, then went back to …
There's a lot of well-deserved excitement around graphics processing units (GPU), not just in the stock market but the software world too. The success stories are too good to ignore. I've heard GPUs described as an "accelerant", and one might get the impression that sprinkling a little GPU on a …
Sooner or later, that application you're writing will need to be configured. At the very least, you'll need a way to adjust inputs without editing source code. Wouldn't it be nice to have a reasonable configuration system from the start? The best way to configure your app will depend on …
So you need a database for an application you're developing. You've looked around and decided that PostgreSQL fits the bill. Excellent choice! Now it's time to start coding. How do you get postgres running locally to devlop and test against it? The typical suggestion for many web application frameworks is …
In my last post I compared two approaches for calculating zonal statistics: A Python approach using the rasterstats library A SQL approach using PostGIS rasters. I came away happy that I could express zonal stats in SQL, but wasn't happy with the performance; an 87x slowdown compared to the equivalent …
Zonal statistics is a technique to summarize the values of a raster dataset overlapped by a set of vector geometries. The analysis can answer queries such as " Average elevation of each nation park " or " Maximum temperature by state ". My goal in this article is to demonstrate a PostGIS implementation of …
Working with geospatial vector data typically involves manipulating collections of features - points, lines and polygons with attributes. You might change their geometries, alter their properties or both. This is nothing new. Tools like this have been around since the first days of GIS. Notice the essential role of many of …
With the recent announcement that AWS Lambda now supports Python , I decided to take a look at using it for geospatial data processing. Previously, I had built queue-based systems with Celery that allow you to run discrete processing tasks in parallel on AWS infrastructure. Just start up as many workers …
Raster data coordinate handling with 6-element geotransforms is a pain. Use the affine Python library instead. The typical geospatial coordinate reference system is defined on a cartesian plane with the 0,0 origin in the bottom left and X and Y increasing as you go up and to the right …
This year I'm starting to delve into some electronics projects and hardware hacking. What follows is an account of my first end-to-end Raspberry Pi project. In terms of functionality, it doesn't do much at the moment - just reads from a photoresistor sensor and plots the light levels in the corner …
sthoijslkfjaslfjlasglaskglsdfjgdjsflkgjdsfl rglkje
The case for topology Simple feature representations of polygon geometries are ubiquitous due to their ease of use. Thinking of spatial features as having a single, independent geometry is easy and fits most use cases. But that ease of use disappears when we need to represent the topological relationship between …
Demonstrates the use of the SALib python module to sample and test the sensitivity of models As (geo)data scientists, we spend much of our time working with data models that try (with varying degrees of success) to capture some essential truth about the world while still being as simple …
Simple leaftlet-based template for mapping tabular point data on a slippy map Anyone who's worked with spatial data and the web has run across the need to take some simple tabular data and put points on an interactive map. It's the fundamental " Hello World " of web mapping. Yet I always …
This article introduces a python module for summarizing geospatial raster datasets based on vector geometries (i.e. zonal statistics). A common task in many of my data workflows involves "zonal statistics"; summarizing raster data based on vector geometries. Despite many alternatives (starspan, the QGIS Zonal Statistics plugin, ArcPy and R …
We've begun to rely on the interactivity provided by UTFGrids in many of our recent web maps. (Quick recap: UTFGrids are "invisible" map tiles that allow direct interactivity with feature attributes without querying the server.) Earlier this year, I created the initial OpenLayers UTFGrid support and was glad to see …
Madrona : A software framework for effective place-based decision making My work at Ecotrust mainly revolves around creating web-based spatial analysis tools - software to bring data-driven science to the place-based descision making process. This began several years ago when I joined the MarineMap team. Since working with Ecotrust, we've taken the …
I just switched this blog from an ancient version of wordpress running on a VPS to a static-file jekyll bootstrap site (hosted by github ). Let me know if you experience any wierdness on the site or feeds. I've taken good measures to make sure links don't break (old URLS should …
python-mbtiles . Check it out. I've been working a bit with Tilemill lately and love the Carto css styling, iteractivity through UTFGrids and being able to export the whole deal as a single mbtiles sqlite database. But when it comes to working with the mbtiles databases, I've found both Tilestache and …
Ever try to figure out what the average aspect of an area is? i.e. What direction does this hillside face? Let's say we want to determine the average elevation of an area based on a raster DEM. Just take the arithmetic mean of all the elevation cells contained in …
A while back, the Development Seed team developed the UTFGrid spec to provide a standard, scalable way of encoding data for hundreds or thousands of features alongside your map tiles. The basics In more detail, the UTFGrids are invisible "ASCII Art" and attribute data embedded in json. They sit "behind …
For all the benefits of KML, it is decidedly a step backwards for handling large vector datasets. Most KML clients, including the cannonical Google Earth application, experience debilitating slow-down when viewing a couple dozen MB of vector data - datasets that I could easily open on a Pentium 4 in ArcView …
RJ Zimmer at American Surveyor magazine did what he described as a comparison of several free GIS application entitled " Something for Nothing " First of all, the title bugs me. The idea that the sole benefit of free software is simply cost savings is pretty naive. It disregards openness, community support …
When the MarineMap team started delving into the Google Earth plugin , it was apparent that it supported the display and rendering of KML files almost as well as the Google Earth desktop application. The missing piece of functionality was the nice tree-style legend that is provided with the desktop app …
For the last year or so, I've had the pleasure of working with the MarineMap Consortium . We just learned yesterday that the U.S. Institute for Environmental Conflict Resolution awarded MarineMap the “Innovation in Technology and Environmental Conflict Resolution”. x I joined the team after the launch of the South …
I don't know how I let this gem slip past my radar for so long. It was only via a post by Dr. JTS himself (aka Martin Davis) that I saw a screenshot of JTS TestBuilder and decided to check it out. I was actually just talking with someone about …
I've been playing around with some distributed version control systems (DVCS) to replace svn. First, the why : I'll leave the details up to Joel in his excellent HgInit tutorial . Its mercurial-specific but the general concepts apply to any DVCS. The takeaway message for any project with > 1 developer is this …
No, not lazy as in REST :-) ... Lazy as in " Lazy evaluation ": In computer programming, lazy evaluation is the technique of delaying a computation until the result is required. Take an example raster processing workflow to go from a bunch of tiled, latlong, GeoTiff digital elevation models to a single shaded …
I had previously bitched and moaned about the licensing restrictions on the TrainingPeaks WKO+ software. Truth be told, the reason I was so put off by their crappy licensing scheme was that my cycling training relied so heavily on their software. It was not perfect but it was the best …
Just thought I'd point out a great presentation about the "new" 9.3 geoprocessing (gp) python module from ESRI. Ghislain Prince and Elizabeth Flanary do a great job of introduction by examples. The latest gp module is much more pythonic and these examples show how to leverage that to its …
I've been using Peaksware's WKO+, a cycling and running training tool to manage data from heart rate monitors, GPS units, power meters, etc. Its a powerful tool with a clunky UI but I've gotten used to it. You pay $100 for a "personal" license. Not a big deal to me …
When I was setting up my linux system a few years ago, I did some research into filesystems and determined that the XFS file system , being particularly proficient in dealing with large files, would be ideal for my home directory. And it was. But the one factor I didn't consider …
Not sure why this didn't occur to me before I wrote that last post but I tried the "pythonic" version of the code under the IronPython 2.6 Beta 1 release and it works! lyr = Carto.LayerFileClass() lyr.Open('C:\\test.lyr') print lyr.Filename Works perfectly now. So IronPython …
Occasionally I find myself in the C#/.NET world in order to write code using ESRI ArcObjects. Today I was toying with the idea of automating the creation of ESRI Layer files (a file which defines the cartographic styling of a dataset). Of course they are in an undocumented binary …
Another disastrous consequence of inaccurate spatial information... Not only can you accidentally tag your neighbor as a criminal , now it appears that sloppy spatial data has lead to the wrong house getting demolished . I've asked it before but its worth repeating ... with all the recent advances in spatial data publishing …
Dealing with corrupted shapefiles can be a painful experience: programs crash for seemingly no reason, attribute tables get screwy, features get lost, queries results don't look right and ArcGIS processing tools fail with mysterious error codes: Never fear, OGR is here. The magic bullet for fixing corrupted shapefiles is, 90 …
Real-time spatial application developers take note... I've been following the Tour of California this week (looking forward to the Solvang Time Trial this Friday) and have been disappointed with the TV coverage on Versus. Its not that the coverage is bad, its just that long-distance endurance sports don't lend themselves …
Last time I posted on this blog, Hillary and Obama were still battling it out for the Democratic nomination. Now Barack Obama is our president with an uphill battle to save the economy. So yeah, it's been a while. I haven't been doing too much innovative Geo-related stuff lately, hence …
Radiohead realeased their video for House of Cards yesterday. Besides being a big radiohead fan, I was also loving the LIDAR technology behind the video. If you want to check it out yourself, there are code samples on the site as well as access to the raw data. The csv …
So, despite frustrations with getting submitted URLs to appear, Geospatial Reddit is still puttering along. Not exactly a vibrant community yet but there are currently 133 subscribers. If you're subscribed, take a minute to submit your favorite URLs. If you haven't subscribed, check it out . I thought 133 subscribers was …
For those of you that use EndNote to keep track of your bibliographies/references , there is an alternative. JabRef . I find the UI to be very intuitive and it has a range of customizable import/export formats. JabRef uses the BibTex format as it's native file format so, of course …
All the great GIS news/blog aggregators out there (planetgs, slashgeo, etc) are moderator driven - a few people act as the gatekeepers and inevitably have to decide what information is useful . This is not the ideal way to do things. There's a more democratic and distributed way to spread the …
Some folks have had trouble submitting links so I figured I should post a bit more detail on that. To get articles to show up on the geospatial reddit (not the main reddit), go to http://reddit.com/r/geospatial/submit or click the "Submit a Link" button on the …
Guides like Tomislav Hengl's Practical Guide to Geostatistical Mapping of Environmental Variables and Rossiter's Introduction to applied geostatistics do an excellent job of providing a grounded, relatively easy to understand, introduction to geostatical prediction and kriging. But if you're an experience learner (like me) you don't absorb the mathematics fully …
As a followup to my previous post on turning Ubuntu Gutsy into a GIS workstation , Here are the revised instructions for Ubuntu 8.04 (The Hardy Heron). Note that there are a few additonal apps and changes in here: Postgis Mapnik New version of QGIS installed via repository OpenStreetMap tools …
In commemoration of the OGC approval of KML as an open standard to share geographic content over the web, I'd like to share our recent "Hike of Doom #2" (kml provided by Mark Dotson). The first weekend to hit 90 degrees, my friends and I travel inland to dive and …
I love python for its beautiful code and practicality. But it's not going to win a pure speed race with most languages. Most people think of speed and ease-of-use as polar opposites - probably because they remember the pain of writing C. Cython tries to eliminate that duality and lets you …
Slashgeo pointed me to a very interesting set of projects - SpatiaLite and VirtualShape . They provide a spatial data engine for the sqlite database. Think of it as the PostGIS of SQLite. It looks like this extends sqlite's spatial capabilities far beyond the sqlite OGR driver . SpatiaLite provides many of the …
What an odd meme .. I don't know why but I expected some more interesting results. I guess the majority of the commands I use are pretty pedestrian. history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}' | sort - rn | head 163 vi 48 screen 29 python 28 …
I don't spend much time with web programming these days but I decided to give web.py (the minimalist python web framework) a shot and, while I was at it, try implementing a simple REST api. First of all, web.py is truly everything it claims to be - small, light …