...about maps, open data, Git, and other tech.
Whenever is my latest passion project: it’s a mappy web database of films, series, books and board games organised around when and where they are set: Paris in the 19th century, Canada in the 1970s, China in the Han Dynasty. You can play with it at whenever.world. It currently has: I had the idea a […]
I make maps. A lot of maps. Almost always of somewhere on Earth. It was time for a change. The challenge: a procedurally generated map of this alternative Earth, produced on the fly, vector tile by vector tile. An easier approach might be to, say, first generate some continents and coastlines, then erode some mountains […]
Most of the time, when you need to host vector tiles, you’ll use one of the two main (only?) commercial vector tile hosting services: Mapbox or Maptiler Cloud. But what if you don’t want to? Maybe you want to host tiles which don’t meet Mapbox’s 500KB-per-tile limit. Maybe you have some very specific requirement which […]
At FOSS4G Oceania 2019, I lamented the lack of free tools for collaboratively maintaining small datasets of locations, a common need in many community groups. So, this weekend I had a go at building one: TinyMap! A simple tool that lets you add and remove points with names and descriptions from any map you care […]
I recently re-engineered the data processing behind OpenTrees.org. It’s a website that lets you explore the combined open tree databases of 21 local councils around Australia (over 800,000!), with some pretty data visualisations. Working on this site has taught me a lot about processing data into vector tiles. Today’s lesson: “You might not need PostGIS”. […]
How to fit the pieces of the vector tile ecosystem together: map engine, styles and OpenStreetMap tiles schemas.
Self-doubt is awful, so this is for Future Steve: here are lots of things you did in 2015 that you can be proud of! Created opentransportdata.org: Swagger API documentation for PTV’s real-time API, as part of the VicTripathon hackfest. PTV’s documentation is a developer’s nightmare. My version has live testing of the API built in and […]
National Map is a pretty awesome place to find geospatial open data from all levels of Australian government. (Disclaimer: I work on it at NICTA). But thanks to some not-so-obvious features in TerriaJS, the software that drives it, you can actually create and share your own private version with your own map layers – without […]
Everyone loves hackathons. And almost as much, everyone loves asking “but what happens to the projects afterwards?” There’s more than one route to follow. I’d like to propose four standard recipes we can use to describe the prospects of each project. #1: Start-up The creators of the hack could form a business. The developers work […]
I try to convince government bodies, especially local councils, to publish more open data. It’s much easier when there is a concrete benefit to point to: if you publish your tree inventory, it could be joined up with all the other councils’ tree inventories, to make some kind of big tree-explorey interface thing. Introducing: opentrees.org. It’s fun! Click on […]
Cycletour.org is a tool for planning cycle tours in Australia, and particularly Victoria. I made it because Google Maps is virtually useless for this: poor coverage in the bush and inappropriate map styling make cycle tour planning a very frustrating experience. Let’s say we want to plan a trip from Warburton to Stratford, through the hills. This […]
Problem You want to do some visualisation magic in Tableau, but your spreadsheet looks like this: All those green columns are dependent variables: independent observations about one location defined by the white columns. Tableau would be so much happier if your spreadsheet looked like this: This is called “normalizing” the “cross-tab” format, or converting from […]
As a data guru in residence, I’m helping government bodies prioritise which datasets to release as open data. Sometimes people say “No one would ever find this data interesting, so why bother releasing it?” I think there are several distinct reasons why a given dataset might be worth releasing. Some datasets are valuable for several reasons […]
Cross-posted at Code for Australia. Last week, Code for Australia launched its first fellowship program, a four-month project where a civic-minded developer will try a new approach to helping government solve problems with their data. For the next few months, I’ll be the Data Guru in Residence, blogging mostly to http://melbdataguru.tumblr.com. The program got a brief mention […]
For such a simple device, Google’s Chromecast has created a surprisingly complex network of technology at my place. Google says that Chromecast works roughly like this: Actually it’s more complicated than that. My setup is about as simple as you can get (no NAS, no existing media servers, no Netflix or Hulu or Foxtel or anything), […]
If you’re making maps, you will probably need to know something about cartographic projections. Here’s the minimum. The globe is round, maps are flat. Each of the hundreds of different methods for converting from round to flat is a projection. When you have a latitude and longitude, you have unprojected coordinates. Anything you can do with these doesn’t […]
I help researchers make maps of their research. An archaeologist recently wanted to visualise the distribution of some iron-age artefacts around the Levant, based on a spreadsheet of thousands of rows. Each row represents one kind of artefact at a given site, such as “3 incised bangles, subtype I.b.iv, at Gath.” What are these maps […]
An editorial published in The Australian on the 21st of March set a new low standard in writing about conflicts between cyclists and cars. Prompted by video of a cyclist colliding with a taxi door, the editorial combined a strong anti-cyclist viewpoint (as it’s entitled to do in the opinion section) with some astonishing ignorance […]
Contributing to OpenStreetMap is diversely rewarding: you help other people, you make open data as a whole more viable, you learn a lot about the area you’re mapping, and it’s fun. But sometimes it’s just plain pragmatic. Last weekend, I organised a cycle tour from Bendigo to Avenel, via the O’Keefe Rail Trail, Lake Eppalock, […]
Credit:Tim Strater from Rotterdam, Nederland CC-BY-SA Of all the well-documented difficulties I’ve had working with Git over the years, a few conceptual difficulties really stand out. They’re quirks in the Git architecture that took me far too long to realise, far too long to believe, or far too long to really grasp. And maybe you […]
Salt, or SaltStack, is an automatic deployment tool for clusters of machines, in the same category as Chef, Puppet and Ansible. I’ve previously used Chef, found the experience frustrating, and thought I’d give Salt a go. The task: converting my tilemill-server bash scripts, which set up a complete TileMill server stack, with OpenStreetMap data, […]
Github, the online version control repository host for Git, recently added support for GeoJSON files. Sounds boring, right? It actually lets you do something very cool: build your own “dots on a map” website with virtually zero code. Here’s what you need to do. Get a Github repository if you don’t have one already. They’re […]
Trove is the National Library of Australia’s “discovery interface” – an amazing catalogue of books, newspapers, maps, music, journal articles etc. The Trove API is a special website that programs can talk to run queries, retrieve individual records etc. With some creative ideas and a bit of coding skill, you could make a pretty nifty […]
Here’s my method of keeping track of stuff to do. Other tools just didn’t work for me. Most to-do list tools make two assumptions, I think: Work arrives from somewhere (eg, a manager), and is readily broken up into pieces roughly 0.5-4 hours in size. Your primary responsibility is processing these tasks without gaps, maintaining […]
I created a basemap for http://cycletour.org with TileMill and OpenStreetMap. It looked…ok. But it felt like there was something missing. Terrain. Elevation. Hills. Mountains. I’d put all that in the too hard basket, until a quick google turned up two blog posts from MapBox, the wonderful people who make TileMill. “Working with terrain data” and “Using TileMill’s […]
Recently, I set up a server for a series of #datahack workshops. We used TileMill to make creative maps with OpenStreetMap and other available data. The major pieces required are: TileMill, which comes with its own installer, and is totally self-sufficient: web application server, Mapnik, etc. Postgres, the database which will hold the OSM data […]
I run lots of Linux servers. I create them, install some stuff, mess around with them, forget them, come back to them…and forget my credentials. My life used to look like this: $ ssh -i ~/steveko.pem steveb@115.146.83.268 Permission denied (publickey). $ ssh -i ~/steveko.pem sbennett@115.146.83.268 Permission denied (publickey). $ ssh -i ~/stevebennett.pem steveb@115.146.83.268 Permission denied…
A researcher at the Australian Research Centre in Sex, Health and Society (ARCSHS, pronounced “archers”) carries out surveys that are both: anonymous: personal details are not collected, and steps are taken to avoid being able to identify participants; and longitudinal: participants are contacted at intervals to answer similar questions. Questions often refer to previous answers given by the […]
I just received an interesting phone call, apparently from a group of Indian scammers. It went roughly like this. (Phrases in bold are things I jotted down during the call) Hello, I’m Caroline from the Computer Technical Department at Windows Best Help [or Windows Based Help, perhaps]. We’re calling to alert you that for the […]
Waterfall development still doesn’t work. Filling an institutional research data registry is still a hard slog. Omero is not just for optical microscopy. Spending money so universities can build tools that the private sector will soon provide for free ends badly. Research data tools that mimicking the design of laymen’s tools (“realestate.com.au for ecologists“) work […]
Here’s a formula for setting up a multi-instrument MyTardis installation. It describes one particular pattern, which has been implemented at RMIT’s Microscopy and Microanalysis Facility (RMMF), and is being implemented at Swinburne. For brevity, the many variations on this pattern are not discussed, and gross simplifications are made. This architecture is not optimal for your […]
Opscode Chef is a powerful tool for automating the configuration of new servers, and indeed, entire clouds, multi-server architectures etc. We now use it to deploy MyTardis. It’s quite daunting at first, so here’s a quick guide to the setup I use. You’ll probably need to refer to more complete tutorials to cover gaps (eg […]
Ok, now that I have your attention, let me start backpedalling. In the Australian e-Research sector, I’ve worked on problems of research data management (getting researchers’ data into some kind of database) and registering (recording the existence of that data somewhere else, like an institutional store, or Research Data Australia). There hasn’t been a huge […]
Git is the source code version control system that is rapidly becoming the standard for open source projects. It has a powerful distributed model which allows advanced users to do tricky things with branches, and rewriting history. What a pity that it’s so hard to learn, has such an unpleasant command line interface, and treats […]
I seem to have picked up the habit of using semantic keywords in my Google searches. I have no idea if Google actually supports any of them, but, if they don’t, they should! buy Example: buy usb extension cable Meaning: I’m specifically looking to purchase something, so don’t show me any site that doesn’t help […]
Consider an action like creating a new “space” in Confluence. Because of the visibility of this action, people want it restricted. Controlled. Managed. So Atlassian makes it only available to people with administrator rights. Which seems ok, until you realise that the workflow of the non-administrator ends up looking like this: …doing stuff… Decide to […]
So you’ve found the thing you want, and you already have both an eBay account and a PayPal account. You don’t want to do anything complicated – send this item to me, and charge my (already stored) card appropriately. Click Buy it Now Sign in Click Commit to Buy Click Pay Now Click Continue Sign […]
Here’s what I hate about recipes: They’re delivered as unstructured narratives. They mix identifying information (“carrots”) with process information (“thinly sliced”) They’re overly specific (3/4 of a teaspoon, does it matter?) They have too many ingredients, and you don’t know which ones you can leave out They always have one or two ingredients you wouldn’t […]
Ok, Gmail, you’re halfway there: you’ve got labels, and you can detect mailing lists (Google Groups ones, anyway). Now, take it a bit further: Automatically create a label for every group that I receive mail from. Don’t wait for me to do it. Make these labels more than just a label. Give them options like […]
swoopo, bidray, bidstick (bids tick, apparently), bidrivals and dozens of others are running what we’ll call “penny auctions“. Using bidrivals.com as the example, they all work on the following principals: There are consumer electronics for auction, usually at big discounts. It costs a certain amount to make a bid, regardless of whether that bid is […]
Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!