Recent content on lost in tab
Reflecting on five-plus years of building Cirrus, Element 84’s data orchestration framework, three core lessons emerge that apply to software projects generally: follow the money to understand what drives development, expect the pendulum of overcorrection and iteration, and hold fast to an unchanging vision. Part 2 of 2.
Cirrus, Element 84’s open-source data orchestration framework, recently turned five years old — and we’ve rebuilt it at least three times along the way. This is a retrospective on where Cirrus came from, from its origins in NASA’s Cumulus through the STAC Workflows idea, the project CLI, the SWOOP detour, and finally the v1.0.0 release, plus a look at where it’s headed…
When configured equivalently, COG and Zarr store the exact same data bytes. So what actually distinguishes one raster format from another? Metadata. We get philosophical about what metadata even is, examine how TIFF and Zarr each represent it, tour the various ways geospatial coordinate information is encoded across formats, ask where metadata should live, and make the case that formal, versioned…
What happens if we question current assumptions around Cloud-Native Geospatial (CNG) and imagine what it could be if we were to redefine it with new technologies? The Coalesced Chunk Retrieval Protocol (CCRP) is one idea born of this thinking, aiming to help make CNG more efficient and easier for both data consumers and producers.
Back from the dead with a new look and a new domain!
Compression is a fundamental and increasingly critical part of modern data storage and processing. But compression is often naively miscategorized as a single processing step. A better understanding of the steps in compression pipelines, the priciples behind them, and how to measure their effectiveness will enable us to move beyond the default compression settings and realize gains from more…
Raster data formats are chunked internally, but the cloud has exposed these internal implementation details. We dig into the finer details of chunking to understand how chunking has become something tyrannical, dictating how we can and can’t access data efficiently. Part 2 of 2.
Raster data formats are chunked internally, but the cloud has exposed these internal implementation details. We dig into the finer details of chunking to understand how chunking has become something tyrannical, dictating how we can and can’t access data efficiently. Part 1 of 2.
We dig into the Zarr and COG raster formats to see if Zarr can and should now be a replacement for COG, and find a surprising answer.
QEMU Services? I’ve been working on an integration testing environment for a few interconnected services. Such a thing would be easy with containers, but I am modeling an environment of bare metal hosts running these services, leaving little choice but VMs. It’s been easy testing the VMs in isolation using QEMU, which by default with -nographic drops one into the console. But when one…
Introduction I do a lot of work with network devices, which are often headless and require the use of serial consoles for out-of-band access. My go-to on MacOS and Linux has been screen, but various little issues over the years have led me to consider other options. I tried minicom on the recommendation of a colleague, but that didn’t work for me either. I’ve struggled to pinpoint my…
As I mentioned in my first post, much of my resistance to blogging has been a blend of prefectionism and an abhorrence to any chance of having to deal with yet more infrastructure issue. That is, I deal with broken crap at my day job; I really don’t want to have to worry about that for my blog. In that vein, I pursued the simplest practical stack with the quickest startup time, ending up…
Hello. I am a software engineer from Portland hoping to spread some knowledge when I have something possibly interesting to share. This is a new thing for me, technical blogging, so it might take me a little time to get started, but I hope I can start to churn out a regular stream of meaningful content quickly. Why do this? I have increasingly been feeling the pressure to start a technical blog. I…