A not-for-profit research lab advancing how we access, understand, and act on data about our world.
Who's the acronym artist at NASA responsible for this tomfoolery? Yes, IMERG is constructed from I ntegrated M ulti-satellit E (!!!) R etrievals for G PM. Next level. NASA IMERG Early and Late are live IMERG combines observations from a constellation of satellites into a global precipitation analysis. We now have two live-updating IMERG datasets in the catalog: NASA IMERG Early is available about…
We want it all: low-latency updates completeness over time, variables, and vertical dimensions access speed for time-series and map-style reads But when your data is 14 PB and 1 billion GRIB messages, making that dream a reality takes some work. Virtual Icechunk Zarrs are a key complement to our materialized (rechunked) Zarrs. Together they get us one step closer to the catalog of our dreams. We…
Are we living inside a Bostrom-esque simulation? Does that simulation exist inside another simulation ? There's only one way to find out: let's add increasing layers of derivative abstraction on already-digital signals that inadequately capture what the sheeple call "reality." Yeah?! Yeah. If anyone has Morpheus' phone number let him know I'm ready to be picked up on the Nebuchadnezzar.…
Forecast production is a factory assembly line, a fulfillment center, and a delivery route all in one. Lewis Fry Richardson's Weather Forecasting Factory was not too far off. “Weather Forecasting Factory” by Stephen Conlin, 1986. Based on the description in Weather Prediction by Numerical Process, by L.F. Richardson, Cambridge University Press, 1922, and on advice from Prof. John Byrne, Trinity…
I was tempted, but alas Gefs (short for Gefsfry) did not make the cut for the name of my newborn son. He's here and everyone is healthy - so I'm getting back to it. Some catch-up: ECMWF AIFS ENS forecast The ensemble sibling to AIFS Single is now in the catalog. ECMWF AIFS ENS : 15-day global forecasts at 0.25°, 6-hourly steps, 51 members, updating every 6 hours. Archive starts 2025-07-02. Docs…
Listen, my second child is arriving any minute now. This transmission will be brief. I haven't decided if I will name him Poseidon, Gefs, or Turbulence. Zarr v3 deprecation We're all-in on Icechunk. Our existing Zarr v3 datasets will continue updating for 90 days, at which point we will sunset them. Consider them deprecated and update to use our Icechunk Zarrs by July 23, 2026 . Speaking of which…
Short one today — a quick housekeeping note for anyone pulling from our icechunk data products. Icechunk 2 is out ! If you're using our datasets via Icechunk, please update your Python library before Thursday April 24 : pip install icechunk --upgrade or uv add icechunk --upgrade Note that icechunk 2 requires Python 3.12 or higher. If you're on an older runtime, now's the time to bump it. We'll be…
Salutations, wxers. To balance out all the April 1 corporate tomfoolery I'm going to be less of a goofball today. Let's see if I can keep it STRICTLY BUSINESS. Lots to report since my last update. The dynamical(.org) core is a-running. ECMWF AIFS Single forecast The AI weather model from the European Centre has entered the catalog. ECMWF's AIFS Single — 15-day global forecasts at 0.25°, 6-hourly…
It's Spring and unlike hobbits (I'm reading LOTR), who would be thinking about gardens and second breakfast and whether Farmer Maggot's mushroom crop will come in early, we are staring at grib files. Just kidding, we are walking outside enjoying the warm weather ... while talking on the phone about grib files. We've been busy with a whole load of updates. GFS analysis is live Our old…
What say the pantheon of forecasts? Shall AMS travel be a disaster? As the flakes that fall thick upon a winter's day, when Zeus is minded to snow and to display these his arrows to humankind - he lulls the wind to rest, and snows hour after hour till he has buried the tops of the high mountains, the headlands that jut into the sea, the grassy plains, and the tilled fields of men... Find Alden @…
Check out the Icechunk example usage in our docs for IFS ENS , GFS , and HRRR forecasts to get started or browse these datasets on the Registry of Open Data on AWS . Why Icechunk? Consistent reads every time. Updating a standard Zarr while users are reading it requires extreme care. Icechunk handles atomic updates correctly, eliminating rare race conditions that can occur during standard Zarr…
Colder temps may be here (where I live, at least), but ensembles are so hot right now . Can we agree on that? No? Ha ha get it!? As of this week The European Centre's IFS ENS forecast is live and updating in our catalog . See the catalog for init times, spatiotemporal domain, and the variable list. I suppose with that HRRR "release trailer" I've signed myself up for something completely…
The last time I wrote you all, the summer sweltered, determinism was still alive (now it is dead , and only 11 AI weather papers had been published in Nature (now the total is up to 188,732). If I keep it up y'all are never going to take me seriously. But lo, some seriously new stuff! HRRR go brrr HRRR forecasts are live and updating! Check out the release trailer (lol) here . Give it a try! It'll…
GFS forecasts are live GFS is GEFS' less ambiguous older sibling. The archive is processed and the updater is chugging: 4 years of 16 day forecasts, hourly step, released 4 times a day. GFS might be wrong, but at least it's confident! A note from Alden, Chef de cuisine Tonight, I've prepared for you a dataset I think you will enjoy. Check out the example notebook linked in the docs to make your…
Not even my proprietary UniverseCast™ Reality Ensemble Forecast⁺ could have predicted this. The most unlikely of scenarios: A podcast ? Yes, a podcast. And it's called Weathering . We are trying to make sense of the current wave of progress occurring in weather forecasting. Each episode, we focus on a paper or group of papers and try to take a intertextual approach -- clicking on mental hyperlinks…
Well, technically it's been live for a few days. Check out our new GEFS analysis , which is derived from the GEFS forecast catalog item. Both are live updating, and the analysis was designed to work easily with the forecast dataset. For example, you can develop your algorithm/model/sentient AI television weatherperson on the analysis, and then deploy it live with the forecast dataset. COOL! 25+…
While modest on the scale of Computers, 1M is not bad for our baby’s first week in the world. The gigabytes are flying and we’ve received requests from 51 countries over the last few days. No single person has downloaded the whole dataset, so thank you but also WHAT ARE YOU WAITING FOR?! A reminder that the GEFS dataset is Zarr v3, so don’t forget to update your Xarray>=2025.1.2 which includes v3…
Weatherlings, the time has come for our first non-demo release. Weighing in at 115TB compressed and 815TB (!) uncompressed, GEFS ain’t messing around. Last year, we launched dynamical.org into the world and watched as hundreds of weather data nerds emerged (sublimated?) from the woodwork. Our community has been growing steadily, united by a shared mission to make weather data Spark Joy . 🌎…
CALLING ALL WEATHER DWEEBS! Bracing for a winter storm here in Nashville, where I moved to from Boston in order to escape winter storms. 3 to 7 inches they say - good thing I brought the XC skis! dynamical.org @ AMS From the look of it the vast majority of our subscribers will be at AMS. We are excited to see you there! Alden will be presenting at the 105th Annual Meeting of the American…
Ahoy weatherheads! It’s been a busy few months for us as we shift gears from dreaming to building. Catalog updates Our dataset for NOAA GFS analysis has getting some awesome usage! Thanks to everyone who has tested it out. We are prepping to launch NOAA GEFS forecasts as the second entry to dynamical.org/catalog . It will include a large historical forecast record and update in real-time. Dealing…
Last week, we announced dynamical.org to the world. The response was unexpected*: in just seven days, over a hundred nerds registered themselves as testers with many more following along for the ride. Apparently, we are not alone in our frustration with yesterday's weather data. Together we can create a future of well-organized data about our most beloved chaotic system! Today, we are excited to…
Introducing dynamical! Thank you for being the first to follow our journey making weather and climate data more accessible. We are humbled by the response. Our mission clearly strikes a chord - weather data today is a pain to work with. Over one hundred people (!?) signed up to test the first versions of the data we endeavor to release. And that’s without having spilled the beans on the best parts…