It’s been a while since we’ve talked about the infrastructure behind Boxer, our cloud-native weather data processing system. Because Boxer ingests and processes weather datasets from a variety of sources, we’ve been investing in automated testing to improve our weather data monitoring. These tests watch our importer stacks, alert us when data is delayed, incomplete, or missing, and help us resolve issues before they affect customers.
How Boxer Handles Weather Data Monitoring
From the outside, our testing is pretty simple. A bit of Python code runs, pokes at a particular data set, and lets us know if something is missing. If it is, we get one of those messages above on Slack.
The tests fall into two categories: observations and forecast models.
For observations, such as our MRMS stack, we mainly care about freshness. If we haven’t received new data for 10 minutes, Boxer generates an alert so we can investigate before it becomes a larger issue.
Forecast models work a little differently. Models like GFS and HRRR are published gradually as NOAA releases forecast hours. Boxer assembles those individual files into complete model runs so customers always have access to both the latest available data and the latest complete forecast.
For HRRR, we’ll trigger an alert if a forecast run hasn’t stabilized within 3 hours of its nominal initialization time.
Together, these automated checks provide continuous weather data monitoring across the datasets our customers depend on, giving us early warning when something falls behind or doesn’t arrive as expected.
Real Problems
The real value of weather data monitoring is whether it uncovers actual problems, and these tests already have.
A few weeks ago, our GFS importer appeared to be running several hours behind. We use Amazon’s public S3 bucket for GFS, and maintenance generated an astonishing amount of unexpected activity.
Long ago, we’d work on every new data set we saw come in, which was a lot of fun when they’d add archival data. We’ve since added filters to catch this problem. These massive changes overwhelmed the filter, but didn’t cost us anything. After a quick tweak, we were caught up.
We’ve also seen a similar issue with HRRR data occasionally piling up, and we tweaked things accordingly. We’re still playing with that one, hence the message above.
Rolling It Out
We’re gradually adding these tests to every Boxer importer stack, rolling them out one or two at a time.
For widely used public datasets like GFS and HRRR, we’ll likely keep the alerts internal so we can resolve issues before customers ever notice them.
Customer-specific data is where weather data monitoring becomes even more valuable. Boxer processes proprietary datasets alongside public weather models, so we can share monitoring alerts directly in your Slack channel and let your team know if something unexpected happens with your data.
If you’d like that level of visibility into your own data pipelines, we’d be happy to talk.