If you’re not familiar, Amazon Simple Storage Service is an object storage product with a well-defined interface. You put file-like objects into it, and you can get whole objects or parts of them back out. What we had to learn, as many others have before us, is how S3 is priced and what that means for our own data storage architecture. We’re in the middle of a transformation that’s reducing our S3 costs, for a certain kind of usage, by a factor of 10. It’s a story of trial, error, and eventually Amazon S3 cost optimization.

Treating S3 as a File System

When you’re just starting out, Amazon S3 cost optimization isn’t top of mind. At low volumes, you can treat S3 as a file system. The objects are basically files, and you can pretend the keys break down into directories. You can connect a file system interface to S3, although I don’t recommend that.

Our approach is to treat S3 as the system it is and interface with it directly. Thus, our (mostly Python) code reads from S3 when it needs data and writes to S3 when it’s finished. Between that and our cache-heavy approach to data display, we were using S3 quite well in some cases.

Just not in this case.

Importing Weather Models: GFS Example

Let’s take one of our importers as an example. The Global Forecast System model is a workhorse for global weather forecasting. It’s not the densest or the best, but it’s well known and reliable. Our customers use it for a variety of cases and locations.

The GFS is published every six hours and contains hundreds of variables and levels. These include temperature, and the levels are typically 2m or 10m and higher. A weather model is a big cube, so there can be a bunch of them.

When we started, we focused on the simpler variables that our customers wanted to, quite literally, see. This was a few dozen at most. The S3 costs were minimal, so Amazon S3 cost optimization wasn’t yet on our radar.

Scaling up our Importer: GFS Example

Customers have requested additional variables to the point where we’re importing around 600 slices or variable/layer combinations specifically for GFS. We’re also deriving a few new ones, and we’re storing some of the data in two different formats for fast lookup. Well, five if you count three of the visual formats.

We store our data in Zarr, which is a discussion for another day. Using the variant we use, this will result in three files per variable, two of which are small and one of which is 1-2 MB.

GFS runs 6 times per day and generates 209 forecast hours we’re interested in. That means we’re creating around 1-2 million small files for S3 daily, just for the import. We’re also generating a few thousand files for querying whole forecasts. A drop in the bucket compared to the rest.

The Mistake: S3 Likes Big Chunks

If you’re an old hand with S3, you immediately see our mistake. Our files are too small. Way, way too small. S3 prefers to deal in 100 MB chunks.

Or to put it another way, S3 is going to charge you as if it’s dealing with 100MB chunks, whether it actually is or not. So you should get on board.

We were saving 1-2 TB of data per day in the form of 1-2 million files. If you do the math… well, it’s much less than 100MB per file.

Switching to One Big File: GFS

We process forecast hours as they arrive, so the largest unit we can easily store is a single forecast hour. Due to the nature of Zarr, it’s actually forecast hour plus data type (e.g., float32, float64, int, etc).

So that’s what we did. Our importers now write an entire forecast out to a single file (per data type). That results in two files per forecast per data type. The way Zarr is structured, there is a metadata JSON file and (as we’ll explain in another post) a single ‘brick’ containing the data. Not exactly standard Zarr, but it’s how we work.

Cutting the number of files by two orders of magnitude significantly reduced our costs. Now our long-term storage costs dominate our costs for uploading (PUT) to S3. Downloading (GET) isn’t significant at the moment.

Maybe Don’t Use S3 For Anything But Storage

We also moved away from signaling on S3. I’ll explain this because it’s common with data in our industry.

When we’re done processing a data set, such as a frame of a radar mosaic, we want to notify other parts of our system or our users that it’s ready. We used to write a .txt file to indicate that. Something with the exact same name as our Zarr file, but ending in .txt.

You can subscribe to NewObject messages on an S3 bucket, which is quite convenient for this purpose. However, not the best use of S3, as we’re being charged for the 100MB object, regardless of its actual size. Kind of.

We moved to generating an actual SNS message. We had to do so since the files were now decoupled from the individual variable/level slices. That had a lot of benefits within our system, such as not needing to puzzle out what the file is based on the .txt file, and the ability to shove in more optional data. 

Lessons Learned in Amazon S3 Cost Optimization

There’s more to the story on the Zarr side. It wasn’t easy to get the file format structured the way we needed it. We aimed for one physical file per Zarr store, and although there are options for that, they’re a bit squirrelly. We ended up creating a ‘Brick’ store, which is very effective, but violates many of the goals of the open Zarr format. More on that in another post.

In the end, this wasn’t all that complicated. We initially used S3 in a very naïve manner. When we scaled up, it started costing us, and we need to re-architect. Doing so significantly reduced the cost and time spent uploading and fetching, providing a clear example of Amazon S3 cost optimization in practice.