In past blog posts, we’ve talked about using AWS Lambda to handle importing weather data. However, as our importers become increasingly complex, we’re encountering limitations in terms of time, cost, and scale. So we’re moving some of that work to Amazon ECS (Elastic Container Service). In this post, we’ll break down our experience with AWS Lambda vs ECS and explain why ECS is becoming a better fit for our import workflows.

What is AWS Lambda?

AWS Lambda, launched in 2014, is an event-driven, serverless function platform. It was a great fit for us initially, as it was simple, stateless, and easy to deploy. We used it to trigger container-based importers when new files were uploaded to S3. It worked great, especially for radar.

However, as our datasets grew larger and more complex, the limitations of Lambda became increasingly difficult to ignore.

Weather Data Import at Scale

Our weather data system, Boxer, supports a variety of importers. Some are low-latency (such as MRMS, updated every 3 minutes), while others are higher-volume (like GFS, updated every 6 hours, or HRRR, updated every hour).

Initially, we ran everything through Lambda. Each importer operated within its own container and ran on a schedule triggered by new files in S3 buckets at regular intervals. This process was very simple and pleasant! For radar, that setup still works beautifully.

The fast response time and built-in scaling are worth the extra cost.

But for everything else, including forecasts, large multi-hour models, and multi-variable pulls, we’re hitting limits.

AWS Lambda vs ECS: Where Lambda Struggles

For radar, we want the minimum latency. If our source backs up (which sadly it does sometimes), we want to catch up immediately. Lambdas let us do that, and it’s worth the extra cost. Moreover, radar frames are fairly small in comparison to whole forecast hours.

With forecasts, Lambdas have become problematic for a couple of reasons: The time limits and the cost.

We have several importers that rely on HTTP queries to get their results. We’ll query time slice by time slice and variable by variable until we get everything we need. I don’t love this approach, but it’s what the providers give us, so it’s what we do.

Sometimes those collections of queries can exceed Lambda’s 15-minute cap. We can overcome this problem by breaking the queries into groups and running them in parallel. That’s nice, but it can be a hassle for configuration. Which variables do you break out, and so forth?

Even when the time limit isn’t a blocker, the cost can be. We receive hourly HRRR updates, and running them in Lambda was simply becoming too expensive.

Why ECS Makes More Sense for Forecast Data

We moved our HRRR importer to ECS with Fargate. Now, when new data arrives, ECS tasks spin up, pull from the queue, and process data in parallel.

This change gave us:

• Lower cost per import cycle
• More control over runtime and memory
• Simpler scaling for large workloads

It wasn’t all upside, though. We lost some of Lambda’s built-in magic. All our log messages are still in CloudWatch, but individual task timing and memory use were missing at the metric level, and will need to be replaced with explicit calls to CloudWatch.

Storage can fill up, memory leaks become a problem, and there are issues with task containers being longer-lived. Moving away from it, you’re suddenly responsible for a bit more of it.

Thoughts on AWS Lambda vs ECS for Serverless Architectures

People often say, “Serverless is just someone else’s servers,” and yeah, fair enough. Boxer is a cloud-native system. We hold our state in S3 and DynamoDB. Our services are stateless. But AWS Lambda vs ECS is a real tradeoff, especially when scaling up.

Lambda remains a great fit for specific jobs, particularly those that require low latency or rapid coordination. But for anything heavier or longer-running, ECS is the way forward for us.

We don’t regret starting with Lambda. It helped us move fast. But now that we’re moving importers to ECS, we’re seeing clearer paths to scale and cost efficiency.

TL;DR:

AWS Lambda vs ECS isn’t about better or worse—it’s about the right tool for the job. For us, the job has changed.