Moving Petabytes: Data Export at AWS RDS Scale

I work on data export infrastructure for Amazon RDS. The job is straightforward to describe: take data from database instances and move it to S3. The execution is not straightforward at all.

RDS is one of the largest managed database fleets in the world. MySQL, PostgreSQL, MariaDB, Oracle, SQL Server — running across hundreds of thousands of customer instances. When you build infrastructure that touches the entire fleet, the numbers get uncomfortable fast.

What Makes This Hard

The obvious challenge is volume. Moving petabytes of data is an I/O problem, a network problem, and a cost problem simultaneously. You can't brute-force it.

The less obvious challenge is consistency. A database export isn't useful if it captures the data mid-transaction, with some rows committed and others not. The way this is solved here is through database snapshots — the export reads from a point-in-time snapshot, not the live instance. The live database is never touched.

That decision has a second-order benefit: the customer's workload is completely unaffected by the export. No locks, no I/O competition, no latency spikes at 2am when someone kicks off an export of their production database. The snapshot is the isolation boundary.

The Long Tail of Weird Databases

The scale problem isn't just petabytes — it's the sheer variety of what customers have built. When you support an export feature across hundreds of thousands of database instances, you encounter configurations that are technically valid but practically extreme.

Tables that have hit column limits. Databases with five million tables. Schemas where a single table has accumulated years of unbounded ALTER TABLE ADD COLUMN calls until it's pushing against the engine's hard ceiling. Instances configured in ways that made sense for some workload in 2014 and have never been touched since.

None of these are bugs in the customer's database. They all have to work. The system needs to handle the long tail gracefully — not just the happy path of a well-structured 10GB OLTP database, but also the 3AM edge case where someone tries to export a database that happens to have 5.3 million tables and your code assumed table counts were in the thousands.

Parallelism and the Data Lake Model

Fast exports need parallelism — split a table into chunks, export concurrently, write multiple Parquet files to S3 in parallel. The question isn't whether to parallelise; it's how to think about what ordering, if any, the output needs to preserve.

The answer comes from the consumer. RDS exports land in S3 as Parquet files, queried downstream via Athena, Redshift Spectrum, or similar services. These tools don't read rows sequentially — they scatter-gather across files in parallel, filter by partition metadata, and push predicates down into the columnar format. Row-level ordering within a file is irrelevant to them. What matters is partition layout.

That reframes the problem. Instead of asking "how do we preserve order while going fast?", the question becomes "how should we partition the output so downstream queries are cheap?" A customer running Athena queries filtered by created_at wants the export partitioned by time. A customer doing key-range scans wants output organised accordingly. The parallelism is unconstrained — the design work is in the S3 prefix structure and Parquet file organisation.

The output format answered a question that looked like a tradeoff.

Operating at This Scale

One thing that surprised me: at this scale, even rare events happen constantly. An error rate of 0.001% across a large enough fleet is still thousands of errors per day. Your error handling paths need to be as well-tested as your happy path, because they run all the time.

Observability becomes load-bearing infrastructure. If you can't measure what your system is doing across the entire fleet — latency distributions, failure modes, resource consumption — you're flying blind. Dashboards aren't a nice-to-have; they're how you find out that your latest deployment is quietly misbehaving on one specific instance type in one specific region.

What I've Taken Away

Building at AWS scale is a lesson in humility about assumptions. The things you'd normally treat as edge cases — clock skew, network partitions, instance failures mid-export — happen constantly and need to be handled gracefully, not just logged and ignored.

The work is fundamentally about reliability: building systems that do what they promise, even when the environment around them doesn't cooperate. That's a hard problem and an interesting one.