Why I’m Breaking the DRY Rule: The Case for Data Duplication

Here is something that sounds uncomfortable coming from a data engineer: sometimes the right design is to store the same data twice. Sometimes even more than twice.

That can look like a violation of DRY — Don’t Repeat Yourself. But the principle is often applied to data without considering what it was designed to solve.

DRY protects code from competing definitions

In software, DRY helps us avoid implementing the same business rule in several places. If every service calculates revenue differently, changing the definition becomes risky. A single shared definition reduces inconsistency and maintenance cost.

Data storage has a different set of pressures. Two copies of the same facts can serve very different access patterns:

• one copy optimised for low-latency lookups;
• another organised for analytical scans;
• a search index designed for text retrieval;
• a cache designed to absorb repeated requests; or
• a historical copy designed for auditing and replay.

The important distinction is between duplicating a definition and materialising data for a purpose.

A real latency trade-off

In 2020, I worked on a platform that needed extremely fast lookups. The normalised source model was correct, but answering a common request required several joins. That was acceptable for some workloads and too slow for the online path.

We kept one representation in HBase for fast key-based reads and another in Hive for analytics and batch processing.

The two systems were not accidental copies. They represented explicit decisions:

• HBase served predictable, low-latency access.
• Hive supported large scans and analytical processing.
• The ingestion workflow defined how both representations were produced.

Trying to force both workloads through one storage model would have made at least one of them worse.

Duplication is not free

Storage may be relatively inexpensive, but duplicated data creates operational costs that are easy to underestimate.

You now need to answer:

1. Which system is authoritative?
2. How quickly must copies become consistent?
3. What happens when one write succeeds and another fails?
4. Can the copies be rebuilt from the source?
5. How will the team detect drift?
6. Who owns schema changes across every representation?
7. Does duplicating the data create new privacy or retention obligations?

If those questions do not have clear answers, duplication can become uncontrolled technical debt.

The design becomes much safer when one system remains the source of truth and every additional representation is treated as a reproducible projection.

When duplication is justified

I consider duplication when at least one of these conditions is true:

• Latency: The primary model cannot meet a measured response-time requirement.
• Workload isolation: Operational and analytical queries interfere with each other.
• Independent scaling: Different consumers have very different throughput patterns.
• Availability: A local copy allows a critical path to continue during an upstream failure.
• Specialised access: Search, graph, vector, time-series, or key-value access requires a different physical model.
• Stable contracts: A curated data product protects consumers from changes in operational schemas.

The decision should follow evidence, not fashion. Measure the current bottleneck, define the required service level, and compare the full operational cost of each alternative.

Guardrails, not laws

“Never duplicate data” is not a useful architecture rule. Neither is “storage is cheap, copy everything.”

A better rule is:

Duplicate data deliberately, identify the source of truth, and design reconciliation before the copy becomes critical.

Best practices are guardrails. Engineering judgment is knowing when the business requirement justifies crossing one — and what safeguards must replace it.

The same principle applies when deciding whether services should communicate directly or asynchronously. I explore that trade-off in Should We Always Build Decoupled Systems?.