Implementing Medallion Architecture

Medallion architecture organizes a data platform into three layers — bronze, silver and gold — with data becoming cleaner and more useful at each step. The idea is simple. Applying it well depends on being strict about what each layer is for.

The three layers

Bronze: raw and complete. Data lands here exactly as it arrived, plus ingestion metadata such as load time and source file. Nothing is cleaned or dropped. Bronze is your insurance policy: when a downstream rule turns out to be wrong, you can rebuild everything from it.

Silver: clean and conformed. Types are enforced, duplicates removed, names standardized and records validated. Silver tables represent business entities — customers, policies, transactions — in a consistent shape, regardless of which source system they came from.

Gold: shaped for use. Aggregates, dimensional models and feature tables built for a specific consumer: a dashboard, a report, a machine-learning model. Gold is where business logic lives.

Bronze to silver in PySpark

from pyspark.sql import functions as F
from pyspark.sql.window import Window

bronze = spark.read.table("bronze.transactions")

latest = Window.partitionBy("transaction_id").orderBy(F.col("_ingested_at").desc())

silver = (
    bronze
    .withColumn("amount", F.col("amount").cast("decimal(18,2)"))
    .withColumn("transaction_date", F.to_date("transaction_date", "yyyy-MM-dd"))
    .withColumn("_rn", F.row_number().over(latest))
    .filter("_rn = 1")                      # keep the most recent version of each record
    .drop("_rn")
)

valid = silver.filter(F.col("transaction_id").isNotNull() & (F.col("amount") >= 0))
rejected = silver.subtract(valid).withColumn("_reason", F.lit("failed validation"))

valid.write.mode("overwrite").saveAsTable("silver.transactions")
rejected.write.mode("append").saveAsTable("quarantine.transactions")

Two details matter here. Deduplication keeps the latest version of each record rather than an arbitrary one. And rows that fail validation go to a quarantine table with a reason, instead of disappearing.

Where data quality checks belong

Put checks at the boundary between layers:

• Into bronze — did the file arrive, is it readable, does the row count look plausible?
• Bronze to silver — schema, types, required fields, uniqueness, referential integrity.
• Silver to gold — business rules and reconciliation totals against the source.

A failed check should stop bad data from moving forward. Whether it stops the whole batch or only the offending rows is a decision to make deliberately for each table.

Common mistakes

1. Cleaning data in bronze. Once raw data is altered, you can no longer replay history when a rule changes.
2. Business logic in silver. Silver should be reusable by every consumer. A calculation that only one report needs belongs in gold.
3. No agreed definition of "clean". What counts as a valid record is a business decision. Write it down and get it agreed before building.
4. Gold tables without owners. Every gold table should have a named consumer. Tables nobody uses still cost money to refresh.
5. Treating the layers as mandatory steps. Small or simple datasets do not always need all three. The pattern is a guide, not a rule.

Incremental processing

Rebuilding every table on every run does not scale. Process only new or changed data where you can, using change data feeds, watermarks or merge operations. Design for re-runs: loading the same batch twice should produce the same result, not duplicates.

Does it fit your platform?

Medallion architecture works well when several sources feed several consumers and correctness has to be demonstrable. It gives you a clear place for raw history, a single trusted version of each entity, and outputs tailored to their users.

Its real value is not the three names. It is that everyone on the team can answer the same question the same way: where does this data come from, and can I trust it?