Dirty Data (Done Dirt Cheap)

The running joke amongst Generation X is that we don’t listen to music after the year 2000. We feel comfortable around the music we grew up with, so we prefer to stick to the music of the 80s, sometimes the 70s, and occasionally the 90s. That’s true for me, although in my case, while my musical taste stayed in the 80s, over the years it’s wandered over the globe sampling music from the decade. When I was in Europe I caught the last few years of European synth pop and Italo Disco. When I moved to Europe it was Celtic rock with Clannad. And just prior to COVID I was digging Soviet rock through Kino and Victor Tsoi.

So we Gen Xers regard it with some bemusement when the younger generation start rediscovering the artists we grew up with. First it was Millennials getting rickrolled and finding they actually liked Rick Astley, leading the ginger crooner to come out of retirement. More recently, the backdrop of the 80s in the Netflix show Stranger Things brought many 80s songs back into the spotlight, such as Kate Bush, David Bowie, and Metallica. The series finale, which dropped on New Year’s Eve 2025, did it again. Prince and the Revolution’s “Purple Rain” re-entered the Billboard Hot 100 at No. 27, its first appearance on the survey since Prince’s death in 2016. “When Doves Cry,” the other Prince song in the finale, was the one new listeners had to look up. It topped US and UK Shazam on January 2, 2026, in part because the title is barely in the lyric.

The thing that most interests me, however, are the reaction videos. Sometimes, new ears notice things that mine have overlooked.

The reaction I keep coming back to is AileenSenpai’s first listen to “When Doves Cry,” posted in January 2026 after the finale had pushed the song back into circulation. She had just come off “Purple Rain,” so she arrived expecting sadness. What she got was a song that opens like a dare and then turns into a family argument:

Maybe I’m just like my father, too bold
Maybe you’re just like my mother
She’s never satisfied (she’s never satisfied)
Why do we scream at each other?
This is what it sounds like
When doves cry

She stops the video on his face and says the only honest thing about the record. She doesn’t know whether to cry or dance.

I’ve known that song since it first hit the airwaves. What I hear, when I hear anything structural at all, is the hole: no bass line, no hi-hats, Prince playing every part and then deleting the bass so the rest of the track could breathe. She never mentions that. The comments do. One of them puts it better than I can. So many layers, and no one notices the one that is missing.

That is the part that struck me during a deep dive into data quality. A fresh listen doesn’t improve the song. It tells you which layer you stopped hearing. The name for that, on a lakehouse, is medallion architecture.

Medallion architecture is a design pattern commonly found in lakehouse architecture, using three distinct layers — Bronze, Silver, and Gold — to manage the ingestion, validation, and transformation of data. With each hop the medallion system elevates the cleanliness, structure, and reliability of the data.

• Bronze. The bronze layer preserves data in the original format it arrived. This allows audit trails to view the original state, and quality checks focus on making sure the data arrives in a timely manner.
• Silver. The silver layer does the bulk of cleaning. This is where schema enforcement, type casting, deduplication, conformance, and many other checks happen. The data here is clean, though it may not be business-ready for consumption.
• Gold. The gold layer prepares the data for business consumption. At this level aggregation, denormalization into star schemas, derived metrics, and other transformations take place according to business rules. Many of the data governance aspects, such as certified entity definitions, also apply at the gold level. Silver ensures the data is accurate; gold ensures that it is useful.

As I spent more time reflecting on what each individual layer does, I came to realize I had already put many of these concepts into practice, though not necessarily in that order, and not under those names.

Bronze, for me, was the argument I got tired of losing. A bad metric would land in my inbox, and the first question was always whether the process had invented it. I started keeping the original data, whether it came from a file or a database pull, so I could find the row in question and show the number was already wrong. That’s an audit habit, not a transformation. I didn’t call it bronze. I called it not getting blamed for someone else’s garbage.

Silver was the work I called “fixing the feed.” In most of my ETL processes I was already doing schema enforcement, type casting, deduplication, and conformance. State codes arriving as “Mass.” or “MA” or “massachusetts” got forced into one value. Duplicate contract numbers got collapsed. Rows that couldn’t be parsed went to a quarantine table instead of downstream. I didn’t know the vocabulary. I knew the data was wrong and that a report built on it would be wrong in a way I couldn’t explain later. Clean, at that point, still wasn’t the same thing as useful.

Gold was the part that started only after I trusted the feed. Derived metrics, aggregations, prorations, and a lot of time reconciling what a fact meant to the business versus what it meant in the source system. Certified definitions, before I had a name for them, were the arguments about whether a contract counted on the sale date, the effective date, or the date the underwriter was willing to accept it. Silver made the row accurate. Gold decided what it was for.

The main difference was that I was doing this all at once, without dividing it into specific tiers. Bronze I had mostly separated already, because I needed the original to defend myself. Silver and gold lived in the same procedure. I knew instinctively that the data had to be clean before I created a derived metric. I didn’t know that the cleaning and the metric were different jobs, with different owners, and that a change to one shouldn’t require redoing the other.

The clearest example came from my first data developer job, almost 20 years ago. I was working for an extended warranty company on a project to automate sending warranty extracts to our underwriting partners. We pulled the data from the warranty contracts table. That pull, kept as it landed, was the bronze layer I didn’t have a name for.

Then I had to decide which rows not to send. Some of them were bad data entered into the system: missing serial numbers, impossible dates, fields an agent had typed over. Some of them were business rules we had to follow contractually. Nearly all of our warranties applied only to the Lower 48 states, so products sold in Canada, Hawaii, or the various US territories weren’t allowed. Both rows ended up in the same place, which was “do not send.” They weren’t the same kind of exclusion. One was a defect. One was a contractual filter. On an instinctual level I knew to quarantine the bad data first, and only then apply the Lower 48 rule to what remained.

There was no medallion architecture in that shop. There wasn’t even a data lake back then. (At least not in flyover country.) There was a sequence I wouldn’t reverse, because filtering a business rule against a dirty feed launders the defect into a decision.

What I didn’t have was a boundary between those two filters. A change to the territory rule meant reopening the job that also cleaned the feed. A bad source row could disappear into the same bucket as a Canadian sale, and a month later nobody could say which was which. That’s the cost of doing silver and gold in one pass. The cheap part, in the AC/DC sense of this title, was writing it all in one procedure. The expensive part was explaining the number afterward.

Once you get past the fancy names, you realize that medallion architecture are just named layers to stop and reflect.

• Bronze is where you prove the source was dirty.
• Silver is where you finish cleaning.
• Gold is where you argue about definitions without re-cleaning the feed.

The reaction video didn’t remake “When Doves Cry”; it revealed what you stopped hearing: the parental lines or the bass that isn’t there. The comment under that video, and the comment under a pipeline, are the same kind of note. So many layers. Nobody notices the one that’s missing.