Mr. Clean

So many leaves, so little daylight ….

November in New England is usually a very busy time. After the Halloween candy has been dumped in the trash but before the Christmas decorations go up, the fall foliage starts coming down in earnest. Next thing you know, the yard is covered in leaves that have to be raked up and disposed of before the first snowfall. Even worse, the clocks move back an hour, so you don’t have a lot of time to do it before it gets dark.

As unpleasant as fall cleanup may be, it’s important to perform these tasks. Leaves left on the ground can suffocate your lawn and attract unwanted pests. (And if those leaves are close to the house, the pests inevitably find their way inside once the winter freeze arrives.) They also can be a safety hazard for slips and falls, as well as clog rain gutters. Cleanliness is more than an aesthetic; it’s actually a strong factor in upholding a high quality of life.

Cleanliness was a habit that was drilled into me ever since my first uniform inspection as a cadet in ROTC. There were plenty of inspections that I was both subjected to and later conducted as an officer. After I got out, the habit still followed me into manufacturing.

In the plant, before we ever talked about cycle time or scrap rate, we talked about the floor. Tools left on benches. Spare parts that hadn’t been used in months. Oil stains that contaminated the epoxy mixer until the epoxy wouldn’t set correctly. The first thing a new manufacturing engineer learned was 5S: Sort, Set in Order, Shine, Standardize, Sustain. 5S was a practice developed by Toyota and is a major feature of the Lean Manufacturing System that spun off from Toyota’s production practices.

Infographic designed by Grok

Most teams stopped after the first three. The last two were the ones that kept the place from sliding back.

Data environments collect the same kind of clutter:

  • Tables no one queries anymore, and that haven’t been written to in years.
  • Columns in active tables that exist only because a source system once sent them, but no longer does.
  • ETL pipelines that still run because no one is sure what would break if they stopped.
  • Dashboards that no one runs anymore, and no one knows this because their usage hasn’t been tracked.

The difference is that the clutter is invisible until the bill arrives. A compliance auditor opens the data dictionary, sees a lot of gaps in the definitions, and starts asking a lot of uncomfortable questions. Or even worse, query performance slows to a crawl at the worst possible time, and trying to pinpoint the bottleneck takes a lot longer because of time wasted sorting through the cruft.

In one place I worked, I took the principle of 5S I learned from my manufacturing days and applied it to the data warehouse. Here’s how I did it:

Sort

Sort is the red-tag step. On the factory floor we hung a red tag on anything we were not sure we needed, then waited a month. If no one came looking for it, it left.

In a data warehouse or lakehouse the same move is simpler and harder: list every table, every column, every scheduled job that hasn’t been read or depended on in ninety days. (This is where it helps to have a built-in information schema to query the metadata.) Move the candidates into a PendingRemoval schema (or equivalent quarantine area). An automated job can drop them later if no one reclaims them. Most stay there. The ones that get pulled back tell you which definitions you actually use. The rest are just inventory that never moved.

Naming helps with sorting. We used singular table names (“Player” instead of “Players”) because we needed the table names to align with our self-service reporting platform. We assigned internal prefixes (, bk, BI) to column names to signal they weren’t for business consumption. We named stored procedures as verbs, because an action was taking place. If you couldn’t say the name out loud without it sounding stupid, it was probably already a candidate for the red tag. When the name itself makes the unused object obvious, the ninety-day list gets shorter.

Set in Order

Set in Order is the shadow-board step. A shadow board is a technique where the outline of each tool used at a work station is painted onto a board to indicate where it should go when not in use. If an outline is empty, it’s a signal that there’s a missing tool.

A typical shadow board in manufacturing (Source: Lean 5S Products UK)

Once the unused pieces are gone, the remaining ones need a place that is obvious. Schemas do that work the way a shadow board does on the floor.

  • Bronze lands what arrived.
  • Silver holds the cleaned, conformed version.
  • Gold holds the version the business has agreed to use.
    In case you forgot (Infographic designed by Grok)

    Domain or purpose schemas (SCD, Aggregation, Xform) keep related objects together. PendingRemoval is the holding area for what you have already decided to discard. Extended properties or catalog tags become the labels on the board so a search finds the object without asking.

    One important point is to avoid the “junk drawer”: a single wide “Misc” or “Temp” schema that just recreates the clutter you just removed.

    If you have an object tree in your graphical development tool, it’s better to go deep than wide. When a new engineer can open the catalog and know, without asking, which table is the certified member dimension and which one is still a landing zone, the place is set in order. Most teams skip this and then spend the next year explaining why three tables all claim to be “the” customer.

    Shine

    Shine is the inspection that happens while you clean. On the floor, wiping down a machine revealed loose bolts and worn seals. In data, the silver layer does the same job. Schema enforcement, type casting, deduplication, and conformance checks are not hygiene for its own sake. They are the moment you notice that a source column changed meaning or that a grain assumption no longer holds. Consistent naming and restrained commentary (the occasional clarifying note, not a running transcript) make those defects stand out instead of hiding in noise. If the silver checks are silent, defects move downstream and reappear as “data quality” tickets in gold. Shine makes the problem visible before it becomes a meeting.

    None of this requires a new platform. It requires the same sequence the floor already knew: remove what is not needed, give what remains a clear home, and clean while you look. Most technical debt is just unsorted inventory. It is correctable because you can red-tag one subject area, move the unused objects into PendingRemoval, write the layer and naming rules for what stays, and turn the silver checks into the inspection that keeps the place from filling up again. Start with one domain. Sort it. Set it. Shine it. The rest of the warehouse can wait until that corner stays clean.

    Standardize and Sustain—the two steps that keep the place from sliding back—are the subject of the next post.

    Leave a Reply

    Your email address will not be published. Required fields are marked *