Against the Grain

Mrs. Fiske Warren (Gretchen Osgood) and Her Daughter Rachel – John Singer Sargent, 1903

One of my favorite pastimes is visiting the Museum of Fine Arts in Boston. They have a vast collection of art from around the world, and they regularly rotate their displays, so you can always discover something new with repeated visits.

One thing that fascinates me there is the different styles of painting on view. In the European art wing you can see works by Impressionist painters such as Claude Monet, which use visible, broken brushwork that looks disjointed up close but vibrantly unified at a distance. Down the hall are Dutch Baroque masters such as Rachel Ruysch, whose fine brush strokes portray impressive detail up close, set against the black backgrounds of her still-life paintings.

Flowers in a glass vase on a marble slab – Raychel Ruysch, 1709

These style variations remind me of a concept in data modeling called grain. Grain, or granularity, is the level of detail represented by a single row in a table, such as a fact table in a data warehouse. It answers a fundamental question: what does one row in this table represent?

Getting the grain right matters. In a purchase order, item costs may be allocated at the line-item level while shipping costs are assigned at the order level. Get that wrong and you can inflate shipping costs on a dashboard because the order-level amount is multiplied by the number of lines.

There are also performance and storage considerations. If the grain is too coarse, it is like examining a Monet painting up close. You might make out the figure’s head, but the flowers in the background look like random clumps of brush strokes.

Camille Monet and a Child in the Artist’s Garden in Argenteuil – Claude Monet, 1875

If the grain is too fine, the table balloons into billions of rows and storage becomes expensive. The brush detail in Ruysch’s still lifes is impressive, but the same density in a wall-sized mural such as Thomas Sully’s The Passage of the Delaware would take far too long to paint and would get lost in everything else.

The Passage of the Delaware – Thomas Sully, 1819

The most effective data models are like the works of John Singer Sargent. Sargent is known for changing brush width inside a single canvas. Faces and key features are handled with tighter, more controlled strokes so they remain legible. Clothing, backgrounds, and secondary areas are allowed broader, more visible marks. The Impressionist works nearby push the broader stroke further. The choice is intentional in both cases.

Note the differences in brush strokes between the face and the dress.

Grain works the same way. A year-over-year growth cube is the face: it needs to be trustworthy at a distance, and an hourly fact table is unnecessary detail that disappears into the aggregate. A 24-hour activity histogram is the dress or the landscape: hourly grain is the stroke size that makes the pattern visible.

Many data-warehouse models never change brushes. They inherit the grain from the source data and apply it to every question. Then people wonder why some dashboards look overworked and others look smeared.

When I was still learning the ropes in these environments, it felt like channeling early-80s Huey Lewis and the News: I’m taking what they’re giving, ’cause I’m working for a living.

I watched one of these play out on a dashboard that was missing a SKU. The investigation ran the usual five whys. A sales analyst expected several product SKUs to appear on a dashboard, but they did not. The fault turned out to be a missing pallet SKU in a different planning table. In the source planning system, one pallet SKU was one row. In the warehouse fact (or in a related fact), that single planning row had become multiple rows—or had disappeared entirely—because the grain had been inherited rather than declared. Each system had processed the grain it was given; none of them preserved a single, queryable representation of the original pallet-level row. The dashboard therefore could not answer the question without a multi-system chase.

A different grain problem showed up in a casino environment. Rated-member play arrived at the grain the source system used: player, slot machine, start time, and end time. The business wanted hourly activity to track peak-time metrics. We had to invent a new fact table, fed by a new pipeline, with a distribution formula to prorate metrics across the hours spanned by each activity. The source grain stayed where it was useful. The analytical grain became the one the business had actually asked for. That new table was the contract.

The same choice appears with health-insurance claims. A claim’s status changes over its life: received, pended, adjudicated, paid, denied, adjusted. Many source systems store only the current status. A warehouse model that simply overwrites the prior state loses the history the business will later need. The better model treats the claim as a slowly-changing fact. When a relevant status changes, the model writes a new version row that freezes the key metrics at that moment. The grain is no longer “one row per claim.” It is “one row per claim per relevant state.” The source system can keep its current-state design. The analytical model keeps the versions the business will ask about when someone wants to know what the claim looked like on the day the status moved.

If you are implementing grain changes from the original source, the best place to do it is at the gold level. Preserve the original grain at the silver level after it has been cleansed, validated, and conformed. If there are disputes about the new grain, you want to be able to go back to the original data before it was folded, spindled, and mutilated.

You’re showing your age if you remember this.

None of these cases required a new platform. They required a decision about grain before the table is named. The source grain is a starting point. It’s not a contract.

The analytical grain is the contract. Write it down. Name the person who answers for it. If the source grain and the business grain differ, build the new fact or the versioned fact instead of stretching the inherited one. The technical steward implements the grain. The owner answers for whether it is the right grain. Metrics, joins, and history then have a place to stand.

Most “data quality” fights that survive past the silver layer are grain fights. They are correctable because grain can be written down and owned the same way a certified definition is owned. Start with one critical fact. State the grain in one sentence. If the sentence does not match the grain the source gave you, that is the signal to build the table the business actually needs. Don’t go against the grain.

Leave a Reply

Your email address will not be published. Required fields are marked *