What Actually Happened?

I've been asking a version of the same question across several kinds of work lately: what actually happened?

Often, the calculation isn't the problem. The source is available. The result can be reproduced. What I want to understand is whether it supports the conclusion being drawn from it.

That takes more work than checking the math.

A business question gets translated into something a system can measure. Choices are made about what counts, what gets excluded, and where the process begins and ends. Those choices may be entirely reasonable. Later, the result appears in a report or product under a short, familiar label. Someone uses it to make a decision without knowing how much interpretation sits underneath it.

I pay particular attention to what happens when a result gets reused. A measure built for one operational question can become evidence for a much larger claim because it's the number people have available. Nobody has to change the calculation for the meaning to change.

This is the kind of problem I tend to get pulled into. I like following the question through the product, the underlying systems, and the way people actually use them. Sometimes the important discovery is that two apparently conflicting answers were never answering the same question.

Two Answers Can Both Be Defensible

Consider the difference between keeping a customer continuously and winning them back after they leave.

Both are valuable. But they tell us different things about the business.

A measure designed to answer whether an earlier customer is with us today might count both. A measure of uninterrupted renewal should not. Each can be calculated correctly, and the results can still differ substantially.

The problem comes when both travel under a word like "retention." Someone evaluating the stability of the customer relationship may read the first number as evidence of the second. We could overlook an interruption or give the wrong work credit for the outcome.

Resolving that requires a choice about what we want to understand. Which question matters to this decision? What would we do differently depending on the answer?

I don't want to choose a definition because it produces the better-looking result. I want the question settled before we decide whether we like the answer.

Nor do I think every team has to use one measure for every purpose. Different questions deserve different answers. They need names and explanations that let people tell them apart, especially when the result leaves the team that produced it.

The Interpretation Is Part of the Product

I think this belongs in product work much earlier than it usually gets discussed.

When we decide what a status means or what a result is called, we are deciding what another person can reasonably believe. The interface may look simple because we have made those choices for them. That puts a responsibility on us to get the meaning right.

I don't want a disclaimer attached to everything. I want the assumptions that could change someone's decision to remain visible. A person should be able to understand the scope of an answer without reconstructing how several systems work.

For me, that is where working across disciplines becomes useful. Engineering can explain what was recorded. The people running the operation can explain what happens around that record. Someone responsible for the business outcome can explain what they need to decide. Getting those perspectives together often changes what we thought the product needed to show.

I enjoy getting into that detail. I also want the work to end in something people can use. We may need to change the way a result is calculated, separate outcomes we had combined, or carry the meaning more clearly into the interface. An explanation I can give in a meeting isn't enough if everyone else will encounter the same confusion tomorrow.

How Much Weight Can the Answer Carry?

I don't apply the same standard of evidence to every decision.

An estimate may be enough to choose a small experiment. I would want stronger support before using it to judge a team's performance or commit to a costly change. The question is how much we are asking the answer to carry, and whether we understand its limits.

Sometimes the useful outcome of an investigation is finding that two numbers differ for a good reason. There may be nothing to fix in either calculation. We need to stop presenting them as interchangeable.

Other times, the evidence cannot support the conclusion yet. I want that to be clear enough that someone can still make a reasonable choice about what to do next. Uncertainty should not automatically stop the work. It should affect how we proceed.

What bothers me is making that work invisible and passing it to the next person. A colleague should not have to discover, halfway through a decision, that a familiar label means something narrower than they assumed.

I keep coming back to this because I dislike unnecessary complexity. Smart people have enough difficult choices to make without spending their time reconciling answers to questions nobody realized were different.

Building a useful system means doing enough of that work upfront that people can rely on it. They can understand what it establishes, recognize where their judgment is still needed, and get on with the decision.

Before I consider an answer ready, I want to ask one more question:

If someone acts on this exactly as we've presented it, what will they believe happened?