Suppose we want to test some responsibility of a technical component. The component might be as small as a single, standalone function or as large as a multi-cluster distributed cache running on multiple machines in different parts of the world. Regardless of the component’s size, we test the responsibility by:

1. Establishing relevant preconditions in the component and in its environment.
2. Sending the component a message that triggers it to carry out the responsibility.
3. Comparing the results that the system produced to the results we want.

Step 1 requires us to control the variables¹ that affect the component as it carries out the responsibility. Step 3 requires us to observe the variables affected by the component as it carries out the responsibilty.

The easier it is to control and observe these variables, the easier the component is to test. The harder it is to control and observe these variables, the harder the component is to test.

Testability = Controllability + Observability.

To make a component more testable:

• Identify the variables that affect how the component carries out its responsibilities, and make those variables easier to control.
• Identify the variables affected by the component as it carries out its responsibilities, and make those variables easier to observe.

These steps may seem obvious or simple. The next section hints at some of the factors that make these seemingly simple, obvious tasks less obvious and simple. In future posts I’ll have more to say about these challenges and how to address them.

Any factor that makes key varaibles harder to control and observe makes a component less testable.

Component Complexity. The larger and more complex a component, the more variables are likely to be involved in its responsibilities. Refactoring a large component into smaller, more focused subcomponents can greatly reduce the number of variables that each test must control or observe.

Coupling. The more coupled a component is to other components, the harder it is to control and observe the variables involved in its responsibilities. Coupling makes a component depend indirectly on the variables that affect its collaborators. Testing a highly-coupled component may require controlling and observing key variables indirectly through the component’s collaborators, the collaborators' collaborators, the collaborators' collaborators' collaborators, and so on through many layers of indirection.

Visibility of Dependencies. Coupling is especially challenging when the component’s dependencies are hidden. A hidden dependency is any dependency that a component acquires in some way other than through its own interface. A component has a hidden dependency on any component it creates or that it retrieves from a hard-coded source, such as a static method of another class. Hidden dependencies can make key variables extremely difficult to control or observe.

An intermittent failure always includes these three parts: Failure, Fault, and Conditions.

• A failure is an incorrect result.
• A fault is an incorrect process, step, or data definition.
• Conditions are combinations of inputs, states, and events that affect the results.

An intermittent failure means that some component has a fault that under some conditions causes a failure.

If there’s a failure, there’s a fault.

If there’s a failure (intermittent or not), some process, step, or data definition is incorrect. The fault may be in the system we’re testing. It may be in the test. There may be multiple faults. But if there’s a failure, there is at least one fault.

Even if the failure is intermittent, the fault is not.

The fault is present even when no failure occurs. We may not know what the fault is, or where to look for it, but it’s there.

If the failure is intermittent, success or failure must depend on conditions that can vary.

If the fault is always present, the difference between success and failure must come from varying conditions. These varying conditions determine (among other things) whether the fault is executed, at what moment, and with what values.

If the failure is intermittent, we are not in control of the conditions that determine success or failure.

We call a failure intermittent only when we cannot produce it at will. We cannot produce the failure at will because at least one relevant condition is not under our control. And very often we don’t even know what the relevant conditions are.

Sometimes you can diagnose an intermittent failure quickly, guided by your intuition and expertise. But if you’re frustrated, that’s a sign that some important condition is outside of your control, and perhaps not yet in your awareness.

Consider trying this systematic approach to diagnosis, to generate a richer set of ideas to power your intuition and expertise.

Write down every variable you can think of that could possibly be involved. By variable, I don’t mean just the things you might declare in some source code. I mean anything that could vary.¹ For this step in particular, it’s often helpful to team up with a few other people.

During this brainstorming step, do not limit yourself to variables that you think are likely to be involved. Accept any variable that could possibly be involved… and don’t be too quick to conclude that a variable could not be involved. If you think of it, write it down. We will assess the utility of these variables in a later step.

Here are some ideas for identifying variables:

• Start with the result. If the failure is intermittent, we know the result is variable. Write it down as a starting point.
• Include any conditions you know or suspect. If you know of any conditions that seem to be involved, or if you suspect any, express them as variables and add them to the list. Even if you don’t know how or why a condition is involved, writing it down can help trigger new ideas.
• Work forward from each variable on the list. If this variable were to change, what other variables might be affected?
• Work backward from each variable on the list. What other variables might affect the value of this one?
• Look for states, durations, and the order of events. Intermittent failures very often arise when a specific event occurs while some part of the system is in a specific state.

Organize the variables into categories (and sub-categories, and sub-sub-categories, …), using whatever criteria seem appropriate.

As you categorize, you will very likely think of additional variables. Write them down and include them in your categorization.

You may want to use mind-mapping software or outlining software for this. Any tool that helps you categorize the variables will do.

Instead of categorizing, or in addition, you may want to create a graph of how the variables affect each other. Any kind of model helps you understand the relevant variables will do.

Assess the categories and pick the one whose seem most likely to affect the results.

Assess the variables in the selected category and pick the one that seems most likely to affect the results.

Assess the possible values of the selected variable and pick one that seems most likely to increase your understanding of what is happening.

Gain control of the selected variable², and arrange for it to have the selected value.

Run the test, or exercise the system in some other way, and observe the results.

Repeat with different values for the selected variable as long as it seems fruitful.

Given what you’ve learned from your experiments so far, return to whatever earlier step is most likely to increase your understanding.