Assessing Which Risk Heuristics to Implement in the Radiator
In short, a risk heuristic should not be added to the radiator unless it meets the following criteria:
- The heuristic should identify an aspect of risk - The heuristics applied in Ratcheting should give us the ability to tactically identify which commits are risky and offer an opportunity to mitigate that risk. We need to identify heuristics that are really good at identifying risk and start with those.
- It should be obvious how to mitigate the risk identified by the heuristic - Once we have identified a risky commit, we need to be able to do something about it.
- The heuristic's sensitivity should be configurable - As time goes on, we might need to tweak the threshold of a heuristic to better identify what is risky. For our first heuristic, test lines of code added, we might want to tweak the ratio of test lines of code to production lines of code over the life of the project to reduce the chances of flagging files that are not risky.
- The heuristic should make sense in a historical context - We need to know how each heuristic will behave against prior recent commits. Are we getting enough information from the heuristic? Is it providing the right level of information? If the heuristic is flagging too many or too few commits, it might not be worth adding.
- The heuristic should provide pre-commit feedback - Developers need to know what they are getting in to before they issue their commit. This feedback should be part of a check-in script and run in the same relative amount of time as your unit test suite.
- The heuristic should be able to be introduced at a minimal stage - Some of these heuristics can get very complex quickly. The key here is to start small by finding the easiest possible point of entry and improve the smartness of the risk analysis over time.
This provides a good lens with which to look at individual risk heuristics. It is important to establish this rule-of-thumb early and stick to it in order to fight feature creep.