A Framework for Evaluating Software Engineering Decisions

People of good will may disagree. This is just as true in software construction as in other areas of life. In life, you can agree to disagree and move on with your day, but in software construction a decision must be reached. Usually, this involves choosing between 2 or more competing ideas or solutions. In this post I’m going to outline a way of thinking about these evaluations.

“Decision” in this context can be very small or very large, ranging from a choice involving a class or method design versus selecting a language or methodology. Obviously, the larger the decision the more complex the evaluation will become, but here I try to provide a way of thinking about the decision that will work at any granularity.

Since engineers — and software engineers in particular — are opinionated buggers, we need to find a way to focus everyone on the same basic set of evaluative criteria. True objectivity may not be entirely possible, but we can at least try leaning in that direction…

The criteria I try to apply are simple: cost and benefit. It clearly makes no sense to do something that has no benefit but a high cost, and it clearly makes no sense not to do something with lots of benefit and no cost. But the fuzzy areas in between are where most of our decisions need to be made, and where we often get mired in competing notions of what constitutes a benefit and what constitutes a cost.

Clearly, when we are talking about “engineering”, or “software design” or “cost vs benefit” we are talking about an actual, pragmatic, real-world problem we are trying to solve. Therefore, we should be pragmatic in the way we resolve the fuzzy areas of decision evaluation.

I propose that both benefits and costs for a given alternative need to be evaluated and placed along a continuum:

  • Theoretical: The benefit is generally agreed to hold some value, but it is difficult to pinpoint its overall effect on the efficacy of the alternative. The cost is generally agreed to be non-zero, but it is difficult to gauge its magnitude.
  • Characterizable: Either a cost or benefit can be clearly characterized in real-world terms — e.g. a performance benefit, or a cost in increased programming errors due to complexity — but it is still an indefinite value either due to the nature of the alternative or some unknown or unknowable environmental influence.
  • Quantifiable: A benefit can be quantified in terms of engineering effort saved, performance increased, etc. A cost can be quantified in terms of engineering effort, error rate, etc.

The key thing we are looking for is to break down the simple idea of cost and benefit in some way that will help us compare alternatives in a systematic way.

A few clarifications and examples:

  • For something to be quantifiable it requires some form of measurement. What is meaningful here will depend on the organization and the team, and two alternatives may use different units: i.e. one alternatives benefit is performance and the other’s is development velocity.
  • When characterizing a cost or benefit it is valid to characterize it relative to another alternative, i.e. alternative A will be more expensive to implement that alternative B.
  • Things like “best practices” are of theoretical benefit unless there is some attribute of the problem or product that ties directly to it.

Correlating the costs and benefits by these categories:

Theoretical Benefit Characterizable Benefit Quantifiable Benefit
Theoretical Cost Meh. May be some benefit, may be some cost. Nothing compelling or distinguishing. Cheap, but with some benefit. This is ideal, especially if the magnitude of the benefit is large.
Characterizable Cost Unless the cost is extremely small, why would you use this alternative. Compare the magnitude of the cost and benefit. Compare the magnitude of the cost and benefit.
Quantifiable Cost Unless this cost is extremely low, don’t use this alternative. Compare the magnitude of the cost and benefit. Compare the magnitude of the cost and benefit.

Generally speaking, the more toward the quantifiable end of the spectrum the more reliable the evaluation of the alternative is.

This doesn’t solve all the problems of course: In the end you still need to pick an alternative and choosing between an alternative with theoretical cost and quantifiable benefit vs one with characterizable cost and benefit could be quite difficult. It’s here that the magnitude of the benefit/cost will come into play.


About jeffkotula

Software engineer, musician, and so on...
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s