Reconsidering Unit Test Mocks

It’s a common practice when writing unit tests to create “mocks” for underlying system functionality. Mocks simulate the behavior of code that is used by the code being tested. (This can be contrasted with “scaffolding” which is conceptually code that lies above the code you’re testing and prepares the environment for it.) A typical example is mocking the behavior of a database: you create a shell of database functionality that is used by the code you are testing.

The reasoning behind this practice is fairly straightforward: if a test fails you can have confidence that the problem resides in your code rather than in some other code that yours depends on. But let’s examine the cost/benefit tradeoffs in a little more detail and see if mocking is actually a Good Thing. (Spoiler: it isn’t…)

First off, it is important to recognize the — often high — cost of creating and using mocks. You have to create an interface describing the mocked behavior. You have to modify your code to use the mock instead of the normal supplier of the behavior. You have to implement the mock. And you have to create some form of factory to create the correct version of the mock implementation depending on whether or not you are running a unit test. This is all work that is required solely because of the choice to mock! Now, I’m not advocating being penny-pinching in unit-testing, but we have at least some nominal duty to avoid being spendthrifts…

Secondly, even when using a mock, isolation is still not guaranteed. With anything but the most trivial mock you still have the likelihood that the mock implementation itself has errors. I do grant that the isolation provided by a database mock is better than if the database itself were used directly, but mocks provide, at best, a partial solution to this problem. (How bad of a problem is it, really?)

Thirdly, I maintain that testing code in context yields more rigorous tests than when the code is tested in an isolation achieved by mocking and scaffolding. While mocks may remove the uncertainty perceived by relying on other components, they also remove the particular idiosyncrasies and irregularities that are integral to these other components. Performance variability, threading, race conditions, memory usage; all these really tricky behaviors are glossed over by your mocks. So when you test with a mock, you aren’t really exercising your code in the same way it will be exercised in production. Your tests are weaker.

There is Another Way…

As much as possible, execute your unit tests in context. Rather than spending time building mocks, spend some time figuring out how to automate your testing at the top-most level, and use that to trigger your low-level unit tests. Every test should exercise as much of the entire system as it will in production.

Fear not: Isolation can still be achieved!

The key is to run your tests often and to ensure reasonably thorough coverage in them. (I didn’t explicitly state it yet, but all tests should be fully automated, naturally.) If you meet these two conditions, errors flagged in your test runs will actually be isolated by the fact that the amount of changed code since the last run will be limited — you can just check your source control system to see what the last check-in touched.

Isolation is best achieved through test coverage and continuous integration, not through the use of unit testing mocks.

Advertisements

About jeffkotula

Software engineer, musician, and so on...
This entry was posted in Coding, Software, Testing and tagged , . Bookmark the permalink.

One Response to Reconsidering Unit Test Mocks

  1. Pingback: Interfaces Increase Complexity | Jeff K

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s