Saturday, April 24, 2010

Isolate Your Microtests

I've been working on and off on the FitNesse code base for several months. As a Windows developer, (maybe the only one contributing to FitNesse?), I run into a few issues that no-one else seems to encounter. One of them has been intermittent failures when running the suite of over 2000 microtests (aka unit tests). Several of the tests create files in a test directory, and then, like good tests should, clean up by deleting the files and the directory. Occasionally, these tests fail with an exception on the delete. Today I finally realized what's been happening. My IDE is receiving notifications of file changes and is trying to read the test directory to update its directory tree display. Most of the time, the directory is gone before the IDE can read it, but sometimes it's reading the directory when the microtest tries to delete it and the delete fails. The solution was simple - I put the test directory name in the IDE's file ignore list. Green bar all the time!

Lesson one: know your IDE. But the bigger lesson, I think, is: isolate your microtests. Isolate them even from the file system. We all know that we need to isolate our microtests from databases and networks, and the file system is another external dependency that we should break as well. I always use a file system interface that my microtests can implement with an 'in-memory' file system or a mocking tool. Sometimes I've wondered if this was really necessary, but today I think it definitely is.

Update: I've found other potential causes, e.g., virus scanning, that might occasionally collide with microtests doing file I/O. So don't.

No comments:

Post a Comment