Saturday, April 24, 2010

Isolate Your Microtests

I've been working on and off on the FitNesse code base for several months. As a Windows developer, (maybe the only one contributing to FitNesse?), I run into a few issues that no-one else seems to encounter. One of them has been intermittent failures when running the suite of over 2000 microtests (aka unit tests). Several of the tests create files in a test directory, and then, like good tests should, clean up by deleting the files and the directory. Occasionally, these tests fail with an exception on the delete. Today I finally realized what's been happening. My IDE is receiving notifications of file changes and is trying to read the test directory to update its directory tree display. Most of the time, the directory is gone before the IDE can read it, but sometimes it's reading the directory when the microtest tries to delete it and the delete fails. The solution was simple - I put the test directory name in the IDE's file ignore list. Green bar all the time!

Lesson one: know your IDE. But the bigger lesson, I think, is: isolate your microtests. Isolate them even from the file system. We all know that we need to isolate our microtests from databases and networks, and the file system is another external dependency that we should break as well. I always use a file system interface that my microtests can implement with an 'in-memory' file system or a mocking tool. Sometimes I've wondered if this was really necessary, but today I think it definitely is.

Update: I've found other potential causes, e.g., virus scanning, that might occasionally collide with microtests doing file I/O. So don't.

Wednesday, April 7, 2010

Tweaking fitSharp performance

It's convenient to use wildcards in the FitNesse path:
!path c:\mypath\*.dll
However, if there's a lot of unnecessary files loaded by this path specification, performance can be impacted. The biggest overhead that fitSharp incurs, on top of running your System Under Test, is searching assemblies looking for classes. So if you can limit the assemblies that it has to search, this can help:
!path c:\mypath\myfixtures.dll
Sometimes every little bit counts!