Microsoft Office

From John Guin’s blog: have a theory about the value of automation (and many other test tools) and the time it takes to develop, maintain and use them. I call this my “Robinson Crusoe Theory” of testing and it goes like this:

Imagine you are cast away on a deserted island. You managed to save a fishing pole, a hoe and a barrel of corn seed. You need time to build a raft to escape, but you face a dilemma. Each day you face starvation and have two choices. The first choice is to go fishing, and assume you will catch enough fish to eat for a day. They may be tasty or not, maybe filling or not, but they will (barely) meet your nutrition needs. Or, you could start a farm, and since you have plenty of seed, at the end of the season, you will have food for a year. If you choose to fish, you have to fish all day every day and never have time to build the raft. Of course, you will not starve. If you decide to plant the crop, the yield is too far into the future for you to prevent starvation, but if implemented, you will have months of time to build a raft after the crop is ready.

The trick, of course, is to figure out how to plant the crop and reap the long term benefits rather than get locked into barely surviving by fishing every day. In the test world, we face the same dilemma. Having automation running is the equivalent of farming - there is little payoff today, and the amount of work necessary to build and maintain a viable automation system is very large. The most obvious alternative is manual testing, which is the equivalent of fishing. If I chose this path, each time a new build of OneNote was ready to test, I would need to run all my manual test cases again to verify nothing broke. If I have more than 8 hours of tests and get a new version to test daily, you can see how quickly I would get behind. My last estimate of how long a test pass would take for one of my features was 31 days - clearly, since we get builds much more often than once per month, I’d be in trouble very quickly if I relied on manual testing.

Fishing for bugs is a short term success - maybe you will catch the “great white shark” types of bugs, but you may also get distracted with the little fish. You won’t know until the end of the day, and even then, you won’t know what you missed. Farming is steadier - you know the expected yield, you will go through the crop daily with your hoe looking for weeds, but you will also know exactly how ripe the crop is every day and know when to expect the yield.

Obviously, the task at hand is to find more efficient ways to test which give me a high yield (to “farm” for bugs, rather than “fish,” if you will). I have a few more resources than Robinson Crusoe, though, but if I get caught up in the excitement or short term benefits of “fishing,” I might overlook them.

To read the rest of this entertaining and informative blog, click the link above.