The popcorn heuristic

17Jul06

At PlateSpin someone mentioned that shipping software is like making popcorn. Healthy testing progress follows similar patterns of bugs or "pops" that making a healthy bag of popcorn does.

 

I decided to run this heuristic by James Bach and Michael Bolton. After much back and forth here is the result (Written by James)

 

POPCORN HEURISTIC OF TEST PROGRESS

————————————-

 

KEY IDEA:

 

Finding bugs is something like popping popcorn. If the test process is healthy, expect the peak bug find rate to occur near the half-way point.

 

IMPLICATIONS:

 

1. If the peak find rate was nearer the beginning or the end of the test process than to the middle, then the test process probably was not healthy.

 

2. If the find rate is near its peak right now, then you probably aren’t close to shipping or you aren’t close to a healthy test process.

 

3. If you believe you are near the beginning of your healthy test process, then you should expect to find bugs more and more quickly.

 

4. If, despite every effort to find bugs, you are finding fewer and fewer of them, that probably means the software is reaching release readiness.

 

FALLIBILITY:

 

1. Unlike popcorn, bugs are very diverse. The same "heat" won’t make different bugs "pop". Therefore, the find rate may be slowing down simply because your testing isn’t good enough, not because the bugs aren’t there to be found. This may lead you to ship too soon.

 

2. Unlike popcorn, different bugs have different levels of importance.

Therefore, it’s the find rate of important bugs that we care most about.

However, there may not be enough important bugs to establish a clear trend.

You may ship too soon or too late.

 

3. Unlike popcorn, new "kernels" are constantly being added to the bag while it is popping. Sufficiently disruptive changes are expected to revive the find rate.

 

4. Unlike popcorn, you can’t hear all of them pop. If many bugs are going unreported, either because they are found by developers, or found by beta testers and not reported, or found by testers and not considered important enough, then the find rate will look artificially low. This is especially true near the end of a test project, as fatigue depresses sets in.

Filed under: Software Testing   |  2 Comments