Human Testing, Unit Testing

The Windows Vista team knows something about finding bugs after all:

“Donnelly [who manages part of Microsoft’s Vista test operation] tries to do the opposite of what an IT manager would recommend. He changes all the default settings, for instance. And instead of testing a clean installation on a new machine, he’ll try to upgrade an older model. ‘You find bugs,’ he said, ‘You absolutely find bugs that way.’”

It’s a start.

The security guy in me has a hard time choking down the Unit Testing doctrine. Programmers don’t find deep bugs in their own code. Machines don’t find deep bugs when running in a contrived development/testing environment. These approaches find surface bugs, and that’s valuable, but they shouldn’t help you sleep at night.

People doing dumb and/or malicious shit finds deep bugs. You can’t script dumb and malicious.