The end of software testing? No, not really

Testing: no longer about establishing
the correctness of a system

A few years ago, I penned a whitepaper that contained these words:

"No amount of testing can fully eliminate the bugs and security holes in a complex software system, as no test suite could possibly anticipate every scenario the system may encounter."

As it turns out, I wasn't whistling dixie. My colleague Chris Hobbs, who has forgotten more about software design that I could hope to learn in multiple lifetimes, notes that:

"... a modern, pre-emptible, embedded operating system with about 800 assembler instructions in its core has more than 10³⁰⁰ possible internal states. To put this into perspective, the Eddington Number (the number of protons in the observable universe) is about 10⁸⁰.

Don't know about you, but those numbers far exceed what my brain can grasp. And if that's not enough, the 10³⁰⁰ figure applies only to the OS core — it doesn't account for the huge number of additional states that are introduced when you start running applications and their supporting libraries.

So why bother with testing when you can only hope to exercise, say,
0.00000000000000000000000000000000000000001% of the system's possible states? It all has to do with a concept called confidence from use.

Rather than attempt an explanation here, I invite you to read a paper that Chris has published, titled "Testing as a road to confidence-from-use". Chris not only explores the concept, but discusses the degree to which confidence-from-use data gathered on one version of a system can be applied to a slightly modified version. Recommended for anyone interested in software testing or reliability.