Program Correctness Matters
Testing seems like a lot of wasted effort. If I run my program and it works, cool, and if it crashes I'll fix it.
I take a lot of care with my code, so it usually works and when it doesn't work I can usually figure out why. Having to write tests as well can feel like busy work.
Hmm. Well, we expect that at least some of you will eventually be involved in professional software development.
In the real world, all sorts of important systems in our lives are controlled by software. Not testing that software can have disastrous consequences.
-
You'll find this semester that it is very easy to write code that catastrophically fails in certain cases and never even notice! In a class project that may not be a huge deal. But it's our ethical responsibility to prevent that as much as we can in software that affects people's lives!
Program Bugs Can Have Very Serious Consequences
The Therac-25 computer-controlled radiation-therapy machine caused (at least) six accidents between 1985 and 1987 where patients received up to 100 times the prescribed dosage of radiation in a concentrated area, causing burns that progressed to radiation poisoning and significant tissue damage, causing three of the six victims to die from complications from their overexposure.
Although previous versions of the machine had hardware interlocks to prevent operators from running the machine in the wrong mode, the Therac-25 relied only on its software, and that software was riddled with bugs, starting with the reuse of code from earlier machines that assumed hardware interlocks would prevent some dangerous conditions.
The code also had various failure modes that only paused the machine momentarily (as well as some that would require a reboot). Because these failure happened frequently, and often didn't seem to have a cause, operators got used to telling the machine to proceed with treatments without checking the settings.
It also had a race condition where a mistaken mode selection by the operator, followed by a quick edit to switch modes to the correct one, would result in the machine missing the edit, and thus going with the wrong mode as originally selected.
Another problem involved an overflowed 1-bit variable, where a test of the position of the upper collimator (used to narrow the radiation beam) would be skipped if the variable was set to 0. During the set-up process, the test routines were run many times, each time incrementing the check variable. If the operator hit the “set” button at the same moment the variable overflowed and flipped from 255 to 0, the system would assume the 0 value meant that everything was set up correctly and turn the beam on at full power.
As with most radiation overexposure incidents, other factors (such as operator sloppiness, maintenance delays or avoidance, and poor internal investigations on the hospital side, and poor communication and quality-control procedures on the manufacturer’s side) also played a role. A 1992 Technical Report, An Investigation of the Therac-25 Accidents by Nancy G. Leveson and Clark S. Turner, published by the UC-Irvine Information and Computer Science Department (available as a PDF) looks at all these factors, including the various investigations, and has a special focus on the software issues, including detailed breakdowns of the key problems (with pseudocode!). It's both a fascinating and sobering read.
(A follow-up article by Leveson, The Therac-25: 30 Years Later, reflects on the lessons learned (or not learned) by industry in the years since the Therac-25 accidents.
Code is Everywhere
In addition to medicine, software interacts financial transactions, medical records, street lights, and almost every other aspect of our lives.
It's no exaggeration to say that testing can be a matter of life and death… and even when it's not actual lives at risk, we have a responsibility as professionals to put well-tested code into the world.
Writing good tests is a skill, though, and like any skill, you get better with practice.
-
And that's why we give you lots of opportunities to practice writing good tests in CS classes, while you're still in school!
What is the purpose of testing?
“Program testing can be used to show the presence of bugs, but never to show their absence!”
-
That's a quote from renowned computer scientist Edsger Dijkstra (1930–2002).
In other words, if our code passes all of our tests, that doesn't necessarily mean there are no bugs, just that it doesn't have the particular bugs we've tested for. There might still be a flaw in the program.
What if we test every single possible input? Then we'd know that there were no bugs at all.
Sounds good!!
-
Almost all programs have far too many inputs or possible behaviors to exhaustively test them all.
-
Suppose we were just going to read the letters
A..Zin some order, that would be 26! = 403,291,461,126,605,635,584,000,000 possible inputs. -
That means we need to change our mindset. Instead of testing everything, we will focus our efforts on finding problems.
We have to be thoughtful in choosing what to test. We don't want to waste our time writing tests that always pass and never catch any bugs, but we also don't want to leave important aspects of our program untested and allow bugs to slip though unnoticed.
Coming up with good tests is a skill that you'll develop in this class. One way to think about it is to come up with cases that might elicit different behavior. For example, testing the inputs "XXX" and "YYY" is probably not as good as testing "" and "XXX", because edge cases like something being empty are often the kind of thing that aren't handled properly.
Always remember that writing tests is like being a detective. Your goal is not to prove that your code already works. It’s to find where it doesn’t work!
Testing in CS 70
There's a lot I'm coming up to speed on, adding testing into the mix feels like a lot!
-
Your first couple of assignments in CS 70 will have visual output, and we won't spend too much time on testing.
-
Starting with Homework 4, though, writing tests will be an important part of the code that you write.
-
We'll start off gently: on Homework 4, we'll give you tests for your data structure, and you'll come up with tests to find the limits of two algorithmic approaches.
-
After that, you'll be responsible for writing your own tests—and uncovering bugs in buggy implementations that we provide.
(When logged in, completion status appears here.)