Writing Tests Using the CS 70 TestingLogger Library

In CS 70, we provide a testing library to make it easy for you to write tests for our classes and functions, run those tests, and see the results. It is based on a TestingLogger class and two global functions,

• affirm(condition)
• affirm_expected(expr, expected_result)

An Example

Here's code for example.cpp that shows how our logging framework works. In your own code, of course, you'd be testing your own functions.

#include <iostream>
#include <string>
#include <sstream>
#include <cs70/testinglogger.hpp>

// This example test will pass
bool test1() {
    TestingLogger log("test 1");

    constexpr int MAX = 4;
    int result = 3;   // imagine this result came from some more complex code
    affirm(result < MAX);

    return log.summarize();
}

// This example test will fail if our loop goes around one time too many
bool test2() {
    TestingLogger log("test 2");

    constexpr int MAX = 4;
    for (size_t i = 0; i <= 4; ++i) {
        affirm(i < MAX);
    }

    return log.summarize();
}

// This example test will pass because 1 + 1 = 2
bool test3() {
    TestingLogger log("test 3");

    affirm_expected(1 + 1, 2);

    return log.summarize();
}

// This example test will fail
bool test4() {
    TestingLogger log("test 4");

    for (int i = 0; i < 3; ++i) {
        int val = i * i;  // imagine we were *supposed* to have computed i + i
        affirm_expected(val, 2*i);
    }

    return log.summarize();
}

// This example tests printed output
bool test5() {
    TestingLogger log("test 5");
    std::stringstream ss;

    ss << "Hello World! " << 42;
    affirm_expected(ss.str(), "Hello World! 42");

    return log.summarize();
}

int main() {
    TestingLogger log("all tests");
    test1();
    test2();
    test3();
    test4();
    test5();

    if (log.summarize(true)) {
        return 0;  // success
    } else {
        return 1;  // failure
    }
}

You can compile the code in our Docker image with

clang++ -g -std=c++17 -Wall -Wextra -pedantic -c example.cpp
clang++ -g -std=c++17 -o example example.cpp -ltestinglogger

Notice that we need to link against the testing-logger library (-ltestinglogger).

Running this example produces the following output:

test 1 passed!
FAILURE (after 4 passes): example.cpp:23:       i < MAX

Summary of affirmation failures for test 2
----
Fails   / Total Issue
1       / 5     example.cpp:23: i < MAX

test 3 passed!
FAILURE (after 1 passes): example.cpp:43:       val, expecting 2*i
        actual:   1
        expected: 2

Summary of affirmation failures for test 4
----
Fails   / Total Issue
1       / 3     example.cpp:43: val, expecting 2*i

test 5 passed!

Summary of affirmation failures for all tests
----
Fails   / Total Issue
0       / 1     [test 1]
1       / 5     [test 2]
0       / 1     [test 3]
1       / 3     [test 4]
0       / 1     [test 5]

affirm

The first two examples (test1 and test2) use affirm, which takes a boolean argument. The test passes if the boolean is true, and fails if it is false. (If you've seen assert in C, affirm is similar, although it does not stop the program if the test fails; it just logs the failure, recording where in the program the failure occurred, including the function and line number.)

Notice that when tests pass, as in test1, the affirm doesn't print anything and log.summarize() just prints a short message that the test passed.

In test2, on the other hand, the failed affirm produces the message

FAILURE (after 4 passes): example.cpp:23:       i < MAX

It shows you which line of the file the problem was on, as well as code that produced a false value. Note that affirm will only print this message once (the first time it fails). Subsequent failures will, however, be noted in the summary.

When a test fails, the summary is more detailed, showing you a list of all the failures:

Summary of affirmation failures for test 2
----
Fails   / Total Issue
1       / 5     example.cpp:23: i < MAX

affirm_expected

The next two examples (test3 and test4) use affirm_expected, which takes two arguments: an expression and an expected result. The test passes if the expression is equal to the expected result. If they aren't equal, the test fails, and the message shows you both the expected result and what the expression actually evaluated to.

In test3, the expression 1 + 1 is equal to the expected result, 2, so the test passes.

In test4, the expression i * i evaluates to 1, but the expected result was 2. The message shows you the actual value and the expected value:

FAILURE (after 1 passes): example.cpp:43:       val, expecting 2*i
        actual:   1
        expected: 2

If the types being tested do not support printing (via operator<<), affirm_expected still works, but the actual/expected values are not shown.

Testing Output

So far, our examples of affirm_expected are just testing the value of an expression. Sometimes, however, we want to print something and see if it looks the way it is supposed to. We can achieve this goal by using C++'s std::stringstream class.

Normally C++'s I/O streams read or write text from/to files or our terminal, but a std::stringstream instead targets an internal std::string object. When we write to a std::stringstream with <<, it adds our printed output characters to this string. std::stringstream's str() function returns the contents of this string object.

In test5, we print a string and a number to ss (which is a std::stringstream) . We then use ss.str() in affirm_expected to check that the string is what we expect it to be.

log.summarize()

Every testing function ends with a call to the summarize() member function of our TestingLogger instance. This function returns true if the all the tests passed, and false if any failed. It also prints a summary of the test results; failed tests get more detailed information in the summary.

summarize() has an optional bool parameter (which defaults to false). Passing true to the function causes it to produce a detailed summary whether any tests failed or not. We use that option in main to get detailed summary that include tests where everything passed.

TestingLogger

The TestingLogger class is the class that actually tracks all of the passed and failed affirmations and knows the name of the set of tests being worked on.

If you create a TestingLogger when one already exists in an outer function, its overall performance (total tests and total fails) will be logged into the outer logger. That feature allows main to produce the test output,

Summary of affirmation failures for all tests
----
Fails   / Total Issue
0       / 1     [test 1]
1       / 5     [test 2]
0       / 1     [test 3]
1       / 4     [test 4]

Running without any TestingLogger

You can use affirm and affirm_expected even without a testing logger; but if there is no logger, they will just abort the program with a message if a test files. This behavior means that you can safely use affirm and affirm_expected in any CS 70 implementation code.

Conclusion

The TestingLogger framework is specific to CS 70. There are other testing frameworks for C++, but most of them have so many features that there is a lot to learn, and we want you to focus on writing good test cases, not how to use a complicated testing framework. With TestingLogger, most of the time affirm_expected is all you need.

(When logged in, completion status appears here.)