Writing Tests Using the CS 70 TestingLogger Library

In CS 70, we provide a testing library to make it easy to write tests for our classes and functions, run those tests, and see the results. It is based on a TestingLogger class, and two global functions:

• affirm(condition)
• affirm_expected(expr, expected_result)

An Example

Here's code for example.cpp which shows how this logging framework works. In reality, you'd be testing your own functions.

#include <iostream>
#include <string>
#include <sstream>
#include <cs70/testinglogger.hpp>

// This example test will pass                                                     
bool test1() {
    TestingLogger log("test 1");

    constexpr int MAX = 4;
    int result = 3;   // imagine this result came from some more complex code      
    affirm(result < MAX);

    return log.summarize();
}

// This example test will fail we our loop goes around one time too many           
bool test2() {
    TestingLogger log("test 2");

    constexpr int MAX = 4;
    for (size_t i = 0; i <= 4; ++i) {
        affirm(i < MAX);
    }

    return log.summarize();
}

// This example test will pass because 1 + 1 = 2                                   
bool test3() {
    TestingLogger log("test 3");

    affirm_expected(1 + 1, 2);

    return log.summarize();
}

// This example test will fail                                                     
bool test4() {
    TestingLogger log("test 4");

    for (int i = 0; i < 3; ++i) {
        int val = i * i;  // imagine we were *supposed* to have computed i + i     
        affirm_expected(val, 2*i);
    }

    return log.summarize();
}

// This example tests printed output
bool test5() {
    TestingLogger log("test 5");
    std::stringstream ss;

    ss << "Hello World! " << 42;
    affirm_expected(ss.str(), "Hello World! 42");

    return log.summarize();
}

int main() {
    TestingLogger log("all tests");
    test1();
    test2();
    test3();
    test4();
    test5();

    if (log.summarize(true)) {
        return 0;  // success
    } else {
        return 1;  // failure
    }
}

You can compile this code in our Docker image with:

clang++ -g -std=c++17 -Wall -Wextra -pedantic -c example.cpp
clang++ -g -std=c++17 -o example example.cpp -ltestinglogger

Notice that we need to link against the testing logger library.

Running this example produces the following output:

test 1 passed!
FAILURE (after 4 passes): example.cpp:23:       i < MAX

Summary of affirmation failures for test 2
----
Fails   / Total Issue
1       / 5     example.cpp:23: i < MAX

test 3 passed!
FAILURE (after 1 passes): example.cpp:43:       val, expecting 2*i
        actual:   1
        expected: 2

Summary of affirmation failures for test 4
----
Fails   / Total Issue
1       / 3     example.cpp:43: val, expecting 2*i

test 5 passed!

Summary of affirmation failures for all tests
----
Fails   / Total Issue
0       / 1     [test 1]
1       / 5     [test 2]
0       / 1     [test 3]
1       / 3     [test 4]
0       / 1     [test 5]

affirm

The first two examples (test1 and test2) use affirm, which takes a boolean argument. The test passes if the boolean is true, and fails if it is false. (If you've seen assert in C, affirm is similar, although it does not stop the program if the test fails, it just logs the failure, recording where in the program the failure occurred, including the function and line number.)

Notice that when tests pass, as in test1, the affirm doesn't print anything and log.summarize() just prints a short message that the test passed.

In test2 on the other hand, the failed affirm produces the message

FAILURE (after 4 passes): example.cpp:23:       i < MAX

It shows you which line of the file the problem was, and what the code that produced a false value was. It will only print this message once however (the first time it fails). Subsequent failures will be noted in the summary.

When a test fails, the summary is more detailed, showing you a list of all the failures:

Summary of affirmation failures for test 2
----
Fails   / Total Issue
1       / 5     example.cpp:23: i < MAX

affirm_expected

The next two examples (test3 and test4) use affirm_expected, which takes two arguments: an expression and an expected result. The test passes if the expression is equal to the expected result. If they aren't equal, the test fails, and the message shows you what the expression evaluated to, and what the expected result was.

In test3, the expression 1 + 1 is equal to the expected result, 2, so the test passes.

In test4, the expression i * i evaluates to 1, but the expected result was 2. The message shows you the actual value and the expected value:

FAILURE (after 1 passes): example.cpp:43:       val, expecting 2*i
        actual:   1
        expected: 2

If the types being tested do not support printing (via operator<<), affirm_expected still works, but the actual/expected values are not shown.

Testing Output

So far, our examples of affirm_expected are just testing the value of an expression. Sometimes, however, we want to print something and see if it looks the way it is supposed to. We can achieve this goal by using C++'s std::stringstream class.

Normally C++'s I/O streams read or write text from/to files or to our terminal, but a std::stringstream instead targets an internal std::string object. When we write to a std::stringstream with <<, it adds our printed output characters to this string. std::stringstream's str() function returns the contents of this string object.

In test5, we print a string and a number to ss (which is a std::stringstream) . We then use ss.str() in affirm_expected to check that the string is what we expect it to be.

log.summarize()

Every testing function ends with a call to the summarize() member function of our TestingLogger instance. This function returns true if the all the tests passed, and false if any failed. It also prints a summary of the test. The summary is more detailed if the test failed.

This function has an optional bool parameter (that defaults to false). Passing true causes it to produce a detailed summary regardless of whether any tests failed or not. We use that in main to get the detailed summary that includes tests where everything passed.

TestingLogger

The TestingLogger class is the class that actually tracks all all the passed and failed affirmations and knows the name of the set of tests being worked on.

If you create a TestingLogger when one already exists in an outer function, its overall performance (total tests and total fails) will be logged into the outer logger. That is why main produced the output

Summary of affirmation failures for all tests
----
Fails   / Total Issue
0       / 1     [test 1]
1       / 5     [test 2]
0       / 1     [test 3]
1       / 4     [test 4]

Running without any TestingLogger

You can use affirm and affirm_expected even without a testing logger; but if there is no logger, they just abort the program with a message if a test files. This behavior means that you can safely use affirm and affirm_expected in any CS 70 implementation code.

Conclusion

The TestingLogger framework is specific to CS 70. There are other testing frameworks for C++, but most of them have so many features that there is a lot to learn, and we want to focus on writing good test cases, not how to use a complicated testing framework. With TestingLogger, most of the time affirm_expected is all you need.

(When logged in, completion status appears here.)