CS 372 Spring 2016  >  Notes for February 10, 2016

CS 372 Spring 2016
Notes
for February 10, 2016

Testing

Unit Testing & TDD

Recall that a unit is a small testable piece of code—for example, a single function. When we do unit testing, we test the units in the software we are developing.

Unit testing is typically done using a unit-testing framework. Many good frameworks are freely available; we will look at a couple of these shortly.

We keep our testing code in the same repository as the project code. A good rule is that no code is checked in unless it makes all tests pass.

In the Test-Driven Development (TDD) methodology, unit tests are written before implementing software features. Specifically, when we do TDD, we follow a four-step procedure when adding a new feature to software.

  1. Add one or more tests for the feature to the project unit tests.
  2. Make sure that the new tests fail. Note: In order for tests to fail, the testing code must compile and execute. If we are testing a new function, then we need to write a dummy version of that function.
  3. Add the feature to the software.
  4. Make sure that all tests pass. If they do not, then keep working until they do.

Note that TDD involves only black-box testing. When you use TDD, you may wish to add white-box tests as well; but these would be in addition to the tests you write as part of TDD.

Our in-class development will be done using TDD. You are encouraged to use TDD outside of class as well.

Unit-Testing Framework Example #1: Catch (C++)

Introduction

Catch is a simple, easy to use, high-quality unit-testing framework for C++ projects. It is available as a single header file (catch.hpp). A link to the GitHub page for Catch is on the class webpage; it includes downloads, a tutorial, and a complete reference.

Catch is distributed under a permissive FLOSS license (“FLOSS” = Free/Libre Open-Source Software). You should read this license; it is not long. But essentially, it says that you may distribute catch.hpp with your project, free of charge, as long as copyright notices are left intact and the license is included. Thus, if you include catch.hpp in your project, then it is a good idea to include the license file there, too.

File catch.hpp requires no special installation; it can simply be included in a project’s source tree—the directory structure holding the project source code.

Code Organization

When we include unit-testing code in a project, we need to be able to execute the tests, and we also need to be able to execute the project code without running the tests.

One way to do this in C++ is to put project code into header files. These headers contain no tests. A separate main-program file holds the unit-test suites; it includes the project headers and the test-framework header(s) (catch.hpp in this case). If the project itself is a complete program, then the project main-program file includes the project headers—again, these contain no tests.

We developed a function to compute Fibonacci numbers using TDD. See fibo.h for the Fibonacci-computation code. See fibo_main.cpp for a simple main program that uses the above code. See fibo_test.cpp for a test suite; this requires catch.hpp.

Note

If you are doing C++ development in this class, then you may use any unit-testing framework you wish, as long as you are using it legally, and you are able to demonstrate execution of your test suites in class.

In particular, if you are using an IDE, then this may include its own unit-testing framework, and you may wish to use that.

However, if you are unsure what testing framework to use, then Catch is certainly a good one.

See the Catch website for complete information on using Catch.

Unit-Testing Framework Example #2: doctest (Python)

Introduction

The doctest module is a part of the Standard Library of the Python programming language. It allows unit tests to be included in the documentation portion of a Python source file, for optional execution.

Docstrings are a Python feature for including documentation in a Python source file in a way that allows for separate documentation files to be generated automatically. It the first item in a Python source file, function, or class is a string, then that string is the docstring for the file, function, or class. Typically, these use the Python multiline string syntax, in which strings begin and end with three double-quote marks (“"""”).

Here is a simple Python function to compute the cube of a number. Note the docstring.

[Python 3.x] 

def cube(n):
    """cube(n) -> the cube of n."""

    return n**3

Writing doctests

When we use the Python interactive environment, we can type Python expressions and statements for immediate execution. Below, “>>>” is the prompt for the Python interactive environment.

[Interactive Python 3.x] 

>>> cube(12)
1728
>>> cube(0)
0

The idea of a doctest is that an interactive Python session is included in a docstring essentially unchanged.

[Python 3.x] 

def cube(n):
    """cube(n) -> the cube of n.

    >>> cube(12)
    1728
    >>> cube(0)
    0

    """
    return n**3

Now we have two test cases: when we call function cube with parameter 12, then it should return 1728. And when we call it with parameter 0, then it should return 0.

Executing doctests

If a Unix-line command-line interface is available, then we can execute the doctests in a file as follows. Suppose our Python source file is called cube.py. Below, “$” is the command-line prompt.

[*ix Command Line] 

$ python3 -m doctest cube.py

The above will print the results of failing doctests, giving no output if all tests pass.

To print the results of all doctests, switch to verbose mode by adding “-v” to the command line, as shown below.

[*ix Command Line] 

$ python3 -m doctest cube.py -v

If a file is meant to be imported as a library in other Python source files, then a common thing to do is to give the file its own main program, which does not execute if the file is imported. The main program executes the doctests. This is done by writing code like the following at file scope (not inside a function or class), usually a the end of the file.

[Python 3.x] 

if __name__ == "__main__":  # Executed as separate progam?
    import doctest          # Get the doctest module
    print("Running doctests")
    doctest.testmod(verbose=True)
                            # Execute all doctests, verbose mode

Above, eliminate “verbose=True” to print only the results of failing doctests.

See euclid.py for a Python 3.x source file that includes doctests, along with a main program, as above, that executes them.

Writing Unit Tests

When we do unit testing, coverage refers to how much of our package’s functionality is checked by the tests we have written. Greater coverage means more effective testing.

Therefore, when writing unit tests for a feature, we probably do not want to write only a single test. Our tests should be comprehensive enough that, if they all pass, then we can have confidence that the feature is implemented correctly.

In particular, consider the following.

When a function takes an object whose type is in some inheritance hierarchy defined in our project, it is common to test using mock objects. A mock object is an object that exists only for testing purposes. It implements whatever interface it needs to, but it may do so in a minimalist manner. Some of its member functions may do logging: keeping track of how they are called.