Testing and Debugging

When we write programs, they don't always work as expected. Sometimes the problems with programs are obvious, but other times they are more subtle. In this lesson, we explore how to test whether a program works correctly.

We will also teach how to use a software tool called a debugger to examine code to find the source of problems. We say that a program is buggy, if it doesn't work as expected and can use a debugger to find bugs.

Testing a program

To check whether a program is correct, we execute it. Let's say that we have written a function to add two numbers. Is executing the add function once enough to ensure that it works as expected? Ideally, we would execute a program with all possible inputs, but that isn't usually realistic. We can't test our add function with all possible combinations of numbers, because we'd have an infinite number of test cases!

Instead of testing with all possible values, we use a set of values that we chose to represent the various categories of inputs. For example, we might try adding two possible numbers, a negative number and a positive number, and so on. We might also test with both integers and floats.

General tips for choosing test cases

  • Size For collections (strings, lists, dictionaries, etc.): 
      * empty collection
  * a collection with one item
  * smallest interesting case
  * collection with several items
  • Dichotomies Depending on the problem, testing dichomomies might be relevant (e.g., vowels/non-vowels, even/odd, positive/negative, empty/full, etc.)
  • Boundaries If a function behaves differently near a particular threshold (e.g., an if statement checking whether a value is 3; 3 is a threshold), then test at that threshold.
  • Order If a function behaves differently when the values are in a different order, identify and test each of those orders.

Example:

Let's test function count_lowercase_vowels:

In [6]:
def count_lowercase_vowels(s):
    """ (str) -> int
    
    Return the number of vowels (a, e, i, o, and u) in s.
    
    >>> count_lowercase_vowels('Happy Anniversary!')
    4
    >>> count_lowercase_vowels('xyz')
    0
    """

To test count_lowercase_vowels, we need to:

  • pick values for the string argument, and
  • call the function to ensure it returns what we expect for each case.

It is not realistic to test using every single possible string argument. Instead, we create relevant categories, and choose one representative string argument from each category. To choose the string argument, consider:

  • the length of the string, and
  • the characters that make up the string.

There are many possible for string lengths. For this example, we'll consider strings that have these lengths:

  • 0 (empty)
  • 1 (single character)
  • 6 (longer)

Which characters should we use? For this example, we'll choose characters based on whether they are vowels or non-vowels. The actual character doesn't matter. If we want a non-vowel, we could use 'b', 'n', '?', or any other character that is not a vowel.

We will make a table of the test cases with the following 3 columns: value of the argument to the function expected return value of the function a description of the test case

s Expected Value Description
'' 0 empty string
'a' 1 single char, vowel
'b' 0 single char, non-vowel
'pfffft' 0 several chars, no vowels
'bandit' 2 several chars, some vowels
'aeioua' 6 several chars, all vowels; note: we included all 5 vowels to ensure each one is properly counted.

We then execute each test case and make sure that the function returns the expected value. We can use the assert function to compare the value returned and the expected value. If the two values differ, an assertion error occurs and you need to investigate the bug.

In [12]:
assert(count_lowercase_vowels('') == 0)
assert(count_lowercase_vowels('a') == 1)
assert(count_lowercase_vowels('b') == 0)
assert(count_lowercase_vowels('pfffft') == 0)
assert(count_lowercase_vowels('bandit') == 2)
assert(count_lowercase_vowels('aeioua') == 6)

Practice Exercise: testing is_teenager

Your task is to choose test cases for function is_teenager:

In [14]:
def is_teenager(age):
   """ (int) -> bool
   Precondition: age >= 0
 
   Return True iff age is a teenager between 13 and 18 inclusive.
   """

Complete the table below, by choosing a set of test cases. Test only with valid input and avoid duplicate tests. The table may contain more rows than necessary.

Test Case Description Value of age Expected result
 
 
 
 
 
 
 

Once you've chosen your tests, check to see whether they catch the bugs in these buggy versions of is_teenager.

Using a Debugger

We've already used the Python Visualizer to visualize memory during Python program execution. The Python Visualizer is a very useful too, but is has some limitations. It can only be used for programs that run without error and on programs of a certain size (there is an upper limit on the number of lines). In addition, it cannot be used with programs that involve reading from or writing to files.

A more common approach for visualizing programs, is to use a debugger, such as the one provided by Wing 101 IDE. We'll now demonstrate how to use Wing's debugger. To review the steps and terminology, you may find this video demonstration helpful.

Here is the code used in our demonstration:

In [18]:
def convert_to_minutes(num_hours):
    """ (int) -> int
    
    Return the number of minutes there are in num_hours hours.
    
    >>> convert_to_minutes(2)
    120
    """
    minutes = num_hours * 60
    return minutes

def convert_to_seconds(num_hours):
    """ (int) -> int
    
    Return the number of seconds there are in num_hours hours.
    
    >>> convert_to_seconds(2)
    7200
    """
    minutes = convert_to_minutes(num_hours)
    seconds = minutes * 60
    return seconds

seconds = convert_to_seconds(2)
seconds = convert_to_seconds(3)
print(seconds)

10800

Practice exercise: using Wing's debugger

Use Wing's debugger to trace the program above. Try setting various breakpoints and using the step into, step over, and step out buttons.