Python Testing Notes

Some Terms

Debugging and Testing

Debugging is generally done by the code's author and is used to check the program's execution process for mismatches between thought process and implementation. Debugging code is usually written alongside the main program and includes error handling and logging. Simple debugging can use print and assert statements, while more complex programs can use the logging module.

Testing, on the other hand, is generally done by a third party. The test code is separated from the program code, and the person writing the tests doesn’t need to understand the exact workings of the program—only whether the program produces the declared output given certain inputs.

Unit Testing

Unit testing is a process that verifies the correctness of a module, function, or class.

The testing method involves writing a set of test cases, where the tester provides various inputs to the module to check if its output is correct, and also verifies if the program correctly identifies and handles invalid inputs by raising exceptions.

Passing all unit tests doesn’t guarantee that the entire program is error-free, but failing any unit test definitely indicates a problem with the program.

Documentation Testing

Documentation provides an overview of a code’s functionality, often with examples that include sample code and expected output. This process is similar to unit testing, except that it's purely descriptive.

Documentation testing is an automated process that finds example code within the documentation, runs it, and compares the output with the documented results.

Integration Testing

Integration testing simulates user behavior by testing the interactions between various modules, ensuring that the program can work properly in a production environment.

Testing in Python

File Structure

In the previous article, I mentioned the src-layout structure, which is also discussed in an official blog post on setuptools. One advantage of this file structure is that the test folder is typically at the same level as src/ instead of the package itself. This way, you need to install the package in a virtual environment before running the tests, avoiding potential discrepancies between test code and user-downloaded content.

<project_name> 
├── LICENSE 
├── pyproject.toml 
├── README.md 
├── src/ 
│ └── <package_name>/ 
│ ├── init.py 
│ └── example.py 
└── tests/

Unit Testing with unittest

unittest is Python’s built-in unit testing library.

A basic test script looks like this:

import unittest

class TestName1(unittest.TestCase):

    def test_sum(self):
        self.assertEqual(sum([1, 2, 3]), 6, "Should be 6")

    def test_sum_tuple(self):
        self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")

class TestName2(unittest.TestCase):

    def test_sum(self):
        self.assertEqual(sum([1, 2, 3]), 6, "Should be 6")

    def test_sum_tuple(self):
        self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")

if __name__ == '__main__':
    unittest.main()

• The test cases are wrapped in a class that inherits from unittest.TestCase.
• All test method names begin with "test". Limited properties are testable as they use built-in methods prefixed with self.. Some of these are in the table below, with [assertTrue(x)](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTrue) and assertRaises(Exception) being especially versatile.
• When run as '__main__', it executes the built-in unittest.main() function.
• Running the script requires specifying the script filename in Python. If you install the nose2 library, you can run python -m nose2, which will automatically discover and execute all tests. (unittest also seems to have an automatic discovery feature.)

Documentation Testing

https://www.liaoxuefeng.com/wiki/1016959663602400/1017605739507840

Python's documentation testing uses the doctest library, as shown below:

class Dict(dict):
    '''
    Simple dict but also supports access as x.y style.

    >>> d1 = Dict()
    >>> d1['x'] = 100
    >>> d1.x
    100
    >>> d1.y = 200
    >>> d1['y']
    200
    >>> d2 = Dict(a=1, b=2, c='3')
    >>> d2.c
    '3'
    >>> d2['empty']
    Traceback (most recent call last):
        ...
    KeyError: 'empty'
    >>> d2.empty
    Traceback (most recent call last):
        ...
    AttributeError: 'Dict' object has no attribute 'empty'
    '''
    def __init__(self, **kw):
        super(Dict, self).__init__(**kw)

    def __getattr__(self, key):
        try:
            return self[key]
        except KeyError:
            raise AttributeError(r"'Dict' object has no attribute '%s'" % key)

    def __setattr__(self, key, value):
        self[key] = value

if __name__ == '__main__':
    import doctest
    doctest.testmod()

With docstring indentation preserved, lines beginning with >>> are treated as tests, with the following lines as expected outputs. For expected errors, you can use ... to omit intermediate error details.

Integration Testing

You can also use unittest for integration testing.

The main difference from unit testing is that integration tests often involve setting up a test dataset. This requires overriding unittest.TestCase.setup().

class TestComplexData(unittest.TestCase):
    def setUp(self):
        # load test data
        self.app = App(database='fixtures/test_complex.json')

    def test_customer_count(self):
        self.assertEqual(len(self.app.customers), 10000)

    def test_existence_of_customer(self):
        customer = self.app.get_customer(id=9999)
        self.assertEqual(customer.name, u"バナナ")
        self.assertEqual(customer.address, "10 Red Road, Akihabara, Tokyo")

if __name__ == '__main__':
    unittest.main()