Introduction to Tests#

A key part of writing code is figuring out why it isn’t working and a key tool for debugging is testing. As code gets more complicated making sure that things stay in working order becomes both increasingly important and increasingly difficult to eyeball.

Part 1: Assertions#

Python provides a builtin assert statement. Conceptually, the assert statement means, “the following must be true”. Or “if the following is not true raise an error.”

The assert statement is a simple statement that takes a boolean expression and an optional message to print if the assertion fails.

The most basic (albeit not terribly useful) assert statement is:

assert True

It is equivalent to:

1if not True:
2    raise AssertionError()

As long as the assertion passes, nothing happens. If it fails though, you’ll see an AssertionError.

Here’s what it looks like when an assert statement fails.

assert False

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 assert False

AssertionError:

And when an assert statement fails with a failure message.

assert False, "Silly, False is not True."

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 assert False, "Silly, False is not True."

AssertionError: Silly, False is not True.

Part 1.1 Examples#

Like an if-statement or a while-loop, the first expression is evaluated in a boolean context. You can use any expression that returns True or False, such as comparison operators or even function calls.

Listing 481 python shell#
1assert 2 + 2 == 4
2assert 3 < 10
3assert "Python".startswith("P")

The boolean context also means that Truthiness and Falsiness apply.

Listing 482 python shell#
1assert "hello", "a non-empty string is Truthy"
2assert "", "an empty string is Falsy"
3assert 100, "an non-zero number is Truthy"
4assert 0, "zero is Falsy"
5assert [1, 2, 3], "a non-empty string is Truthy"
6assert [], "an empty list is Falsy"

Part 1.2: Exercise 1#

Exercise 115 (Assertions)

In a IPython shell, see what happens when you assert the following.

  1. An empty dictionary.

  2. Your name lowercase is equal to your name capitalized.

  3. Save a random number between 1 and 100 to the num variable. Assert that it is less than 50. Have the assert message say The number: num should be less than 50. Repeat with different random numbers until you’ve gotten the assertion to pass and fail at least once each.

Part 2: Unit tests#

While there are many kinds of tests, this lesson will primarily focus on unit tests. A unit tests is one that tests one part of our code–for our purposes a single function.

Part 2.1: A basic unit test#

Say we have the following function.

Listing 484 greeting.py#
1def greeting(name):
2    """Return a welcome message string, including formatted name unless it's blank."""
3    name = name.strip().title()
4    if not name:
5        return "Welcome."
6    else:
7        return f"Welcome {name}."

Here’s a simple unit test for the function.

Listing 485 greeting.py#
 9def test_greeting():
10    assert greeting("buffy") == "Welcome Buffy.", 'should return "Welcome Buffy."'

Let’s break that down.

The assertion on line 10 has three parts.

  • It begins with the assert keyword.

  • The expression greeting("buffy") == "Welcome Buffy." is the heart of the test:

    • It calls the greeting() function with the argument "buffy".

    • The return value is compared to the string "Welcome Buffy" using the == operator which will evaluate to either True or False.

    • The assertion will pass if this evaluates to True. Otherwise, the assertion will fail.

  • The string 'should return "Welcome Buffy."' is a description of what is being tested. This will be shown in the event that the test fails.

Now we can run the test by calling the test function.

Listing 486 greeting.py#
12test_greeting()

Part 2.2: Exercise 2#

Exercise 116 (Unit Test)

1. Copy the following function into a script.

1def endgame(is_winner):
2    """Return a string to tell the player if they won or lost."""
3    if is_winner:
4        return "Congratulations, you won!"
5    else:
6        return "You lost. Better luck next time!"

2. Write a test function to assert that when True is passed to the endgame() function it should return "Congratulations, you won!".

3. Don’t forget to call the test function!

Part 2.3: Testing more cases#

As is often the case, the greeting() function may return different kinds of results depending on its arguments. So we should be sure to test the different behavior that we expect.

Hint

You can put the failure message on the next line by adding a \ at the end of the previous one.

Listing 487 greeting.py#
 9def test_greeting():
10    assert greeting("buffy") == "Welcome Buffy.", \
11     'should return "Welcome Buffy." with the lowercase name capitalized.'
12
13    assert greeting("XANDER") == "Welcome Xander.", \
14        'should return "Welcome Xander." with all caps name capitalized.'
15
16    assert greeting("SpongeBob SquarePants") == "Welcome Spongebob Squarepants.", \
17        'should return "Welcome Spongebob Squarepants." with all words capitalized.'
18
19    assert greeting("") == "Welcome.", \
20        'should return "Welcome." if name is blank.'
21
22    assert greeting("  ") == "Welcome.", \
23        'should return "Welcome." if name is only whitespace.'

Testing can also be a sort of documentation. For example, what would happen if we pass a number to the greeting() function?

Listing 488 greeting.py#
 9def test_greeting():
10    assert greeting("buffy") == "Welcome Buffy.", \
11     'should return "Welcome Buffy." with the lowercase name capitalized.'
12
13    assert greeting("XANDER") == "Welcome Xander.", \
14        'should return "Welcome Xander." with all caps name capitalized.'
15
16    assert greeting("SpongeBob SquarePants") == "Welcome Spongebob Squarepants.", \
17        'should return "Welcome Spongebob Squarepants." with all words capitalized.'
18
19    assert greeting("") == "Welcome.", \
20        'should return "Welcome." if name is blank.'
21
22    assert greeting("  ") == "Welcome.", \
23        'should return "Welcome." if name just whitespace.'
24
25    assert greeting("42") == "Welcome 42.", \
26        'should return "Welcome 42." with no special handling for numbers.'

Part 2.4: Exercise 3#

Exercise 117 (Detailed Unit Test)

1. Copy the following function into a script.

 1def letter_grade(score):
 2    """Return the letter grade for a particular number score"""
 3    ranges = {
 4        (90, 100): "A",
 5        (80, 89): "B",
 6        (70, 79): "C",
 7        (60, 69): "D",
 8        (0, 59): "F",
 9    }
10
11    for score_range, letter in ranges.items():
12        min_score, max_score = score_range
13        if score >= min_score and score <= max_score:
14            return letter
15    
16    return False

2. Write a test function that includes assertions for each of the following arguments:

  • a score for that returns each of the letters A, B, C, D, and F

  • that 110 returns False

  • that -5 returns False

Part 3: Pytest#

While it’s easy to run tests by calling them from inside a script, it’s more common to use a test runner.

A test runner is a CLI tool that will run your tests and print a nicely formatted report of the results. Runners also often include a library that can be imported in your tests to provide tools and/or frameworks to aid in writing tests.

Part 3.1: Install pytest#

For this lesson we’ll be using the pytest runner. It’s a Python module, so you can install it as you normally do modules.

For poetry users:

Listing 489 command line#
poetry add --dev pytest

Otherwise:

Listing 490 command line#
python -m pip install pytest

Part 3.2: Migrate to Pytest#

To use the runner part of pytest we only have to make a minor change to the script to make it work with pytest. Simply put your main() call under a if __name__ == "__main__" statement. This allows pytest to import your file as a module without running the script by calling main().

We also no longer need to call the test_greeting() function in the script itself, because pytest will handle that.

Listing 491 greeting.py#
33if __name__ == "__main__":
34    # test_greeting()
35    main()

Now you can run the tests at the command line with the pytest command followed by the filename.

Listing 492 command line#
pytest greeting.py
Listing 493 output#
=================== test session starts ====================
platform darwin -- Python 3.8.1, pytest-6.2.1, py-1.10.0, ...
rootdir: ...
collected 1 item                                           

greeting.py .                                        [100%]

==================== 1 passed in 0.08s =====================

Let’s add a failing assertion so we can see what that looks like.

Listing 494 greeting.py#
 9def test_greeting():
10    assert greeting("buffy") == "", \
11        'demo of a test failure'
12
13    assert greeting("buffy") == "Welcome Buffy.", \
14        'should return "Welcome Buffy." with the lowercase name capitalized.'
15
16    assert greeting("XANDER") == "Welcome Xander.", \
17        'should return "Welcome Xander." with all caps name capitalized.'
18
19    assert greeting("SpongeBob SquarePants") == "Welcome Spongebob Squarepants.", \
20        'should return "Welcome Spongebob Squarepants." with all words capitalized.'
21
22    assert greeting("") == "Welcome.", \
23        'should return "Welcome." if name is blank.'
24
25    assert greeting("  ") == "Welcome.", \
26        'should return "Welcome." if name just whitespace.'
27
28    assert greeting("42") == "Welcome 42.", \
29        'should return "Welcome 42." no special handling for numbers.'

Now rerun the tests at the command line.

Listing 495 command line#
pytest greeting.py
Listing 496 output#
=================== test session starts ====================
platform darwin -- Python 3.8.1, pytest-6.2.1, py-1.10.0, ...
rootdir: ...
collected 1 item                                           

greeting.py F                                        [100%]

========================= FAILURES =========================
______________________ test_greeting _______________________

    def test_greeting():
>       assert greeting("buffy") == "", \
            'demo of a test failure'
E       AssertionError: demo of a test failure
E       assert 'Welcome Buffy.' == ''
E         + Welcome Buffy.

greeting.py:10: AssertionError
================= short test summary info ==================
FAILED greeting.py::test_greeting - AssertionError: demo ...
==================== 1 failed in 0.22s =====================

Part 4: Test best practices#

Part 4.1: Make a test file#

In Python tests are usually kept in a separate file starting with test_.

Let’s move our test_greeting() function to a new file named test_greeting.py file.

Now that it’s in a separate file, we’ll need to import the greeting function from our greeting.py file.

Listing 497 test_greeting.py#
 1from greeting import greeting
 2
 3def test_greeting():
 4    assert greeting("buffy") == "", \
 5        'demo of a test failure'
 6
 7    assert greeting("buffy") == "Welcome Buffy.", \
 8        'should return "Welcome Buffy." with the lowercase name capitalized.'
 9
10    assert greeting("XANDER") == "Welcome Xander.", \
11        'should return "Welcome Xander." with all caps name capitalized.'
12
13    assert greeting("SpongeBob SquarePants") == "Welcome Spongebob Squarepants.", \
14        'should return "Welcome Spongebob Squarepants." with all words capitalized.'
15
16    assert greeting("") == "Welcome.", \
17        'should return "Welcome." if name is blank.'
18
19    assert greeting("  ") == "Welcome.", \
20        'should return "Welcome." if name just whitespace.'
21
22    assert greeting("42") == "Welcome 42.", \
23        'should return "Welcome 42." no special handling for numbers.'

To run the tests we’ll use test_greeting.py for the filename instead of greeting.py.

Listing 498 command line#
pytest test_greeting.py
Listing 499 output#
=================== test session starts ====================
platform darwin -- Python 3.8.1, pytest-6.2.1, py-1.10.0, ...
rootdir: ...
collected 1 item                                           

greeting.py F                                        [100%]

========================= FAILURES =========================
______________________ test_greeting _______________________

    def test_greeting():
>       assert greeting("buffy") == "", \
            'demo of a test failure'
E       AssertionError: demo of a test failure
E       assert 'Welcome Buffy.' == ''
E         + Welcome Buffy.

greeting.py:10: AssertionError
================= short test summary info ==================
FAILED greeting.py::test_greeting - AssertionError: demo ...
==================== 1 failed in 0.22s =====================

Part 4.2: One case per test function#

One test function with a bunch of assert messages is fine for a quick and dirty test. There are a few downsides though. For one thing, if one assertion fails, none of the others in the function will be run. It also can make it a bit more difficult to tell exactly which test failed.

It’s generally a good idea to have one use case per function.

Let’s split the test_greeting() function up.

Listing 500 test_greeting.py#
 1from greeting import greeting
 2
 3def test_greeting_fail():
 4    assert greeting("buffy") == "", \
 5        'demo of a test failure'
 6
 7
 8def test_greeting_lower():
 9    assert greeting("buffy") == "Welcome Buffy.", \
10        'should return "Welcome Buffy." with the lowercase name capitalized.'
11
12
13def test_greeting_upper_to_title():
14    assert greeting("XANDER") == "Welcome Xander.", \
15        'should return "Welcome Xander." with all caps name capitalized.'
16
17
18def test_greeting_multi_word():
19    assert greeting("SpongeBob SquarePants") == "Welcome Spongebob Squarepants.", \
20        'should return "Welcome Spongebob Squarepants." with all words capitalized.'
21
22
23def test_greeting_empty_string():
24    assert greeting("") == "Welcome.", \
25        'should return "Welcome." if name is empty.'
26
27
28def test_greeting_blank():
29    assert greeting("  ") == "Welcome.", \
30        'should return "Welcome." if name just whitespace.'
31
32
33def test_greeting_number():
34    assert greeting("42") == "Welcome 42.", \
35        'should return "Welcome 42." no special handling for numbers.'

Now we can run the tests using the -v flag to get verbose output. This will show us the status of each individual test.

Listing 501 command line#
pytest -v test_greeting.py
Listing 502 output#
=================== test session starts ====================
platform darwin -- Python 3.8.1, pytest-6.2.2, ...
rootdir: ..., configfile: pyproject.toml, testpaths: tests
collected 7 items                                          

tests/test_greeting.py F......                       [100%]

========================= FAILURES =========================
____________________ test_greeting_fail ____________________

    def test_greeting_fail():
>       assert greeting("buffy") == "", \
            'demo of a test failure'
E       AssertionError: demo of a test failure
E       assert 'Welcome Buffy.' == ''
E         + Welcome Buffy.

tests/test_greeting.py:4: AssertionError
================= short test summary info ==================
FAILED tests/test_greeting.py::test_greeting_fail - Asser...
=============== 1 failed, 6 passed in 0.20s ================

Part 4.3: File locations#

For simple projects with just one or two Python modules it’s fine to keep your test files in the same directory (folder) as your python files. However, there’s a standard structure that is recommended for Python projects.

Step 1: Move your files#

If you use Poetry and started your project with the poetry new command, then the correct directory structure was already created for you and looks something like this.

Listing 503 example directory layout#
├── README.md
├── poetry.lock
├── pyproject.toml
├── testing_demo
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_testing_demo.py

I won’t go into all of the details, but here are the things you need to know that are related to tests:

  • Your Python files should be in a directory with your project name in lowercase_with_underscores style.

  • Python files should also be named with lowercase_with_underscores style.

  • Your test files should be in a directory named tests.

  • Each test file should start with test_ and usually correlate with the file that contains the code being tested, also in lowercase_with_underscores style.

  • Both __init__.py files can be empty, they just need to exist.

Step 2: Change your import in your test file#

Once you’ve moved your files where they should be, you’ll need to change the import statement in your test file so that it includes the name of the parent directory followed by a . before the module name.

Listing 504 test_greeting.py#
from pythonclass.greeting import greeting

Step 3: Change any local imports#

If you are importing any code from your own files you will need to change the way they are imported so that pytest can find them.

Here’s an example of how you would change an import from a private.py file.

BEFORE

Listing 505 local import example#
from private import KEY, TOKEN

AFTER

Listing 506 local import example#
from .private import KEY, TOKEN

The . in front of private means that the module is part of the same package. Which is essentially the same as saying that the file is in same directory.

Step 4: Run the tests#

Now you can run all the test files in that directory at the same time by using the directory name instead of a specific file.

Listing 507 command line#
pytest -v tests
Listing 508 output#
=================== test session starts ====================
platform darwin -- Python 3.8.1, pytest-6.2.1, py-1.10.0, ...
cachedir: .pytest_cache
rootdir: ...
collected 7 items                                          

test_greeting.py::test_greeting_fail FAILED          [ 14%]
test_greeting.py::test_greeting PASSED               [ 28%]
test_greeting.py::test_greeting_upper_to_title PASSED [ 42%]
test_greeting.py::test_greeting_multi_word PASSED    [ 57%]
test_greeting.py::test_greeting_empty_string PASSED  [ 71%]
test_greeting.py::test_greeting_blank PASSED         [ 85%]
test_greeting.py::test_greeting_number PASSED        [100%]

========================= FAILURES =========================
____________________ test_greeting_fail ____________________

    def test_greeting_fail():
>       assert greeting("buffy") == "", \
            'demo of a test failure'
E       AssertionError: demo of a test failure
E       assert 'Welcome Buffy.' == ''
E         + Welcome Buffy.

test_greeting.py:4: AssertionError
================= short test summary info ==================
FAILED test_greeting.py::test_greeting_fail - AssertionEr...
=============== 1 failed, 6 passed in 0.21s ================

Part 4.4: Configure pytest#

See also

If you use Poetry, you can configure pytest by adding a tool.pytest.ini_options section to your pyproject.toml file.

Listing 509 pyproject.toml#
# ...

[tool.pytest.ini_options]
testpaths = ["tests"]

Alternately you can add a pytest.ini file to your project root directory.

Listing 510 pytest.ini#
testpaths =
    tests

This will ensure that pytest is always run using the tests directory without needing to include the path in your command.

Listing 511 command line#
pytest -v
Listing 512 output#
=================== test session starts ====================
platform darwin -- Python 3.8.1, pytest-6.2.1, py-1.10.0, ...
cachedir: .pytest_cache
rootdir: ...
collected 7 items                                          

test_greeting.py::test_greeting_fail FAILED          [ 14%]
test_greeting.py::test_greeting PASSED               [ 28%]
test_greeting.py::test_greeting_upper_to_title PASSED [ 42%]
test_greeting.py::test_greeting_multi_word PASSED    [ 57%]
test_greeting.py::test_greeting_empty_string PASSED  [ 71%]
test_greeting.py::test_greeting_blank PASSED         [ 85%]
test_greeting.py::test_greeting_number PASSED        [100%]

========================= FAILURES =========================
____________________ test_greeting_fail ____________________

    def test_greeting_fail():
>       assert greeting("buffy") == "", \
            'demo of a test failure'
E       AssertionError: demo of a test failure
E       assert 'Welcome Buffy.' == ''
E         + Welcome Buffy.

test_greeting.py:4: AssertionError
================= short test summary info ==================
FAILED test_greeting.py::test_greeting_fail - AssertionEr...
=============== 1 failed, 6 passed in 0.21s ================

Part 5: Writing code for testing#

One of the benefits of writing tests it that it encourages you to write better code. However, this can take a bit of getting used to. So in this part of the lesson we’ll go over a few tips for writing code so that it’s easier to test.

Part 5.1: Keep your interface separate#

Things like input() and print() statements are not easy to tests via unit tests. So keep the parts that take user input or display output to the user separate from the functions that determine behavior.

  • Put any calls to input() and print() in a main() function that does little else, and calls to other functions.

  • Anywhere else, use return instead of print and arguments instead of input().

BEFORE

Listing 513 palindrome.py#
def main():
    """Ask the user for text, then print a message telling the user if it is
       an palindrome or not."""

    text = input("Enter a word to determine if it's an palindrome: ")
    if text == "".join(reversed(text)):
        print(f'Yes, "{text}" is an palindrome.')
    else:
        print(f'No, "{text}" is not an palindrome.')


main()

AFTER

Listing 514 palindrome.py#
def is_palindrome(text):
    """Return True if text is the same forward and backwards."""

    return text == "".join(reversed(text))

def message(isit, text):
    if isit:
        msg = f'Yes, "{text}" is a palindrome.'
    else:
        msg = f'No, "{text}" is not a palindrome.'

    return msg


def main():
    """Ask the user for text, then print a message telling the user if it is
       an palindrome or not."""

    text = input("Enter a word to determine if it's an palindrome: ")
    word_is_palindrome = is_palindrome(text)
    output = message(word_is_palindrome, text)
    print(output)

if __name__ == "__main__":
    main()
Listing 515 test_palindrome.py#
 1from pythonclass.lessons.palindrome import is_palindrome, message
 2
 3def test_is_palindrome_true():
 4    assert is_palindrome("radar"), \
 5        "should return True if text is the same forwards and backwards"
 6
 7
 8def test_is_palindrome_false():
 9    assert not is_palindrome("something"), \
10        "should return False if text is not the same forwards and backwards"
11
12
13def test_message_no():
14    assert message(False, "nope") == 'No, "nope" is not a palindrome.', \
15        "should return a message saying text is not a palindrome if isit is False"
16
17
18
19def test_message_yes():
20    assert message(True, "level") == 'Yes, "level" is a palindrome.', \
21        "should return a message saying text is a palindrome if isit is True"

Part 5.2: Isolate external services / dependencies#

When running tests you rarely want to modify real data or make live calls to external services. Instead, separate the code that makes those calls from the code that deals with the resulting data.

BEFORE

Listing 516 palindrome.py#
 1import requests
 2
 3def main():
 4    """Print the local weather"""
 5
 6    response = requests.get("http://wttr.in/", params={"format": "j1"})
 7    weather = response.json()
 8    print("Current weather")
 9    print("---------------")
10    print(weather["current_condition"][0]["temp_F"],
11     weather["current_condition"][0]["weatherDesc"][0]["value"])
12
13main()

AFTER

Listing 517 weather.py#
 1import requests
 2
 3def get_weather(data):
 4    """Return a dictionary containing the temperature and description from
 5       wttr.in response data"""
 6    conditions = {}
 7    conditions["temp"] = data["current_condition"][0]["temp_F"]
 8    conditions["desc"] = data["current_condition"][0]["weatherDesc"][0]["value"]
 9
10    return conditions
11
12def format_weather(temp, desc):
13    """Return the formatted weather string to display."""
14    text  = "Current weather\n"
15    text += "---------------\n"
16    text += f"{temp} {desc}\n"
17    return text
18
19def main():
20    """Print the local weather"""
21
22    response = requests.get("http://wttr.in/", params={"format": "j1"})
23    weather = get_weather(response.json())
24    text = format_weather(weather["temp"], weather["desc"])
25    print(text)
26
27if __name__ == "__main__":
28    main()
Listing 518 test_weather.py#
 1from pythonclass.lessons.weather import format_weather, get_weather
 2
 3def test_format_weather():
 4
 5    text = """Current weather
 6---------------
 7-25 Overcast
 8"""
 9
10    assert format_weather(-25, "Overcast") == text, \
11         "should return formatted weather"
12
13def test_get_weather():
14    data = {
15        "current_condition": [
16            {
17                "temp_F": -29,
18                "weatherDesc": [
19                    {
20                        "value": "Overcast"
21                    }
22                ]
23            }
24        ]
25    }
26
27    assert get_weather(data) == {"temp": -29, "desc": "Overcast"}, \
28       "should extract a dict with temp and desc from request data"
29
30
31def test_get_weather_from_file():
32    testdir = Path(__file__).parent
33    filepath = testdir.joinpath("weather.json")
34
35    fp = open(filepath)
36    data = json.load(fp)
37    fp.close()
38
39    assert get_weather(data) == {"temp": "27", "desc": "Partly cloudy"}, \
40       "should extract a dict with temp and desc from request data"

Part 6: Testing in VS Code#

Part 6.1: Setup#

From the Command Palette select Python: Configure Tests.

Select pytest from the dropdown.

Select the tests directory from the dropdown.

Part 6.2: Running tests#

Step 1: Click Tests#

Click the Test icon from the activity bar.

Step 2: Review list of tests#

You will see the TESTING sidebar panel, where your tests will be listed grouped by directory and file.

Step 3: Run tests#

You can run all tests by selecting Python: Run All Tests from the Command Palette

Or clicking the play icon at the top of the TESTING sidebar.

Once your tests have run the failing and passing tests will be marked accordingly.

Step 4: View test output#

Details about any test failures as well as any other pytest output can be seen in the Python Test Log which you can access by selecting Python: Show Test Output from the Command Palette:

Or by clicking the icon at the top of the TESTING sidebar:

Or by clicking the OUTPUT tab on the panel then selecting Python Test Log from the dropdown.

Part 6.3: Changing tests#

If you make changes to your tests, you may need to nudge VS Code so that they are reflected.

You can run Python: Discover Tests from the Command Palette:

Or by clicking the icon at the top of the TESTING sidebar:

See also#

See also