useful resources:
- https://stackabuse.com/test-driven-development-with-pytest/
- https://docs.pytest.org/en/latest/goodpractices.html#conventions-for-python-test-discovery
- https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure
- https://github.com/vanzaj/tdd-pytest/blob/master/docs/tdd-pytest/content/tdd-basics.md
- https://opensource.com/article/18/6/pytest-plugins
setup
- install
pytest - install
pytest-sugarwhich will give us nicer output
pip -q install pytest pytest-sugar
# move to tdd directory
from pathlib import Path
if Path.cwd().name != 'tdd':
%mkdir tdd
%cd tdd
%pwd
# cleanup all files
%rm *.py
How pytest discovers tests
pytests uses the following conventions to automatically discovering tests:
- files with tests should be called
test_*.pyor*_test.py - test function name should start with
test_
our first test
to see if our code works, we can use the assert python keyword. pytest adds hooks to assertions to make them more useful
%%file test_math.py
import math
def test_add():
assert 1+1 == 2
def test_mul():
assert 6*7 == 42
def test_sin():
assert math.sin(0) == 0
now lets run pytest
!python -m pytest test_math.py
Great! we just wrote 3 tests that shows that basic math still works
Hurray!
your turn
write a test for the following function.
if there is a bug in the function, fix it
%%file make_triangle.py
# version 1
def make_triangle(n):
"""
draws a triangle using '@' letters
for instance:
>>> print('\n'.join(make_triangle(3))
@
@@
@@@
"""
for i in range(n):
yield '@' * i
solution
%%file test_make_triangle.py
from make_triangle import make_triangle
def test_make_triangle():
expected = "@"
actual = '\n'.join(make_triangle(1))
assert actual == expected
!python -m pytest test_make_triangle.py
so the expected starts with '@' and the actual starts with '' β¦
this is a bug! lets fix the code and re-run
%%file make_triangle.py
# version 2
def make_triangle(n):
"""
draws a triangle using '@' letters
for instance:
>>> print('\n'.join(make_triangle(3))
@
@@
@@@
"""
for i in range(1, n+1):
yield '@' * i
!python -m pytest test_make_triangle.py
Pytest context-sensitive comparisons
pytest has rich support for providing context-sensitive information when it encounters comparisons.
Special comparisons are done for a number of cases:
- comparing long strings: a context diff is shown
- comparing long sequences: first failing indices
- comparing dicts: different entries
Hereβs how this looks like for set:
%%file test_compare_fruits.py
def test_set_comparison():
set1 = set(['Apples', 'Bananas', 'Watermelon', 'Pear', 'Guave', 'Carambola', 'Plum'])
set2 = set(['Plum', 'Apples', 'Grapes', 'Watermelon','Pear', 'Guave', 'Carambola', 'Melon' ])
assert set1 == set2
!python -m pytest test_compare_fruits.py
your turn
test the following function count_words() and fix any bugs.
the expected output from the function is given in expected_output
expected_output = {
'and': 2,
'chief': 2,
'didnt': 1,
'efficiency': 1,
'expected': 1,
'expects': 1,
'fear': 2,
'i': 1,
'inquisition': 2,
'is': 1,
'no': 1,
'one': 1,
'our': 1,
'ruthless': 1,
'spanish': 2,
'surprise': 3,
'the': 2,
'two': 1,
'weapon': 1,
'weapons': 1,
'well': 1}
%%file spanish_inquisition.py
# version 1: buggy
import collections
quote = """
Well, I didn't expected the Spanish Inquisition ...
No one expects the Spanish Inquisition!
Our chief weapon is surprise, fear and surprise;
two chief weapons, fear, surprise, and ruthless efficiency!
"""
def remove_punctuation(quote):
quote.translate(str.maketrans('', '', "',.!?;")).lower()
return quote
def count_words(quote):
quote = remove_punctuation(quote)
return dict(collections.Counter(quote.split(' ')))
solution
%%file test_spanish_inquisition.py
from spanish_inquisition import *
expected_output = {
'and': 2,
'chief': 2,
'didnt': 1,
'efficiency': 1,
'expected': 1,
'expects': 1,
'fear': 2,
'i': 1,
'inquisition': 2,
'is': 1,
'no': 1,
'one': 1,
'our': 1,
'ruthless': 1,
'spanish': 2,
'surprise': 3,
'the': 2,
'two': 1,
'weapon': 1,
'weapons': 1,
'well': 1}
def test_spanish_inquisition():
actual = count_words(quote)
assert actual == expected_output
!python -m pytest -vv test_spanish_inquisition.py
%%file spanish_inquisition.py
# version 2: fixed
import collections
quote = """
Well, I didn't expected the Spanish Inquisition ...
No one expects the Spanish Inquisition!
Our chief weapon is surprise, fear and surprise;
two chief weapons, fear, surprise, and ruthless efficiency!
"""
def remove_punctuation(quote):
# quote.translate(str.maketrans('', '', "',.!?;")).lower() # BUG: missing return
return quote.translate(str.maketrans('', '', "',.!?;")).lower()
def count_words(quote):
quote = remove_punctuation(quote)
# return dict(collections.Counter(quote.split(' '))) # BUG
return dict(collections.Counter(quote.split()))
!python -m pytest -vv test_spanish_inquisition.py
Using fixtures to simplify tests
Motivating example
Lets look at an example of class Person, where each person has a name and remembers their friends.
%%file person.py
#version 1
class Person:
def __init__(self, name, favorite_color, year_born):
self.name = name
self.favorite_color = favorite_color
self.year_born = year_born
self.friends = set()
def add_friend(self, other_person):
if not isinstance(other_person, Person): raise TypeError(other_person, 'is not a', Person)
self.friends.add(other_person)
other_person.friends.add(self)
def __repr__(self):
return f'Person(name={self.name!r}, ' \
f'favorite_color={self.favorite_color!r}, ' \
f'year_born={self.year_born!r}, ' \
f'friends={[f.name for f in self.friends]})'
Lets write a test for add_friend() function.
notice how the setup for the test is taking so much of the function, while also requiring inventing a lot of repetitious data
is there a way to reduce this boiler plate code
%%file test_person.py
from person import Person
def test_name():
# setup
terry = Person(
'Terry Gilliam',
'red',
1940
)
# test
assert terry.name == 'Terry Gilliam'
def test_add_friend():
# setup for the test
terry = Person(
'Terry Gilliam',
'red',
1940
)
eric = Person(
'Eric Idle',
'blue',
1943
)
# actual test
terry.add_friend(eric)
assert eric in terry.friends
assert terry in eric.friends
!python -m pytest -q test_person.py
Fixtures to the rescue
what is we had a magic factory that can conjure up a name, favorite color and birth year?
then we could write our test_name() more simply like this:
def test_name(person_name, favorite_color, birth_year):
person = Person(person_name, favorite_color, birth_year)
# test
assert person.name == person_name
furthermore, if we had a magic factory that can create terry and eric we could write our test_add_friend() function like this:
def test_add_friend(eric, terry):
eric.add_friend(terry)
assert eric in terry.friends
assert terry in eric.friends
fixtures in pytest allow us to create such magic factories using the @pytest.fixture notation.
hereβs an example:
%%file test_person_fixtures1.py
import pytest
from person import Person
@pytest.fixture
def person_name():
return 'Terry Gilliam'
@pytest.fixture
def birth_year():
return 1940
@pytest.fixture
def favorite_color():
return 'red'
def test_person_name(person_name, favorite_color, birth_year):
person = Person(person_name, favorite_color, birth_year)
# test
assert person.name == person_name
!python -m pytest test_person_fixtures1.py
whatβs happening here?
pytest sees that the test function test_person_name(person_name, favorite_color, birth_year) requires three parameters, and searches for fixtures annotated with @pytest.fixture with the same name.
when it finds them, it calls these fixtures on our behalf, and passes the return value as the parameter. in effect, it calls
test_person_name(person_name=person_name(), favorite_color=favorite_color(), birth_year=birth_year()
note how much code this saves
your turn
- rewrite the
test_add_friendfunction to accept two parametersdef test_add_friend(eric, terry) - write fixtures for eric and terry
- run pytest
solution
%%file test_person_fixtures2.py
import pytest
from person import Person
@pytest.fixture
def eric():
return Person('Eric Idle', 'red', 1943)
@pytest.fixture
def terry():
return Person('Terry Gilliam', 'blue', 1940)
def test_add_friend(eric, terry):
eric.add_friend(terry)
assert eric in terry.friends
assert terry in eric.friends
!python -m pytest -q test_person_fixtures2.py
parameterizing fixtures
Fixture functions can be parametrized in which case they will be called multiple times, each time executing the set of dependent tests, i. e. the tests that depend on this fixture.
Test functions usually do not need to be aware of their re-running. Fixture parametrization helps to write exhaustive functional tests for components which themselves can be configured in multiple ways.
%%file test_primes.py
import pytest
import math
def is_prime(x):
return all(x % factor != 0 for factor in range(2, int(x/2)))
@pytest.fixture(params=[2,3,5,7,11, 13, 17, 19, 101])
def prime_number(request):
return request.param
def test_prime(prime_number):
assert is_prime(prime_number) == True
!python -m pytest --verbose test_primes.py
your turn
test is_prime() for non prime numbers
bonus: can you find and fix the bug in
is_prime()using a test?
solution
%%file test_non_primes.py
import pytest
FIX_BUG = True
if FIX_BUG:
def is_prime_fixed(x):
# notice the +1 - it is important when x=4
return all(x % factor != 0 for factor in range(2, int(x/2) + 1))
is_prime = is_prime_fixed
else:
from test_primes import is_prime
@pytest.fixture(params=[4, 6, 8, 9, 10, 12, 14, 15, 16, 28, 60, 100])
def non_prime_number(request):
return request.param
def test_non_primes(non_prime_number):
assert is_prime(non_prime_number) == False
!python -m pytest --verbose test_non_primes.py
all([factor for factor in range(2, int(4/2))])
!python -m pytest --verbose test_primes.py
printing and logging within tests
printing
You can use prints within tests to provide additional debug info.
pytest redirects the output and captured the output of each test. it then:
- suppresses the output of all successful tests (for brevity)
- shows the output off all failed tests (for debugging)
- both
stdoutandstderrare captured
%%file test_prints.py
import sys
def test_print_success():
print(
"""
@@@@@@@@@@@@@@@
this statement will NOT be printed
@@@@@@@@@@@@@@@
"""
)
assert 6*7 == 42
def test_print_fail():
print(
"""
@@@@@@@@@@@@@@@
this statement WILL be printed
@@@@@@@@@@@@@@@
"""
)
assert True == False
def test_stderr_capture_success():
print(
"""
@@@@@@@@@@@@@@@
this STDERR statement will NOT be printed
@@@@@@@@@@@@@@@
""",
file=sys.stderr
)
assert True
def test_stderr_capture_fail():
print(
"""
@@@@@@@@@@@@@@@
this STDERR statement WILL be printed
@@@@@@@@@@@@@@@
""",
file=sys.stderr
)
assert False
!python -m pytest -q test_prints.py
logging
pytest captures log messages of level WARNING or above automatically and displays them in their own section for each failed test in the same manner as captured stdout and stderr.
- WARNING and above will displayed for failed tests
- INFO and below will not be displayed
example:
%%file test_logging.py
import logging
logger = logging.getLogger(__name__)
def test_logging_warning_success():
logger.warning('\n\n @@@ this will NOT be printed \n\n')
assert True
def test_logging_warning_fail():
logger.warning('\n\n @@@ this WILL be printed @@@ \n\n')
assert False
def test_logging_info_fail():
logger.info('\n\n @@@ this will NOT be printed @@@ \n\n')
assert False
!python -m pytest test_logging.py
your turn
We give below an implementation of the FizzBuzz puzzle:
Write a function that returns the numbers from 1 to 100. But for multiples of three returns βFizzβ instead of the number and for the multiples of five returns βBuzzβ. For numbers which are multiples of both three and five return βFizzBuzzβ.
thus this SHOULD be true
>>> fizzbuzz() # should return the following (abridged) output
[1, 2, 'Fizz', 4, 'Buzz', 6, 7, 8, 'Fizz', 'Buzz', 11, 'Fizz', 13, 14, 'FizzBuzz', ... ]
BUT the implementation is buggy. can you write tests for it and fix it?
%%file fizzbuzz.py
def is_multiple(n, divisor):
return n % divisor == 0
def fizzbuzz():
"""
expected output: list with elements numbers
[1, 2, 'Fizz', 4, 'Buzz', 6, 7, 8, 'Fizz', 'Buzz', 11, 'Fizz', 13, 14, 'FizzBuzz', ... ]
"""
result = []
for i in range(100):
if is_multiple(i, 3):
return "Fizz"
elif is_multiple(i, 5):
return "Buzz"
elif is_multiple(i, 3) and is_multiple(i, 5):
return "FizzBuzz"
else:
return i
return result
solution
%%file test_fizzbuzz.py
FIX_BUG = 1
if not FIX_BUG:
from fizzbuzz import fizzbuzz
else:
def fizzbuzz_fixed():
def translate(i):
if i%3 == 0 and i%5 == 0:
return "FizzBuzz"
elif i%3 == 0:
return "Fizz"
elif i%5 == 0:
return "Buzz"
else:
return i
return [translate(i) for i in range(1, 100+1)]
fizzbuzz = fizzbuzz_fixed
import pytest
@pytest.fixture
def fizzbuzz_result():
result = fizzbuzz()
print(result)
return result
@pytest.fixture
def fizzbuzz_dict(fizzbuzz_result):
return dict(enumerate(fizzbuzz_result, 1))
def test_fizzbuzz_len(fizzbuzz_result):
assert len(fizzbuzz_result) == 100
def test_fizzbuzz_len(fizzbuzz_result):
assert type(fizzbuzz_result) == list
def test_fizzbuzz_first_element(fizzbuzz_dict):
assert fizzbuzz_dict[1] == 1
def test_fizzbuzz_3(fizzbuzz_dict):
assert fizzbuzz_dict[3] == 'Fizz'
def test_fizzbuzz_5(fizzbuzz_dict):
assert fizzbuzz_dict[5] == 'Buzz'
def test_fizzbuzz_15(fizzbuzz_dict):
assert fizzbuzz_dict[15] == 'FizzBuzz'
!python -m pytest test_fizzbuzz.py
float: when things are (almost) equal
consider the following code, what do you expect the result to be?
x = 0.1 + 0.2
y = 0.3
print('x == y', x ==y) # what will it print?
x = 0.1 + 0.2
y = 0.3
print('x == y:', x == y) # what will it print?
if you had anticipated True it means you havenβt tried testing code with float data yet
print(x, '!=', y)
the issue is that float is approxiamtely accurate (enough for most calculations) but may have small rounding errors.
hereβe a common but ugly way to test for float equivalence
abs((0.1 + 0.2) - 0.3) < 1e-6
hereβs a more pythonic and pytest-tic way, using pytest.approx
from pytest import approx
0.1 + 0.2 == approx(0.3)
your turn
test that
math.sin(0) == 0,math.sin(math.pi / 2) == 1math.sin(math.pi) == 0math.sin(math.pi * 3/2) == -1math.sin(math.pi * 2) == 0
solution
%%file test_sin.py
from pytest import approx
import math
def test_sin():
assert math.sin(0) == 0
assert math.sin(math.pi / 2) == 1
assert math.sin(math.pi) == approx(0)
assert math.sin(math.pi * 3/2) == approx(-1)
assert math.sin(math.pi * 2) == approx(0)
!python -m pytest test_sin.py
adding timeouts to tests
Sometimes code gets stuck in an infinite loop, or waiting for a response from a server. Sometimes, tests that run too long is in itself an indication of failure.
how can we add timeouts to tests to avoid getting stuck?
the package pytest-timeout solves for that by providing a plugin to pytest.
- install the package using
pip install pytest-timeout - you can set timeouts individually on tests by marking them with the
@pytest.mark.timeout(timeout=60)decorator - you can set the timeout for all tests globally by using the timeout commandline parameter for pytest, like so:
pytest --timeout=300
pip install -q pytest-timeout
%%file test_timeouts.py
import pytest
@pytest.mark.timeout(5)
def test_infinite_sleep():
import time
while True:
time.sleep(1)
print('sleeping ...')
def test_empty():
pass
!python -m pytest --verbose test_timeouts.py
notice how the test_empty test still runs and passes, even though the previous test was aborted
your turn
- use the
requestsmodule to.get()the url http://httpstat.us/101 and call.raise_for_status() - since this will hang forever, use a timeout on the test so that it fails after 5 seconds
- since the test is guranteed to fail, mark it with the
xfail(expected fail) annotation@pytest.mark.xfail(reason='timeout')
%%file test_http101_timeout.py
import pytest
import requests
@pytest.mark.xfail(reason='timeout')
@pytest.mark.timeout(2)
def test_http101_timeout():
response = requests.get('http://httpstat.us/101')
response.raise_for_status()
!python -m pytest test_http101_timeout.py
testing for exceptions
consider the following code fragment from person.py:
class Person:
def add_friend(self, other_person):
if not isinstance(other_person, Person) raise TypeError(other_person, 'is not a', Person)
self.friends.add(other_person)
other_person.friends.add(self)
the add_friend() method will raise an exception if it is used with a parameter which is not a Person
how can we test this?
if we wrap the code that is supposed to throw the exc
%%file test_add_person_exception.py
from person import Person
from test_person_fixtures2 import *
def test_add_person_exception(terry):
with pytest.raises(TypeError):
terry.add_friend("a shrubbey!")
def test_add_person_exception_detailed(terry):
with pytest.raises(TypeError) as excinfo:
terry.add_friend("a shrubbey!")
assert 'Person' in str(excinfo.value)
@pytest.mark.xfail(reason='expected to fail')
def test_add_person_no_exception(terry, eric):
with pytest.raises(TypeError): # is expecting an exception that won't happen
terry.add_friend(eric) # this does not throw an exception
!python -m pytest test_add_person_exception.py
your turn
use the requests module and the .raise_for_status() method
- test that
.raise_for_statuswill raise an exception when accessing the following URLs:- http://httpstat.us/401
- http://httpstat.us/404
- http://httpstat.us/500
- http://httpstat.us/501
- test that
.raise_for_statuswill NOT raise an exception when accessing the following URLs:- http://httpstat.us/200
- http://httpstat.us/201
- http://httpstat.us/202
- http://httpstat.us/203
- http://httpstat.us/204
- http://httpstat.us/303
- http://httpstat.us/304
hints:
- the
requestsmodule raises exceptions of typerequests.HTTPError - use parameterized fixtures to avoid writing a lot of tests or boilerplate code
- use timeouts to avoid tests that wait forever
solution
%%file test_requests.py
import pytest
import requests
@pytest.fixture(params=[200, 201, 202, 203, 204, 303, 304])
def good_url(request):
return f'http://httpstat.us/{request.param}'
@pytest.fixture(params=[401, 404, 500, 501])
def bad_url(request):
return f'http://httpstat.us/{request.param}'
@pytest.mark.timeout(2)
def test_good_urls(good_url):
response = requests.get(good_url)
response.raise_for_status()
@pytest.mark.timeout(2)
def test_bad_urls(bad_url):
response = requests.get(bad_url)
with pytest.raises(requests.HTTPError):
response.raise_for_status()
pip install pytest-sugar
!python -m pytest --verbose test_requests.py
running tests in parallel
The pytest-xdist plugin extends pytest with some unique test execution modes:
- test run parallelization: if you have multiple CPUs or hosts you can use those for a combined test run. This allows to speed up development or to use special resources of remote machines.
- βlooponfail: run your tests repeatedly in a subprocess. After each run pytest waits until a file in your project changes and then re-runs the previously failing tests. This is repeated until all tests pass after which again a full run is performed.
- Multi-Platform coverage: you can specify different Python interpreters or different platforms and run tests in parallel on all of them.
- βboxed and pytest-forked: running each test in its own process, so that if a test catastrophically crashes, it doesnβt interfere with other tests
Weβre going to cover only test run parallelization.
first, lets install pytest-xdist:
pip install -qq pytest-xdist
now, lets write a few long running tests
%%file test_parallel.py
import time
def test_t1():
time.sleep(2)
def test_t2():
time.sleep(2)
def test_t3():
time.sleep(2)
def test_t4():
time.sleep(2)
def test_t5():
time.sleep(2)
def test_t6():
time.sleep(2)
def test_t7():
time.sleep(2)
def test_t8():
time.sleep(2)
def test_t9():
time.sleep(2)
def test_t10():
time.sleep(2)
now, we can run these tests in parallel using the pytest -n NUM commandline parameter.
Lets use 10 threads, this will allow us to finish in 2 seconds rather than 20
!python -m pytest -n 10 test_parallel.py
Codebase to test: class Person
Lets reuse the Person and OlympicRunner classes weβve defined in earlier chapters in order to see how to write tests
%%file person.py
# Person v1
class Person:
def __init__(self, name):
name = name
def __repr__(self):
return f"{type(self).__name__}({self.name!r})"
def walk(self):
print(self.name, 'walking')
def run(self):
print(self.name,'running')
def swim(self):
print(self.name,'swimming')
class OlympicRunner(Person):
def run(self):
print(self.name,self.name,"running incredibly fast!")
def show_medals(self):
print(self.name, 'showing my olympic medals')
def train(person):
person.walk()
person.swim()
person.run()
our first test
- conventions
- files with tests should be called
test_*.pyor*_test.py - test function name should start with
test_
- files with tests should be called
- to see if our code works, we can use the
assertpython keyword. pytest adds hooks to assertions to make them more useful
%%file test_person1.py
from person import Person
# our first test
def test_preson_name():
terry = Person('Terry Gilliam')
assert terry.name == 'Terry Gilliam'
!python -m pytest
lets run our tests
# execute the tests via pytest, arguments are passed to pytest
ipytest.run('-qq')
running our first test
# very simple test
def test_person_repr1():
assert str(Person('terry gilliam')) == f"Person('terry gilliam')"
# test using mock object
def test_train1():
person = mocking.Mock()
train(person)
person.walk.assert_called_once()
person.run.assert_called_once()
person.swim.assert_called_once()
# create factory for person's name
@pytest.fixture
def person_name():
return 'terry gilliam'
# create factory for Person, that requires a person_name
@pytest.fixture
def person(person_name):
return Person(person_name)
# test using mock object
def test_train2(person):
# this makes sure no other method is called
person = mocking.create_autospec(person)
train(person)
person.walk.assert_called_once()
person.run.assert_called_once()
person.swim.assert_called_once()
# test Person using and request a person, person_name from the fixtures
def test_person_repr2(person, person_name):
assert str(person) == f"Person('{person_name}')"
# fixture with multiple values
@pytest.fixture(params=['usain bolt', 'Matthew Wells'])
def olympic_runner_name(request):
return request.param
@pytest.fixture
def olympic_runner(olympic_runner_name):
return OlympicRunner(olympic_runner_name)
# test train() using mock object for print
@mocking.patch('builtins.print')
def test_train3(mocked_print, olympic_runner):
train(olympic_runner)
mocked_print.assert_called()
# execute the tests via pytest, arguments are passed to pytest
ipytest.run('-qq')