Resources:
- https://pragprog.com/titles/bopytest2/python-testing-with-pytest-second-edition/
- https://cheatography.com/nanditha/cheat-sheets/pytest/
- https://gist.github.com/kwmiebach/3fd49612ef7a52b5ce3a#file-pytest-md
- Getting Started
- Running the pytest testing framework
- Why pytest?
- Features
- Fixtures
- Test Parameterization
Python virtual environments enable you to set up a Python sandbox with its own set of packages separate from the system site-packages in which to work.
$ python -m venv env_dir_name
To activate on macOS and Linux.
$ source env_dir_name/bin/activate
To activate on Windows.
$ env_dir_name\Scripts\activate.bat
To activate on Windows with PowerShell.
$ env_dir_name\Scripts\Activate.ps
When done,run:
$ deactivate
pip is the tool used to install Python packages, and it is installed as part of your Python installation.
Confirm pip version by running:
$ pip --version
You should see:
pip 24.0 from /Users/ ... /env_dir_name/lib/python3.11/site-packages/pip (python 3.11)
Install pytest by running:
$ pip install pytest
Confirm the pytest version by running:
$ pytest --version
You should see:
pytest 8.1.1
Run pytest with the following command:
$ pytest [options] [file_or_dir] [file_or_dir]
Show help message and configuration info:
$ pytest --help
Command | Description |
---|---|
pytest test_one.py |
Run tests in a module. |
pytest tests |
Run tests in a directory. |
pytest test_one.py::test_function |
Run a specific test within a module. |
pytest test_one.py::TestClass:test_method |
Run a specific method of a class. |
Running:
$ pytest test_one.py
Returns:
====================== test session starts =======================
collected 1 item
test_one.py. [100%]
======================= 1 passed in 0.01s ========================
The dot after test_one.py
means that one test was run and it passed. The [100%]
is a percentage indicator showing how much of the test suite is done so far.
Because there is just one test in the test session, one test equals 100% of the
tests. If you need more information, you can use -v
or --verbose
.
Running
$ pytest test_two.py
Returns
===================== test session starts ======================
collected 1 item
test_two.py F [100%]
=========================== FAILURES ===========================
_________________________ test_failing _________________________
def test_failing():
> assert (1, 2, 3) == (3, 2, 1)
E assert (1, 2, 3) == (3, 2, 1)
E At index 0 diff: 1 != 3
E Use -v to get the full diff
test_two.py:2: AssertionError
=================== short test summary info ====================
FAILED test_two.py::test_failing - assert (1, 2, 3) == (3, 2, 1)
====================== 1 failed in 0.03s =======================
The failing test, test_failing
, gets its own section to show us why it failed. And pytest
tells us exactly what the first failure is: index 0 is a mismatch. If you have a color terminal, much of this appears in red to make it really stand out. This extra section shows exactly where the test failed and some of the surrounding code is called a traceback.
Option | Description |
---|---|
-v , -–verbose |
Increase verbosity |
-q , --quiet |
Decrease verbosity |
Here are a few of the reasons pytest
stands out above many other testing frameworks:
- Simple tests are simple to write in
pytest
. - Complex tests are still simple to write.
- Tests are easy to read.
- You can get started in seconds.
- You use
assert
in tests for verifications, not things likeself.assertEqual()
orself.assertLessThan()
- You can use
pytest
to run tests written forunittest
ornose
.
This section covers a few features of pytest
.
Part of pytest
execution finds which tests to run using test discovery. If we name them according to the default pytest
naming conventions, pytest
looks at your current directory and all subdirectories for test files and runs the test code it finds.
The default naming conventions:
- Test files named
test_<something>.py
or<something>_test.py
- Test methods and functions should be named
test_<something>
- Test classes should be named
Test<Something>
There are ways to alter these discovery rules via pytest
configuration.
pytest
allows you to use the standard Python assert
for verifying expectations and values in Python tests (pytest
uses assert rewriting). These assert statements communicate test failure.
Other testing frameworks use assert helper functions.
Assertions in pytest versus unittest:
pytest | unittest |
---|---|
assert something |
assertTrue(something) |
assert not something |
assertFalse(something) |
assert a == b |
assertEqual(a, b) |
assert a != b |
assertNotEqual(a, b) |
assert a is None |
assertIsNone(a) |
assert a is not None |
assertIsNotNone(a) |
assert a <= b |
assertLessEqual(a, b) |
assert 1 in [2, 3, 4] |
assertNotIn(1, [2, 3, 4]) |
assert 'fizz' not in 'fizzbuzz' |
assertNotIn('fizz', 'fizzbuzz') |
Tests can fail from:
- assertion failures
- from calls to
pytest.fail()
- from any uncaught exception
When the test code calls pytest.fail()
, it raises an exception and allows you to explicitly fail an executing test with a message.
Pass and fail are not the only outcomes possible.
Possible outcomes include:
- PASSED (.) — The test ran successfully.
- FAILED (F) — The test did not run successfully.
- SKIPPED (s) — The test was skipped. Skip a test by using decorators:
@pytest.mark.skip()
@pytest.mark.skipif()
- XFAIL (x) — The test was not supposed to pass, and it ran and failed.
- Tell
pytest
that a test is expected to fail by using the@pytest.mark.xfail()
- Tell
- XPASS (X) — The test was marked with
xfail
, but it ran and passed. - ERROR (E) — An exception happened either during the execution of a fixture or hook function, and not during the execution of a test function.
For complete details on skip
and xfail
, see:
https://docs.pytest.org/en/latest/how-to/skipping.html
By default, during test execution any output sent to stdout and stderr is captured.
If a test or a setup method fails its corresponding captured output will usually be shown along with the failure traceback.
Use the --capture=no
option to allow print statements to be seen even in passing tests.
NOTE: the option -s
is a shortcut for --capture=no
For complete details, see: https://docs.pytest.org/en/latest/how-to/capture-stdout-stderr.html
Use the --tb=no flag to turn off tracebacks, which is useful when you don’t need the full output.
Running:
$ pytest --tb=no
Returns:
===================== test session starts ======================
collected 2 items
test_one.py . [ 50%]
test_two.py F [100%]
=================== short test summary info ====================
FAILED test_two.py::test_failing - assert (1, 2, 3) == (3, 2, 1)
================= 1 failed, 1 passed in 0.03s ==================
The options are:
Option | Description |
---|---|
--tb=auto |
(default) 'long' tracebacks for the first and last entry, but 'short' style for the other entries |
--tb=no |
no traceback at all |
--tb=long |
exhaustive, informative traceback formatting |
--tb=short |
shorter traceback format |
--tb=line |
only one line per failure |
--tb=native |
Python standard library formatting |
For complete details on managing pytest
output, see:
https://docs.pytest.org/en/latest/how-to/output.html
Use pytest.raises()
to test for expected exceptions.
Example:
import pytest
def insert_coins(quarters):
if type(quarters) is not int:
raise TypeError('quarters must be an integer')
def test_insert_coin_when_quarters_is_string_raises():
with pytest.raises(TypeError):
insert_coins('foobar')
If the expected exception is raised, the test passes. If no exception is raised, the test fails. If the test raises a different exception, it fails.
In most cases, use test classes only for the purpose of grouping tests. This will allow you to easily run the test methods together.
NOTE: The use of test classes that involve inheritance will likely confuse people.
Running all tests in a class:
$ pytest path/test_module.py::TestClass
Running small subsets of tests is useful, especially when debugging tests. pytest allows you to run a small batch of tests in many different ways:
Subset of tests | Example |
---|---|
All tests in a directory | pytest path |
All tests in a module | pytest path/test_module.py |
Single test function | pytest path/test_module.py::test_function |
All tests in a class | pytest path/test_module.py::TestClass |
Single test method | pytest path/test_module.py::TestClass::test_method |
Tests matching a name pattern | pytest -k EXPRESSION |
For the -k
option, pytest
will only run tests which match the given substring expression.
Example: -k 'test_method or test_other'
matches all test functions and classes whose name contains 'test_method' or 'test_other'. And -k 'not test_method'
matches those that don't contain 'test_method' in their names.
Fixtures are identified by the @pytest.fixture()
decorated functions.
import pytest
@pytest.fixture()
def fixture_name():
fixture_value = 'some data'
return fixture_value
def test_something_with_fixture(fixture_name):
assert fixture_name == 'some data'
Test functions or other fixtures depend on a fixture by putting its name in their parameter list.
Fixtures can return data using return or yield.
import pytest
@pytest.fixture()
def database():
db = MyDatabase() # setup
yield db
db.close() # teardown
def test_database(database):
result = database.retrieve_answer()
assert result == 42
Code before the yield is the setup code. Code after the yield is the teardown code.
Fixtures can be set to function, class, module, package, or session scope. The default is function scope. You can even define the scope dynamically.
Multiple test functions can use the same fixture.
Multiple test modules can use the same fixture if it’s in a conftest.py file.
Multiple fixtures at different scope can speed up test suites while maintaining test isolation.
Tests and fixtures can use multiple fixtures.
pytest --setup-show
is used to see the order of execution.
pytest --fixtures
is used to list available fixtures and where the fixture is located.
NOTE: the pytest
keyword is spelled parametrize (not parameterize or parameterise)
Three ways to parametrize tests:
- Parametrize test functions, creating many test cases, by applying the
@pytest.mark.parametrize()
decorator. - Parametrize fixtures with
@pytest.fixture(params=())
- Generate complex parameterization sets with
pytest_generate_tests
.
To parametrize a test function, add parameters to the test definition and use the @pytest.mark.parametrize()
decorator to define the sets of arguments to test.
Example:
import pytest
@pytest.mark.parametrize(
('n', 'expected'), [
(1, 2),
(2, 3),
(3, 4),
(4, 5),
]
)
def test_increment(n, expected):
assert n + 1 == expected
Run subsets of parameterized test cases using the -k
option.
For details on parameterizing fixtures, see: https://docs.pytest.org/en/latest/how-to/fixtures.html#parametrizing-fixtures
For details on complex parameterization, see: https://docs.pytest.org/en/latest/how-to/parametrize.html#basic-pytest-generate-tests-example
For complete details on parametrization, see: https://docs.pytest.org/en/latest/how-to/parametrize.html