Testing with pytest

For bpt’s more rigorous test suite, we use the pytest testing framework. These tests are stored in the tests/ directory and written in test_*.py files.

The test suite can be run separately without Dagon by executing pytest from within the Poetry virtual environment:

$ pytest tests/

Note that individual tests can take between a few seconds and a few minutes to execute, so it may be useful to execute only a subset of the tests based on the functionality you want to test. Refer to the pytest documentation for more information about using and executing pytest. If you are running the full test suite, you may also want to pass the -n argument with a number of parallel jobs to execute.

Writing Tests

If a particular aspect of bpt can be tested in isolation and within a few dozen milliseconds, you should prefer to test it as a unit test in a *.test.cpp file. The pytest tests are intended to perform full end-to-end feature and error handling tests.

Tests are grouped into individual Python files in the tests/ directory. Any Python file containing tests must have a filename beginning with test_. Individual test functions should begin with test_. All test functions should be properly type-annotated and successfully check via pyright.

The bpt test suite has access to a set of test fixtures that can be used throughout tests to perform complex setup and teardown for complete test-by-test isolation.

Here is a simple test that simple executes bpt with --help:

def test_get_help(bpt: BPTWrapper) -> None:
    bpt.run(['--help'])

In this test function, the bpt object is a test fixture that wraps the bpt executable under test.

Testing Error Handling

It is important that bpt handle errors correctly, of course, including user error. It is not simply enough to check that a certain operation fails: We must be sure that it fails correctly. To check that the correct code path is executed, bpt can write a file containing a simple constant string designating the error handling path that was taken. The file will be written to the path indicated by the BPT_WRITE_ERROR_MARKER environment variable.

For examples of these error strings, search for usage of write_error_marker in the bpt source code. These should only execute within error-handling contexts, should appear near the log messages that issue diagnostics, and should be specific to the error at hand.

To write a test that checks for a given error-handling path, use the expect_error_marker() context manager function:

def test_sdist_invalid_project(tmp_project: Project) -> None:
  # Trying to create a package archive from a project without a
  # bpt.yaml is invalid. Check that it creates the correct
  # error-message string
  with error.expect_error_marker('no-package-yaml'):
    tmp_project.pkg_create()