NAME

unittest — Utilities to implement test programs

LIBRARY

shtk_import unittest

DESCRIPTION

The unittest module provides a framework with which to implement test programs.

A test program implemented with the unittest library is made up of a combination of standalone test cases and/or test cases within test fixtures. The test program will exit with 0 (success) if and only if all tests pass, and 1 (failure) if any single test fails. Skips and expected failures are considered passes.

Program entry point

Test programs implemented using unittest must use the shtk_unittest_main(3) function as their entry point. There are two mechanisms to achieve this.

The first is to end the test program with the following definition of main:

main() { shtk_unittest_main "${@}"; }

The second is to tell shtk(1) at build time to use shtk_unittest_main(3) as the program's entry point:

$ shtk build -m shtk_unittest_main module_test.sh

In general, prefer the latter mechanism.

Asserts vs. expects

The unittest library provides a variety of helper check functions. One such example is the ‘equal’ check, which allows the caller to compare two values for equality and fail the test case with the appropriate error message if the values differ.

Each check is offered in two flavors: an assert and an expect, which in the previous example means we have access to the two corresponding functions assert_equal and expect_equal. Assert-style checks are fatal: when the condition they test for is not met, the test case is immediately aborted; instead, expect-style checks are not fatal: they record the currently-running test case as failed but they let the test case continue execution. Assert-style checks should be used to validate conditions that must hold true in order to run the test (e.g. ensuring that the setup of the test case succeeds); expect-style checks should be used for all other cases.

Scoping

All functions provided by the unittest library are prefixed by ‘shtk_unittest_’ as you would expect from the coding practices of shtk. However, such long prefix is inconvenient when writing test programs, as test programs are a very special case of scripts in which a domain-specific language makes sense.

Therefore, for simplicity reasons, test cases have access to shortened names of the unittest functions without the ‘shtk_unittest_’ prefix. In order words, a test case can simply call fail instead of having to call shtk_unittest_fail.

Standalone test cases

Standalone test cases are defined by top-level functions whose name ends with the ‘_test’ suffix and are registered as test cases with the shtk_unittest_add_test(3) function. Such test cases are run independently of each other, with no common setup nor teardown code among them.

Test fixtures

Test fixtures are collections of related test cases that may share optional setup and teardown methods. Test fixtures are defined by top-level functions whose name ends with the ‘_fixture’ suffix and are registered as fixtures with the shtk_unittest_add_fixture(3) function.

Within fixture fuctions, there may be a setup and a teardown method, and there should be one or more test cases registered with shtk_unittest_add_test(3). Each test within the fixture is executed independently of each other, but the setup and teardown methods (if any) are also executed with the test.

Test environment

The runtime engine (usually kyua(1)) is responsible for executing the test program under a controlled directory and with a sanitized environment.

However, due to the nature of shell-based test programs, unittest creates one extra directory for each test case, and such directory becomes the work directory of the test case. These directories are all created within the directory in which the test program was executed. To ensure that any files created by the test case remain within its own subdirectory as much as possible, the HOME and TMPDIR variables are set to point to the individual test's work directory.

One time setup and teardown

Test programs may optionally define top-level one_time_setup and one_time_teardown.

The code supplied in the one_time_setup function is executed exactly once at the beginning of the test program and all state set up by the function is shared across all tests.

The code supplied in the one_time_teardown function is executed exactly once at the end of the test program and should be used to clean up any state prepared by one_time_setup.

It is important to mention that these one-time setup and teardown routines run in the parent directory of the executed test cases. In other words: if the setup routine creates any file to be shared across all tests, the tests will have access to such files by looking them up in their parent directory.

Test results

Any test case can complete with one of the following results:

Expected failure (pass): The test case detected a failure, but such failure is expected. This condition is often use to denote test cases that are either incomplete or test cases that exercise a known bug. See shtk_unittest_set_expected_failure(3) for details.
Failure: The test case explicitly failed or it unexpectedly terminated. See shtk_unittest_delayed_fail(3) and shtk_unittest_fail(3) for details.
Pass: The test case completed successfully without reporting any errors.
Skip (pass): The test case determined that it cannot run through completion and decided to abort early instead of reporting a false negative. See shtk_unittest_skip(3) for details.

EXAMPLES

The most basic test program that can be written is the following, which provides a simple test case for cp(1)'s -f flag:

shtk_import unittest

shtk_unittest_add_test cp_f_forces_override
cp_f_forces_override_test() {
    echo "first" >first
    echo "second" >second
    chmod 555 second

    # First make sure that cp without -f fails.
    assert_command -s exit:1 -e ignore cp first second

    # Now run a second attempt with -f and verify that the command
    # exits successfully and it is silent.
    assert_command cp -f first second

    cmp -s first second || fail "source and destination do not match"
}

main() { shtk_unittest_main "${@}"; }

A more complex test program can use fixtures to group related test cases and to provide common setup and teardown code for each of them:

shtk_import unittest

shtk_unittest_add_fixture cp
    setup() {
        # Create common files used by all test cases.  Note that this runs
        # once per test case, so state is not shared among them.
        echo "first" >first
        echo "second" >second
    }

    teardown() {
        # Common cleanup to be executed at the end of the test case, no
        # matter if it passes or fails.
        rm -f first second
    }

    shtk_unittest_add_test override_existing_file
    override_existing_file_test() {
        assert_command cp first second
        cmp -s first second || fail "source and destination do not match"
    }

    shtk_unittest_add_test f_forces_override
    f_forces_override_test() {
        chmod 555 second
        assert_command -s exit:1 -e ignore cp first second
        assert_command cp -f first second
        cmp -s first second || fail "source and destination do not match"
    }
}

main() { shtk_unittest_main "${@}"; }

Lastly, the most complex test program is depicted here, which includes a combination of fixtures and test cases with one-time setup and teardown routines:

shtk_import unittest

one_time_setup() {
    ... initialization code shared by all tests in the program ...
}

one_time_teardown() {
    ... clean up code shared by all tests in the program ...
}

shtk_unittest_add_fixture clients
clients_fixture() {
    setup() {
        ... initialization code shared by all tests in the fixture ...
    }
    teardown() {
        ... cleanup code shared by all tests in the fixture ...
    }

    shtk_unittest_add_test add
    add_test() {
        ... first test in the fixture ...
    }

    shtk_unittest_add_test modify
    modify_test() {
        ... second test in the fixture ...
        fail "And it fails"
    }
}

shtk_unittest_add_test initialization
initialization_test() {
    ... standalone test not part of any fixture ...
    skip "But cannot run due to some unsatisfied condition"
}

# Either do this or, preferably, pass -mshtk_unittest_main to
# "shtk build" when compiling the test program and don't define main.
main() { shtk_unittest_main "${@}"; }

HISTORY

unittest first appeared in shtk 1.6.