Testing Cataclysm

When you make Cataclysm from source, an executable tests/cata_test is built from test cases found in the tests/ directory. These tests are written in the Catch2 framework.

Run tests/cata_test --help to see the available command-line options, and/or consult the Catch2 tutorial for a more thorough introduction.

Guidelines

When creating tests, ensure that all objects used (directly or indirectly) are fully reset before testing. Several tests have been rendered flaky by properties of randomly generated objects or interactions between tests via global objects (often the player object). As a general guideline, test cases should be standalone (one test should not rely on the output of another).

When generating objects with json definitions, use REQUIRE statements to assert the properties of the objects that the test needs. This protects the test from shifting json definitions by making it apparent what about the object changed to cause the test to break.

Writing test cases

You can choose several ways to organize and express your tests, but the basic unit is a TEST_CASE. Each test .cpp file should define at least one test case, with a name, and optional (but strongly encouraged) list of tags:

TEST_CASE( "sweet junk food", "[food][junk][sweet]" )
{
    // ...
}

Within the TEST_CASE, the Catch2 framework allows a number of different macros for logically grouping related parts of the test together. One approach that encourages a high level of readability is the BDD (behavior-driven-development) style using GIVEN, WHEN, and THEN sections. Here’s an outline of what a test might look like using those:

    TEST_CASE( "sweet junk food", "[food][junk][sweet]" )
    {
        GIVEN( "character has a sweet tooth" ) {

            WHEN( "they eat some junk food" ) {

                THEN( "they get a morale bonus from its sweetness" ) {
                }
            }
        }
    }

Thinking in these terms may help you understand the logical progression from setting up the test and initializing the test data (usually expressed by the GIVEN part), performing some operation that generates a result you want to test (often contained in the WHEN part), and verifying this result meets your expectations (the THEN part, naturally).

Filling in the above with actual test code might look like this:

    TEST_CASE( "sweet junk food", "[food][junk][sweet]" )
    {
        avatar dummy;
        dummy.clear_morale();

        GIVEN( "character has a sweet tooth" ) {
            dummy.toggle_trait( trait_PROJUNK );

            WHEN( "they eat some junk food" ) {
                item necco( "neccowafers" );
                dummy.eat( necco );

                THEN( "they get a morale bonus from its sweetness" ) {
                    CHECK( dummy.has_morale( MORALE_SWEETTOOTH ) >= 5 );
                }
            }
        }
    }

Let’s look at each part in turn to see what’s going on. First, we declare an avatar, representing the character or player. This test is going to check the player’s morale, so we clear it to ensure a clean slate:

avatar dummy;
dummy.clear_morale();

Inside the GIVEN, we want some code that implements what the GIVEN is saying - that the character has a sweet tooth. In the game’s code, this is represented with the PROJUNK trait, so we can set that using toggle_trait:

GIVEN( "character has a sweet tooth" ) {
    dummy.toggle_trait( trait_PROJUNK );

Now, notice we are nested inside the GIVEN - for the rest of the scope of that GIVEN, the dummy will have this trait. For this simple test it will only affect a couple more lines, but when your tests become larger and more complex (which they will), you will need to be aware of these nested scopes and how you can use them to avoid cross-pollution between your tests.

Anyway, now that our dummy has a sweet tooth, we want them to eat something sweet, so we can spawn the neccowafers item and tell them to eat some:

WHEN( "they eat some junk food" ) {
    dummy.eat( item( "neccowafers" ) );

The function(s) you invoke at this point are often the focus of your testing; the goal is to exercise some pathway through those function(s) in such a way that your code will be reached, and thus covered by the test. The eat function is used as an example here, but that is quite a high-level, complex function itself, with many behaviors and sub-behaviors. Since this test case is only interested in the morale effect, a better test would invoke a lower-level function that eat invokes, such as modify_morale.

Our dummy has eaten the neccowafers, but did it do anything? Because they have a sweet tooth, they should get a specific morale bonus known as MORALE_SWEETTOOTH, and it should be at least 5 in magnitude:

THEN( "they get a morale bonus from its sweetness" ) {
    CHECK( dummy.has_morale( MORALE_SWEETTOOTH ) >= 5 );
}

This CHECK macro takes a boolean expression, failing the test if the expression is false. Likewise, you can use CHECK_FALSE, which will fail if the expression is true.

Requiring or Checking

While the CHECK and CHECK_FALSE macros make assertions about the truth or falsity of expressions, they still allow the test to continue, even when they fail. This lets you do several CHECKs, and be informed if one or more of them do not meet your expectations.

Another kind of assertion is the REQUIRE (and its counterpart REQUIRE_FALSE). Unlike the CHECK assertions, REQUIRE will not continue if it fails - this assertion is considered essential for the test to continue.

A REQUIRE is useful when you wish to double-check your assumptions after making some change to the system state. For example, here are a couple of REQUIREs added to the sweet-tooth test, to ensure our dummy really has the desired trait, and that the neccowafers really are junk food:

    GIVEN( "character has a sweet tooth" ) {
        dummy.toggle_trait( trait_PROJUNK );
        REQUIRE( dummy.has_trait( trait_PROJUNK ) );

        WHEN( "they eat some junk food" ) {
            item necco( "neccowafers" );
            REQUIRE( necco.has_flag( "ALLERGEN_JUNK" ) );

            dummy.eat( necco );

            THEN( "they get a morale bonus from its sweetness" ) {
                CHECK( dummy.has_morale( MORALE_SWEETTOOTH ) >= 5 );
            }
        }
    }

We use REQUIRE here, because there is no reason to continue the test if these fail. If our assumptions are wrong, nothing that follows is valid. Clearly, if toggle_trait failed to give the character the PROJUNK trait, or if the neccowafers turn out not to be made of sugar after all, then our test of the morale bonus is meaningless.

You can think of REQUIRE as being a prerequisite for the test, while CHECK is looking at the results of the test.