pw_compilation_testing#

The pw_compilation_testing module provides for negative compilation (NC) testing. Negative compilation tests ensure that code that should not compile does not compile. Negative compilation testing is helpful in a variety of scenarios, for example:

  • Testing for compiler errors, such as [[nodiscard]] checks.

  • Testing that a template cannot be instantiated with certain types.

  • Testing that a static_assert statement is triggered as expected.

  • For a constexpr function, testing that a PW_ASSERT is triggered as expected.

Negative compilation tests are only supported in GN currently. Negative compilation tests are not currently supported in GN on Windows due to b/241565082.

Warning

This module is in an early, experimental state. Do not use it unless you have consulted with the Pigweed team.

Negative compilation test example#

Pigweed AI summary: This code is an example of negative compilation testing using the Google Test framework. The code includes a struct with a static_assert statement and a function that uses PW_ASSERT. The negative compilation tests are created in preprocessor #if or #elif blocks to check that a static_assert statement fails if the code is compiled. The code also includes a test that checks that a specific PW_ASSERT() fails when expected.

#include "gtest/gtest.h"
#include "pw_compilation_testing/negative_compilation.h"

template <int kValue>
struct MyStruct {
  static_assert(kValue % 2 == 0, "wrong number!");

  constexpr int MultiplyOdd(int runtime_value) const {
    PW_ASSERT(runtime_value % 2 == 0);
    return kValue * runtime_value;
  }
};

[[maybe_unused]] MyStruct<16> this_one_works;

// NC tests cannot be compiled, so they are created in preprocessor #if or
// #elif blocks. These NC tests check that a static_assert statement fails if
// the code is compiled.
#if PW_NC_TEST(NegativeOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<-1> illegal;
#elif PW_NC_TEST(PositiveOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<5> this_is_illegal;
#endif  // PW_NC_TEST

struct Foo {
  // Negative compilation tests can go anywhere in a source file.
#if PW_NC_TEST(IllegalValueAsClassMember)
  PW_NC_EXPECT("wrong number!");
  MyStruct<12> also_illegal;
#endif  // PW_NC_TEST
};

TEST(MyStruct, MultiplyOdd) {
  MyStruct<5> five;
  EXPECT_EQ(five.MultiplyOdd(3), 15);

  // This NC test checks that a specific PW_ASSERT() fails when expected.
  // This only works in an NC test if the PW_ASSERT() fails while the compiler
  // is executing constexpr code. The test code is used in a constexpr
  // statement to force compile-time evaluation.
  #if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber)
  [[maybe_unused]] constexpr auto fail = [] {
    PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);");
    MyStruct<3> my_struct;
    return my_struct.MultiplyOdd(4);  // Even number, PW_ASSERT should fail.
  }();
  #endif  // PW_NC_TEST
}

Creating a negative compilation test#

  • Declare a pw_cc_negative_compilation_test() GN target or set negative_compilation_test = true in a pw_test() target.

  • Add the test to the build in a toolchain with negative compilation testing enabled (pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true).

  • In the test source files, add #include "pw_compilation_testing/negative_compilation.h".

  • Use the PW_NC_TEST(TestName) macro in a #if statement.

  • Immediately after the PW_NC_TEST(TestName), provide one or more Python-style regular expressions with the PW_NC_EXPECT() macro, one per line.

  • Execute the tests by running the build.

To simplify parsing, all PW_NC_TEST() statements must fit on a single line. PW_NC_EXPECT() statements may span multiple lines, but must contain a single regular expression as a string literal. The string may be comprised of multiple implicitly concatenated string literals. The PW_NC_EXPECT() statement cannot contain anything else except for //-style comments.

Test assertions#

Negative compilation tests must have at least one assertion about the compilation output. The assertion macros must be placed immediately after the line with the PW_NC_TEST() or the test will fail.

PW_NC_EXPECT(regex_string_literal)#

When negative compilation tests are run, checks the compilation output for the provided regular expression. The argument to the PW_NC_EXPECT() statement must be a string literal. The literal is interpreted character-for-character as a Python raw string literal and compiled as a Python re regular expression.

For example, PW_NC_EXPECT("something (went|has gone) wrong!") searches the failed compilation output with the Python regular expression re.compile("something (went|has gone) wrong!").

PW_NC_EXPECT_GCC(regex_string_literal)#

Same as PW_NC_EXPECT, but only applies when compiling with GCC.

PW_NC_EXPECT_CLANG(regex_string_literal)#

Same as PW_NC_EXPECT, but only applies when compiling with Clang.

Test expectation tips

Be as specific as possible, but avoid compiler-specific error text. Try matching against the following:

  • static_assert messages.

  • Contents of specific failing lines of source code: PW_NC_EXPECT("PW_ASSERT\(!empty\(\));").

  • Comments on affected lines: PW_NC_EXPECT("// Cannot construct from nullptr").

  • Function names: PW_NC_EXPECT("SomeFunction\(\).*private").

Do not match against the following:

  • Source file paths.

  • Source line numbers.

  • Compiler-specific wording of error messages, except when necessary.

Design#

Pigweed AI summary: This article discusses Pigweed's negative compilation testing framework, which allows users to define negative compilation tests using preprocessor #if blocks and PW_NC_TEST() and PW_NC_EXPECT() macros. The build process involves invoking the pw_compilation_testing.generator script to extract test cases and generate build targets, compiling the test source file with all tests disabled, and running the negative compilation test targets using the pw_compilation_testing.runner script. The article also highlights the advantages of Pigweed's framework over existing systems, including

The basic flow for negative compilation testing is as follows.

  • The user defines negative compilation tests in preprocessor #if blocks using the PW_NC_TEST() and PW_NC_EXPECT macros.

  • The build invokes the pw_compilation_testing.generator script. The generator script:

    • finds PW_NC_TEST() statements and extracts a list of test cases,

    • finds all associated PW_NC_EXPECT statements, and

    • generates build targets for each negative compilation tests, passing the test information and expectations to the targets.

  • The build compiles the test source file with all tests disabled.

  • The build invokes the negative compilation test targets, which run the pw_compilation_testing.runner script. The test runner script:

    • invokes the compiler, setting a preprocessor macro that enables the #if block for the test.

    • captures the compilation output, and

    • checks the compilation output for the PW_NC_EXPECT expressions.

  • If compilation failed, and the output matches the test case’s PW_NC_EXPECT expressions, the test passes.

  • If compilation succeeded or the PW_NC_EXPECT expressions did not match the output, the test fails.

Existing frameworks#

Pigweed AI summary: Pigweed's negative compilation testing framework was inspired by Chromium's no-compile tests and a similar framework used internally at Google. It improves on these systems by allowing trivial integration with unit tests, using a safer and more natural macro-based API for test declarations, providing more readable and flexible test assertions, and requiring assertions to ensure compilation fails for the expected reason.

Pigweed’s negative compilation tests were inspired by Chromium’s no-compile tests tests and a similar framework used internally at Google. Pigweed’s negative compilation testing framework improves on these systems in a few respects:

  • Trivial integration with unit tests. Negative compilation tests can easily be placed alongside other unit tests instead of in separate files.

  • Safer, more natural macro-based API for test declarations. Other systems use #ifdef macro checks to define test cases, which fail silently when there are typos. Pigweed’s framework uses function-like macros, which provide a clean and natural API, catch typos, and ensure the test is integrated with the NC test framework.

  • More readable, flexible test assertions. Other frameworks place assertions in comments after test names, while Pigweed’s framework uses function-like macros. Pigweed also supports compiler-specific assertions.

  • Assertions are required. This helps ensure that compilation fails for the expected reason and not for an accidental typo or unrelated issue.