pw_fuzzer: Adding Fuzzers Using FuzzTest#

Pigweed AI summary: The document provides a step-by-step guide on how to add fuzzers using FuzzTest in the pw_fuzzer module. It explains how to set up FuzzTest for a project, write a unit test for the target, convert the unit test to a function, add a FUZZ_TEST macro invocation, add the fuzzer to the build, build the fuzzer, and run the fuzzer locally. It also mentions additional resources for more detailed documentation on FuzzTest and related topics.

Note

FuzzTest is currently only supported on Linux and MacOS using Clang.

Step 0: Set up FuzzTest for your project#

Pigweed AI summary: This guide explains how to set up FuzzTest and its dependencies for a project. The process only needs to be done once for a project. The guide provides instructions for using upstream Abseil C++, FuzzTest, GoogleTest and GoogleMock, and RE2. It also provides instructions for setting up FuzzTest for GN, CMake, and Bazel. The guide includes code examples for each setup option.

Note

This workflow only needs to be done once for a project.

FuzzTest and its dependencies are not included in Pigweed and need to be added.

See the following:

You may not want to use upstream GoogleTest all the time. For example, it may not be supported on your target device. In this case, you can limit it to a specific toolchain used for fuzzing. For example:

import("$dir_pw_toolchain/host/target_toolchains.gni")

my_toolchains = {
  ...
  clang_fuzz = {
    name = "my_clang_fuzz"
    forward_variables_from(pw_toolchain_host.clang_fuzz, "*", ["name"])
    pw_unit_test_MAIN = "$dir_pw_fuzzer:fuzztest_main"
    pw_unit_test_GOOGLETEST_BACKEND = "$dir_pw_fuzzer:gtest"
  }
  ...
}

CMake

FuzzTest is enabled by setting several CMake variables. The easiest way to set these is to extend your toolchain.cmake file.

For example:

include(my_project_toolchain.cmake)

set(dir_pw_third_party_fuzztest
    "path/to/fuzztest"
  CACHE STRING "" FORCE
)
set(dir_pw_third_party_googletest
    "path/to/googletest"
  CACHE STRING "" FORCE
)
set(pw_unit_test_GOOGLETEST_BACKEND
    "pw_third_party.fuzztest"
  CACHE STRING "" FORCE
)

Bazel

FuzzTest provides a build configuration that can be imported into your .bazelrc file. Add the following:

# Include FuzzTest build configurations.
try-import %workspace%/third_party/fuzztest/fuzztest.bazelrc
build:fuzztest --@pigweed_config//:fuzztest_config=//pw_fuzzer:fuzztest

Step 1: Write a unit test for the target#

Pigweed AI summary: The paragraph provides an overview of the first step in writing a unit test for a target behavior. It suggests identifying the target behavior that needs testing and starting with a unit test. It also mentions that the guide will use code from a specific location and provides a cautionary note about the code's compatibility with certain devices. Additionally, the paragraph includes code examples of a Metric struct and a Metrics class, along with unit tests for the Metrics class.

As noted previously, the very first step is to identify one or more target behavior that would benefit from testing. See FuzzTest Use Cases for more details on how to identify this code.

Once identified, it is useful to start from a unit test. You may already have a unit test writtern, but if not it is likely still be helpful to write one first. Many developers are more familiar with writing unit tests, and there are detailed guides available. See for example the GoogleTest documentation.

This guide will use code from //pw_fuzzer/examples/fuzztest/. This code includes the following object as an example of code that would benefit from fuzzing for undefined behavior and from roundtrip fuzzing.

Note

To keep the example simple, this code uses the standard library. As a result, this code may not work with certain devices.

// Represent a named value. In order to transmit these values efficiently, they
// can be referenced by fixed length, generated keys instead of names.
struct Metric {
  using Key = uint16_t;
  using Value = uint32_t;

  static constexpr size_t kMaxNameLen = 32;

  Metric() = default;
  Metric(std::string_view name_, Value value_);

  InlineString<kMaxNameLen> name;
  Key key = 0;
  Value value = 0;
};

// Represents a set of measurements from a particular source.
//
// In order to transmit metrics efficiently, the names of metrics are hashed
// internally into fixed length keys. The names can be shared once via `GetKeys`
// and `SetKeys`, after which metrics can be efficiently shared via `Serialize`
// and `Deserialize`.
class Metrics {
 public:
  static constexpr size_t kMaxMetrics = 32;
  static constexpr size_t kMaxSerializedSize =
      sizeof(size_t) +
      kMaxMetrics * (sizeof(Metric::Key) + sizeof(Metric::Value));

  // Retrieves the value of a named metric and stores it in `out_value`. The
  // name must consist of printable ASCII characters. Returns false if the named
  // metric was not `Set` or `Import`ed.
  std::optional<Metric::Value> GetValue(std::string_view name) const;

  // Sets the value of a named metric. The name must consist of printable ASCII
  // characters, and will be added to the mapping of names to keys.
  Status SetValue(std::string_view name, Metric::Value value);

  // Returns the current mapping of names to keys.
  const Vector<Metric>& GetMetrics() const;

  // Replaces the current mapping of names to keys.
  Status SetMetrics(const Vector<Metric>& metrics);

  // Serializes this object to the given `buffer`. Does not write more bytes
  // than `buffer.size()`. Returns the number of number of bytes written or an
  // error if insufficient space.
  StatusWithSize Serialize(pw::ByteSpan buffer) const;

  // Populates this object from the data in the given `buffer`.
  // Returns whether this buffer could be deserialized.
  Status Deserialize(pw::ConstByteSpan buffer);

 private:
  Vector<Metric, kMaxMetrics> metrics_;
};

Unit tests for this class might attempt to deserialize previously serialized objects and to deserialize invalid data:

TEST(MetricsTest, SerializeAndDeserialize) {
  std::array<std::byte, Metrics::kMaxSerializedSize> buffer;

  // Add and copy the names only.
  Metrics src, dst;
  EXPECT_TRUE(src.SetValue("one", 0).ok());
  EXPECT_TRUE(src.SetValue("two", 0).ok());
  EXPECT_TRUE(src.SetValue("three", 0).ok());
  EXPECT_TRUE(dst.SetMetrics(src.GetMetrics()).ok());

  // Modify the values.
  EXPECT_TRUE(src.SetValue("one", 1).ok());
  EXPECT_TRUE(src.SetValue("two", 2).ok());
  EXPECT_TRUE(src.SetValue("three", 3).ok());

  // Transfer the data and check.
  EXPECT_TRUE(src.Serialize(buffer).ok());
  EXPECT_TRUE(dst.Deserialize(buffer).ok());
  EXPECT_EQ(dst.GetValue("one").value_or(0), 1U);
  EXPECT_EQ(dst.GetValue("two").value_or(0), 2U);
  EXPECT_EQ(dst.GetValue("three").value_or(0), 3U);
}

TEST(MetricsTest, DeserializeDoesNotCrash) {
  std::array<std::byte, Metrics::kMaxSerializedSize> buffer;
  std::fill(buffer.begin(), buffer.end(), std::byte(0x5C));

  // Just make sure this does not crash.
  Metrics dst;
  dst.Deserialize(buffer).IgnoreError();
}

Step 2: Convert your unit test to a function#

Pigweed AI summary: In this step, you need to convert your unit test into a function by identifying fixed values that could vary and turning them into parameters. This ensures that your unit test can be preserved and called with the previously fixed values. The provided code examples demonstrate the conversion process. Additionally, it is mentioned that modifications may be required in unit tests if constraints on parameters are not expressed by domains. For more information on domains, you can refer to the provided link.

Examine your unit tests and identify any places you have fixed values that could vary. Turn your unit test into a function that takes those values as parameters. Since fuzzing may not occur on all targets, you should preserve your unit test by calling the new function with the previously fixed values.

void ArbitrarySerializeAndDeserialize(const Vector<Metric>& metrics) {
  std::array<std::byte, Metrics::kMaxSerializedSize> buffer;

  // Add and copy the names only.
  Metrics src, dst;
  for (const auto& metric : metrics) {
    EXPECT_TRUE(src.SetValue(metric.name, 0).ok());
  }
  EXPECT_TRUE(dst.SetMetrics(src.GetMetrics()).ok());

  // Modify the values.
  for (const auto& metric : metrics) {
    EXPECT_TRUE(src.SetValue(metric.name, metric.value).ok());
  }

  // Transfer the data and check.
  EXPECT_TRUE(src.Serialize(buffer).ok());
  EXPECT_TRUE(dst.Deserialize(buffer).ok());
  for (const auto& metric : metrics) {
    EXPECT_EQ(dst.GetValue(metric.name).value_or(0), metric.value);
  }
}

// This unit test will run on host and may run on target devices (if supported).
TEST(MetricsTest, SerializeAndDeserialize) {
  Vector<Metric, 3> metrics;
  metrics.emplace_back("one", 1);
  metrics.emplace_back("two", 2);
  metrics.emplace_back("three", 3);
  ArbitrarySerializeAndDeserialize(metrics);
}

void ArbitraryDeserialize(pw::ConstByteSpan buffer) {
  // Just make sure this does not crash.
  Metrics dst;
  dst.Deserialize(buffer).IgnoreError();
}

// This unit test will run on host and may run on target devices (if supported).
TEST(MetricsTest, DeserializeDoesNotCrash) {
  ArbitraryDeserialize(std::vector<std::byte>(100, std::byte(0x5C)));
}

Note that in ArbitrarySerializeAndDeserialize we no longer assume the marshalling will always be successful, and exit early if it is not. You may need to make similar modifications to your unit tests if constraints on parameters are not expressed by domains as described below.

Step 3: Add a FUZZ_TEST macro invocation#

Pigweed AI summary: To add a FUZZ_TEST macro invocation, you need to include "fuzztest/fuzztest.h" and pass the test suite name and your function name to the FUZZ_TEST macro. You can then call WithDomains on the returned object to specify the input domain for each parameter of the function. The example code provided demonstrates this process. Additionally, you can include specific values as seeds to guide the fuzzer towards certain code paths. However, it is recommended to include a unit test with the

Now, include "fuzztest/fuzztest.h" and pass the test suite name and your function name to the FUZZ_TEST macro. Call WithDomains on the returned object to specify the input domain for each parameter of the function. For example:

auto ArbitraryMetric() {
  return ConstructorOf<Metric>(PrintableAsciiString<Metric::kMaxNameLen>(),
                               Arbitrary<uint32_t>());
}

// This fuzz test will only run on host.
FUZZ_TEST(MetricsTest, ArbitrarySerializeAndDeserialize)
    .WithDomains(VectorOf<Metrics::kMaxMetrics>(ArbitraryMetric()));

// This fuzz test will only run on host.
FUZZ_TEST(MetricsTest, ArbitraryDeserialize)
    .WithDomains(VectorOf<Metrics::kMaxSerializedSize>(Arbitrary<std::byte>()));

You may know of specific values that are “interesting”, i.e. that represent boundary conditions, involve, special handling, etc. To guide the fuzzer towards these code paths, you can include them as seeds. However, as noted in the comments of the examples, it is recommended to include a unit test with the original parameters to ensure the code is tested on target devices.

FuzzTest provides more detailed documentation on these topics. For example:

Refer to The FUZZ_TEST Macro reference for more details on how to use this macro.
Refer to the FuzzTest Domain Reference for details on all the different types of domains supported by FuzzTest and how they can be combined.
Refer to the Test Fixtures reference for how to create fuzz tests from unit tests that use GoogleTest fixtures.

Step 4: Add the fuzzer to your build#

Pigweed AI summary: This section provides instructions on how to add the fuzzer to your build by indicating that the unit test includes one or more fuzz tests and adding a dependency on FuzzTest. It provides examples of how to do this in GN, CMake, and Bazel.

Indicate that the unit test includes one or more fuzz tests by adding a dependency on FuzzTest.

For example, consider the following BUILD.gn:

pw_test("metrics_fuzztest") {
  sources = [ "metrics_fuzztest.cc" ]
  deps = [
    ":metrics_lib",
    "$dir_pw_fuzzer:fuzztest",  # <- Added!
  ]

  # TODO: b/283156908 - Re-enable with a fixed seed.
  enable_if = false
}

CMake

For example, consider the following CMakeLists.txt:

pw_add_test(pw_fuzzer.examples.fuzztest.metrics_fuzztest
  SOURCES
    metrics_fuzztest.cc
  PRIVATE_DEPS
    pw_fuzzer.fuzztest  # <- Added!
    pw_fuzzer.examples.fuzztest.metrics_lib
  GROUPS
    modules
    pw_fuzzer
)

Bazel

For example, consider the following BUILD.bazel:

pw_cc_test(
    name = "metrics_fuzztest",
    srcs = ["metrics_fuzztest.cc"],
    deps = [
        ":metrics_lib",
        "//pw_fuzzer:fuzztest",  # <- Added!
    ],
)

Step 5: Build the fuzzer#

Pigweed AI summary: This section provides instructions on how to build a fuzzer using different tools such as ninja, cmake, and bazel. The fuzzer should be built using a fuzzing toolchain, and examples are provided for each tool. The section includes code snippets for each tool to help with the building process.

Build using ninja on a target that includes your fuzzer with a fuzzing toolchain.

For example, Pigweed itself includes a //:host_clang_fuzz target that builds all tests, including those with fuzzers, using a fuzzing toolchain:

group("host_clang_fuzz") {
  deps = [ ":pigweed_default($_internal_toolchains:pw_strict_host_clang_fuzz)" ]
}

CMake

Build using cmake with the FuzzTest and GoogleTest variables set. For example:

cmake ... \
  -Ddir_pw_third_party_fuzztest=path/to/fuzztest \
  -Ddir_pw_third_party_googletest=path/to/googletest \
  -Dpw_unit_test_GOOGLETEST_BACKEND=pw_third_party.fuzztest

Bazel

By default, bazel will simply omit the fuzz tests and build unit tests. To build these tests as fuzz tests, specify the fuzztest config. For example:

bazel build //... --config=fuzztest

Step 6: Running the fuzzer locally#

Pigweed AI summary: This section provides instructions for running a fuzzer locally using different toolchains such as GN, CMake, and Bazel. It explains how to pass additional flags to the fuzzer binary and provides examples of the output that should be produced. The section also includes TODOs for adding tooling to make it easier to find and run fuzzers, improving fuzzers, and running them continuously on a fuzzing infrastructure. It also includes references to relevant documentation such as the fuzztest toolchain, GoogleTest

When building. Most toolchains will simply omit the fuzz tests and build and run unit tests. A fuzzing toolchain will include the fuzzers, but only run them for a limited time. This makes them suitable for automated testing as in CQ.

To run a fuzz with different options, you can pass additional flags to the fuzzer binary. This binary will be in a subdirectory related to the toolchain. For example:

out/host_clang_fuzz/obj/my_module/test/metrics_test \
  --fuzz=MetricsTest.Roundtrip

Additional sanitizer flags may be passed uisng environment variables.

CMake

When built with FuzzTest and GoogleTest, the fuzzer binaries can be run directly from the CMake build directory. By default, the fuzzers will only run for a limited time. This makes them suitable for automated testing as in CQ. To run a fuzz with different options, you can pass additional flags to the fuzzer binary.

For example:

build/my_module/metrics_test --fuzz=MetricsTest.Roundtrip

Bazel

By default, bazel will simply omit the fuzz tests and build and run unit tests. To build these tests as fuzz tests, specify the “fuzztest” config. For example:

bazel test //... --config=fuzztest

This will build the tests as fuzz tests, but only run them for a limited time. This makes them suitable for automated testing as in CQ.

To run a fuzz with different options, you can use run and pass additional flags to the fuzzer binary. For example:

bazel run //my_module:metrics_test --config=fuzztest
  --fuzz=MetricsTest.Roundtrip

Running the fuzzer should produce output similar to the following:

[.] Sanitizer coverage enabled. Counter map size: 21290, Cmp map size: 262144
Note: Google Test filter = MetricsTest.Roundtrip
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MetricsTest
[ RUN      ] MetricsTest.Roundtrip
[*] Corpus size:     1 | Edges covered:    131 | Fuzzing time:    504.798us | Total runs:  1.00e+00 | Runs/secs:     1
[*] Corpus size:     2 | Edges covered:    133 | Fuzzing time:    934.176us | Total runs:  3.00e+00 | Runs/secs:     3
[*] Corpus size:     3 | Edges covered:    134 | Fuzzing time:   2.384383ms | Total runs:  5.30e+01 | Runs/secs:    53
[*] Corpus size:     4 | Edges covered:    137 | Fuzzing time:   2.732274ms | Total runs:  5.40e+01 | Runs/secs:    54
[*] Corpus size:     5 | Edges covered:    137 | Fuzzing time:   7.275553ms | Total runs:  2.48e+02 | Runs/secs:   248

pw_fuzzer#

Better C++ code through easier fuzzing

pw_fuzzer: Adding Fuzzers Using FuzzTest#

Step 0: Set up FuzzTest for your project#

Step 1: Write a unit test for the target#

Step 2: Convert your unit test to a function#

Step 3: Add a FUZZ_TEST macro invocation#

Step 4: Add the fuzzer to your build#

Step 5: Build the fuzzer#

Step 6: Running the fuzzer locally#