0101: pigweed.json#

Pigweed AI summary: This paragraph summarizes the content of the given text: The proposal suggests combining various configuration options used in Pigweed-based projects into a single file called "pigweed.json". This file would contain options for environment setup, plugin configuration, and other configurable features. The file would be located at the root of the project's checkout. The proposal also discusses different file formats that could be used, such as JSON, YAML, TOML, or Protobuf Text Format. The "pw_env_setup" Python module would

SEED-0101: pigweed.json

Status: Open for Comments Last Call Accepted Rejected

Proposal Date: 2023-02-06

CL: pwrev/128010

Summary#

Pigweed AI summary: The summary states that Pigweed will combine various configuration options used by downstream projects into one place for easier access and further configuration options.

Combine several of the configuration options downstream projects use to configure parts of Pigweed in one place, and use this place for further configuration options.

Motivation#

Pigweed AI summary: Pigweed-based projects use a JSON file to control their environment setup, which is referenced in bootstrap.sh files and internal infrastructure repositories. The plugins to the pw command-line utility are configured in PW_PLUGINS. Changes have been proposed to configure the behavior of pw format and the formatting steps of pw presubmit from config files, but there is no standard place to put these configuration options.

Pigweed-based projects configure Pigweed and themselves in a variety of ways. The environment setup is controlled by a JSON file that’s referenced in bootstrap.sh files and in internal infrastructure repos that looks something like this:

{
  "root_variable": "<PROJNAME>_ROOT",
  "cipd_package_files": ["tools/default.json"],
  "virtualenv": {
    "gn_args": ["dir_pw_third_party_stm32cube=\"\""],
    "gn_root": ".",
    "gn_targets": [":python.install"]
  },
  "optional_submodules": ["vendor/shhh-secret"],
  "gni_file": "build_overrides/pigweed_environment.gni"
}

The plugins to the pw command-line utility are configured in PW_PLUGINS, which looks like this:

# <name> <Python module> <function>
console pw_console.__main__ main
format pw_presubmit.format_code _pigweed_upstream_main

In addition, changes have been proposed to configure some of the behavior of pw format and the formatting steps of pw presubmit from config files, but there’s no standard place to put these configuration options.

Guide reference#

Pigweed AI summary: This proposal affects two groups of developers: those who want to use Pigweed and those who want to add configurable features to Pigweed. Developers who want to use Pigweed will have one config file that contains all the necessary options. Documentation for individual Pigweed modules will only show relevant configuration options. Developers who want to add configurable features to Pigweed no longer need to define a new file format or figure out where to find it in the tree.

This proposal affects two sets of people: developers looking to use Pigweed, and developers looking to add configurable features to Pigweed.

Developers looking to use Pigweed will have one config file that contains all the options they need to set. Documentation for individual Pigweed modules will show only the configuration options relevant for that module, and multiple of these examples can simply be concatenated to form a valid config file.

Developers looking to add configurable features to Pigweed no longer need to define a new file format, figure out where to find it in the tree (or how to have Pigweed-projects specify a location), or parse this format.

Problem investigation#

Pigweed AI summary: The current system has multiple issues that need to be addressed, including the narrow custom format of PW_PLUGINS, the lack of an accepted place for other modules to retrieve configuration options, and the specific nature of the environment config file. These issues should be combined into a single file, but there are several formats to choose from, each with their own arguments for and against. The location of the file is also an issue, as environment config files can be found in a variety of locations depending on the project

There are multiple issues with the current system that need to be addressed.

  • PW_PLUGINS works, but is a narrow custom format with exactly one purpose.

  • The environment config file is somewhat extensible, but is still specific to environment setup.

  • There’s no accepted place for other modules to retrieve configuration options.

These should be combined into a single file. There are several formats that could be selected, and many more arguments for and against each. Only a subset of these arguments are reproduced here.

  • JSON does not support comments

  • JSON5 is not supported in the Python standard library

  • XML is too verbose

  • YAML is acceptable, but implicit type conversion could be a problem, and it’s not supported in the Python standard library

  • TOML is acceptable, and was selected for a similar purpose by Python, but it’s not supported in the Python standard library before Python v3.11

  • Protobuf Text Format is acceptable and widely used within Google, but is not supported in the Python standard library

The location of the file is also an issue. Environment config files can be found in a variety of locations depending on the project—all of the following paths are used by at least one internal Pigweed-based project.

  • build/environment.json

  • build/pigweed/env_setup.json

  • environment.json

  • env_setup.json

  • pw_env_setup.json

  • scripts/environment.json

  • tools/environment.json

  • tools/env_setup.json

PW_PLUGINS files can in theory be in any directory and pw will search up for them from the current directory, but in practice they only exist at the root of checkouts. Having this file in a fixed location with a fixed name makes it significantly easier to find as a user, and the fixed name (if not path) makes it easy to find programmatically too.

Detailed design#

Pigweed AI summary: The pw_env_setup Python module will provide an API to retrieve a parsed pigweed.json file from the root of the checkout. The pigweed.json file contains options grouped by module name within the pw namespace. Other projects can use other namespaces for their own needs. The file won't add any new files to the checkout root.

The pw_env_setup Python module will provide an API to retrieve a parsed pigweed.json file from the root of the checkout. pw_env_setup is the correct location because it can’t depend on anything else, but other modules can depend on it. Code in other languages does not yet depend on configuration files.

A pigweed.json file might look like the following. Individual option names and structures are not final but will evolve as those options are implemented—this is merely an example of what an actual file could look like. The pw namespace is reserved for Pigweed, but other projects can use other namespaces for their own needs. Within the pw namespace all options are first grouped by their module name, which simplifies searching for the code and documentation related to the option in question.

{
  "pw": {
    "pw_cli": {
      "plugins": {
        "console": {
          "module": "pw_console.__main__",
          "function": "main"
        },
        "format": {
          "module": "pw_presubmit.format_code",
          "function": "_pigweed_upstream_main"
        }
      }
    },
    "pw_env_setup": {
      "root_variable": "<PROJNAME>_ROOT",
      "rosetta": "allow",
      "gni_file": "build_overrides/pigweed_environment.gni",
      "cipd": {
        "package_files": [
          "tools/default.json"
        ]
      },
      "virtualenv": {
        "gn_args": [
          "dir_pw_third_party_stm32cube=\"\""
        ],
        "gn_targets": [
          "python.install"
        ],
        "gn_root": "."
      },
      "submodules": {
        "optional": [
          "vendor/shhh-secret"
        ]
      }
    },
    "pw_presubmit": {
      "format": {
        "python": {
          "formatter": "black",
          "black_path": "pyink"
        }
      }
    }
  }
}

Some teams will resist a new file at the root of their checkout, but this seed won’t be adding any files, it’ll be combining at least one top-level file, maybe two, into a new top-level file, so there won’t be any additional files in the checkout root.

Alternatives#

Pigweed AI summary: The Pigweed configuration may become more fragmented if the pw format and pw presubmit formatting steps read from another config file. However, an alternative file format could be used instead of JSON, as long as it can be parsed into the same internal structure as Python lists, dicts, and primitives.

pw format and the formatting steps of pw presubmit could read from yet another config file, further fracturing Pigweed’s configuration.

A different file format could be chosen over JSON. Since JSON is parsed into only Python lists, dicts, and primitives, switching to another format that can be parsed into the same internal structure should be trivial.

Open questions#

Pigweed AI summary: There is a section titled "Open questions" with a title and a single paragraph that says "None?" indicating that there are no open questions in this section.

None?