Python GN Templates

The Python build is implemented with GN templates defined in pw_build/python.gni. See the .gni file for complete usage documentation.

See also

• Pigweed’s GN Python Build for an overview on how Python in GN is built.

• The pw_build docs for other GN templates available within Pigweed.

Python Base Templates

Pigweed AI summary: This document describes the Python Base Templates in the pw_build module, which includes templates for creating Python packages, actions, scripts, and groups. The pw_python_package template expands to several subtargets, including linting, testing, installing, and building a Python wheel. The pw_python_script template represents standalone Python scripts and tests, and can include an embedded pw_python_action. The pw_python_group template represents a group of pw_python_package and pw_python_script targets. The document also includes examples and arguments for

The core subset of templates where you can create Python packages, actions, scripts and group them together are listed below.

• pw_python_package

• pw_python_action

• pw_python_script

• pw_python_group

pw_python_package

Pigweed AI summary: This section describes the main Python template, pw_python_package, which represents a Python package and expands to several subtargets. The section also explains how to use abbreviated labels and set the toolchain for actions within a pw_python_package. It provides a list of arguments for pw_python_package, including setup, sources, tests, and dependencies, and gives an example of a pw_python_package declaration for a pw_my_module module.

The main Python template is pw_python_package. Each pw_python_package target represents a Python package. As described in pw_python_package targets, each pw_python_package expands to several subtargets. In summary, these are:

• <name> - Represents the files themselves

• <name>.lint - Runs static analysis

• <name>.tests - Runs all tests for this package

• <name>.install - Installs the package

• <name>.wheel - Builds a Python wheel

GN permits using abbreviated labels when the target name matches the directory name (e.g. //foo for //foo:foo). For consistency with this, Python package subtargets are aliased to the directory when the target name is the same as the directory. For example, these two labels are equivalent:

//path/to/my_python_package:my_python_package.tests
//path/to/my_python_package:tests

The actions in a pw_python_package (e.g. installing packages and running Pylint) are done within a single GN toolchain to avoid duplication in multi-toolchain builds. This toolchain can be set with the pw_build_PYTHON_TOOLCHAIN GN arg, which defaults to $dir_pw_build/python_toolchain:python.

Arguments

Pigweed AI summary: This section describes the various arguments that can be used in the configuration file for a Python package. These include a list of setup file paths, an option to generate setup files, sources and test files for the package, dependencies on other packages and GN targets, other files to track, and options for static analysis tools like mypy and pylint. It also includes information on embedding a pw_proto_library target in the package and using a pylintrc or mypy configuration file.

• setup - List of setup file paths (setup.py or pyproject.toml & setup.cfg), which must all be in the same directory.

• generate_setup: As an alternative to setup, generate setup files with the keywords in this scope. name is required. This follows the same structure as a setup.cfg file’s declarative config For example:

  generate_setup = {
    metadata = {
      name = "a_nifty_package"
      version = "1.2a"
    }
    options = {
      install_requires = [ "a_pip_package" ]
    }
  }
  

• sources - Python sources files in the package.

• tests - Test files for this Python package.

• python_deps - Dependencies on other pw_python_packages in the GN build.

• python_test_deps - Test-only pw_python_package dependencies.

• other_deps - Dependencies on GN targets that are not pw_python_packages.

• inputs - Other files to track, such as package_data.

• proto_library - A pw_proto_library target to embed in this Python package. generate_setup is required in place of setup if proto_library is used. See Adding Python proto modules to an existing package.

• static_analysis List of static analysis tools to run; "*" (default) runs all tools. The supported tools are "mypy" and "pylint".

• pylintrc - Optional path to a pylintrc configuration file to use. If not provided, Pylint’s default rcfile search is used. Pylint is executed from the package’s setup directory, so pylintrc files in that directory will take precedence over others.

• mypy_ini - Optional path to a mypy configuration file to use. If not provided, mypy’s default configuration file search is used. mypy is executed from the package’s setup directory, so mypy.ini files in that directory will take precedence over others.

Example

Pigweed AI summary: This is an example Python package declaration for a module called "pw_my_module". It includes a list of sources, tests, and dependencies, as well as a reference to a pylintrc file. The package is declared using the "pw_python_package" function and is imported using two import statements.

This is an example Python package declaration for a pw_my_module module.

import("//build_overrides/pigweed.gni")

import("$dir_pw_build/python.gni")

pw_python_package("py") {
  setup = [
    "pyproject.toml",
    "setup.cfg",
    "setup.py",
  ]
  sources = [
    "pw_my_module/__init__.py",
    "pw_my_module/alfa.py",
    "pw_my_module/bravo.py",
    "pw_my_module/charlie.py",
  ]
  tests = [
    "alfa_test.py",
    "charlie_test.py",
  ]
  python_deps = [
    "$dir_pw_status/py",
    ":some_protos.python",
  ]
  python_test_deps = [ "$dir_pw_build/py" ]
  pylintrc = "$dir_pigweed/.pylintrc"
}

pw_python_action

Pigweed AI summary: The pw_python_action template is a wrapper for GN's action function that allows for running Python scripts. The documentation for usage can be found in the pw_build documentation.

The pw_python_action template is a convenience wrapper around GN’s action function for running Python scripts. See pw_python_action in the pw_build documentation for usage.

pw_python_script

Pigweed AI summary: The pw_python_script is a set of standalone Python scripts and/or tests that support all arguments of pw_python_package except those for setup. These targets can be installed, but only their dependencies are installed. The pw_python_script allows creating a pw_python_action associated with the script, and an action in pw_python_script can always be replaced with a standalone pw_python_action. Using the embedded action has some advantages, including bridging the gap between actions and Python targets, automatic dependency on pw_python_script, and

A pw_python_script represents a set of standalone Python scripts and/or tests. These files support all of the arguments of pw_python_package except those setup. These targets can be installed, but this only installs their dependencies.

pw_python_script allows creating a pw_python_action associated with the script. To create an action, pass an action scope to pw_python_script. If there is only a single source file, it serves as the action’s script by default.

An action in pw_python_script can always be replaced with a standalone pw_python_action, but using the embedded action has some advantages:

• The embedded action target bridges the gap between actions and Python targets. A Python script can be expressed in a single, concise GN target, rather than in two overlapping, dependent targets.

• The action automatically depends on the pw_python_script. This ensures that the script’s dependencies are installed and the action automatically reruns when the script’s sources change, without needing to specify a dependency, a step which is easy to forget.

• Using a pw_python_script with an embedded action is a simple way to check an existing action’s script with Pylint or Mypy or to add tests.

pw_python_group

Pigweed AI summary: The pw_python_group represents a group of pw_python_package and pw_python_script targets that do not add any files. Their subtargets simply forward to those of their dependencies. The example code provided shows a pw_python_group named "solar_system_python_packages" with a list of python dependencies for various planets and planetoids.

Represents a group of pw_python_package and pw_python_script targets. These targets do not add any files. Their subtargets simply forward to those of their dependencies.

pw_python_group("solar_system_python_packages") {
  python_deps = [
    "//planets/mercury/py",
    "//planets/venus/py",
    "//planets/earth/py",
    "//planets/mars/py",
    "//planets/jupiter/py",
    "//planets/saturn/py",
    "//planets/uranus/py",
    "//planets/neptune/py",
    "//planetoids/ceres/py",
    "//planetoids/pluto/py",
  ]
}

Python Environment Templates

Pigweed AI summary: This paragraph provides an overview of Python Environment Templates. It mentions two specific templates: pw_python_venv and pw_python_pip_install. The pw_python_venv template is used to create a Python virtualenv for use within the GN build. The pw_python_pip_install template is used to pip install pw_python_package targets into the developer environment. The paragraph also provides examples and arguments for each template.

Templates that manage the Python build and bootstrap environment are listed here.

• pw_python_venv

• pw_python_pip_install

pw_python_venv

Pigweed AI summary: The "pw_python_venv" module is used by Pigweed to define and create a Python virtualenv. It is used in the GN build to create a virtualenv that all Python actions will run in. The module takes several arguments, including the path where the virtualenv will be created, a list of constraint files, a list of requirements files to install, and a list of in-tree targets to check for external third-party pip dependencies. The module also has an option to generate hashes when

Defines and creates a Python virtualenv. This template is used by Pigweed in https://cs.pigweed.dev/pigweed/+/main:pw_env_setup/BUILD.gn to create a virtualenv for use within the GN build that all Python actions will run in.

Example

Pigweed AI summary: This paragraph is a summary of the given text: The example provided is a typical Python venv definition in a top-level BUILD.gn file. It includes a caption with the filename "BUILD.gn" and a literal block containing the code for declaring arguments, defining Python dependencies, and setting up a Python virtual environment. The virtual environment is named "my_build_venv" and has a specified path, constraints, requirements, and source packages. The "pip_generate_hashes" flag is set to

Example of a typical Python venv definition in a top level BUILD.gn

declare_args() {
  pw_build_PYTHON_BUILD_VENV = "//:my_build_venv"
}

pw_python_group("my_product_packages") {
  python_deps = [
    "//product_dev_tools/py",
    "//product_release_tools/py",
  ]
}

pw_python_venv("my_build_venv") {
  path = "$root_build_dir/python-build-venv"
  constraints = [ "//tools/constraints.list" ]
  requirements = [ "//tools/requirements.txt" ]
  source_packages = [
    "$dir_pw_env_setup:core_pigweed_python_packages",
    "//tools:another_pw_python_package",
    "//:my_product_packages",
  ]
  pip_generate_hashes = true
}

Arguments

Pigweed AI summary: The paragraph provides an overview of the arguments and options for creating a virtual environment using the pw_build Python package. It mentions the "path" argument for specifying the directory where the virtualenv will be created, the "constraints" argument for specifying constraint files, the "requirements" argument for specifying requirements files, and the "pip_generate_hashes" option for using "--generate-hashes" when running pip-compile. It also mentions the "source_packages" argument for specifying in-tree targets that will be checked

• path: The directory where the virtualenv will be created. This is relative to the GN root and must begin with “$root_build_dir/” if it lives in the output directory or “//” if it lives in elsewhere.

• constraints: A list of constraint files used when performing pip install into this virtualenv. By default this is set to the pw_build_PIP_CONSTRAINTS GN arg.

• requirements: A list of requirements files to install into this virtualenv on creation. By default this is set to the pw_build_PIP_REQUIREMENTS GN arg.

  See also

  For more info on the pw_build_PIP_CONSTRAINTS and pw_build_PIP_REQUIREMENTS GN args see: Third-party Python Requirements and Constraints

• pip_generate_hashes: (Default: false) Use --generate-hashes When running pip-compile to compute the final requirements.txt

• source_packages: A list of in-tree pw_python_package or targets that will be checked for external third_party pip dependencies to install into this virtualenv. Note this list of targets isn’t actually installed into the virtualenv. Only packages defined inside the [options] install_requires section of each pw_python_package’s setup.cfg will be pip installed.

  See also

  For an example setup.cfg file see: Configuring setuptools using setup.cfg files

pw_python_pip_install

Pigweed AI summary: The pw_python_pip_install module is used to pip install pw_python_package targets into the developer environment. The packages to be installed are specified in the "packages" argument, and all packages are installed using a single pip install command with a --constraint argument for each constraint file in the pw_build_PIP_CONSTRAINTS GN arg. The module also has "editable" and "force_reinstall" arguments. An example of a typical Python venv definition in a top-level BUILD.gn file is

This will pip install pw_python_package targets into the bootstrapped developer environment.

Example

Pigweed AI summary: This is an example of a typical Python venv definition in a top level BUILD.gn file. It includes a pw_python_pip_install function that installs packages for a product's development and release tools. The packages are specified using their respective paths.

Example of a typical Python venv definition in a top level BUILD.gn

pw_python_pip_install("pip_install_my_product_packages") {
  packages = [
    "//product_dev_tools/py",
    "//product_release_tools/py",
  ]
}

Arguments

Pigweed AI summary: This paragraph describes the "Arguments" section of a module called "pw-build-python-dist". It explains that the section includes a list of packages to be installed using a single "pip install" command with constraints specified in the "pw_build_PIP_CONSTRAINTS" GN arg. It also mentions two optional arguments, "editable" and "force_reinstall", that can be passed to the pip install command.

• packages: A list of pw_python_package targets to be pip installed. All packages specified will be installed using a single pip install command with a --constraint argument for each constraint file in the pw_build_PIP_CONSTRAINTS GN arg.

• editable: If true, –editable is passed to the pip install command.

• force_reinstall: If true, --force-reinstall is passed to the pip install command.

Python Distributable Templates

Pigweed AI summary: Pigweed provides templates to simplify the bundling of Python packages for deployment. These templates can be found in the `pw_build/python_dist.gni` file. The templates include `pw_python_wheels`, `pw_python_zip_with_setup`, and `pw_python_distribution`. The `pw_python_wheels` template collects Python wheels for specified `pw_python_package` targets and their dependencies. It does not include Python dependencies from outside the GN build. The template takes a list of `packages`

Pigweed also provides some templates to make it easier to bundle Python packages for deployment. These templates are found in pw_build/python_dist.gni.

• pw_python_wheels

• pw_python_zip_with_setup

• pw_python_distribution

pw_python_wheels

Pigweed AI summary: The pw_python_wheels module collects Python wheels for specified targets and their dependencies, but not external dependencies. The module takes a list of pw_python_package targets and an output directory as arguments. The module works by generating a wheel for each pw_python_package target and then collecting them by traversing the pw_python_package_wheels GN metadata key.

Collects Python wheels for one or more pw_python_package targets, plus any additional pw_python_package targets they depend on, directly or indirectly. Note that this does not include Python dependencies that come from outside the GN build, like packages from PyPI, for example. Those should still be declared in the package’s setup.py file as usual.

Arguments

Pigweed AI summary: This paragraph describes the "Arguments" section of a document, which includes two items: "packages" and "directory". "Packages" is a list of targets that should be included as wheels, along with their dependencies. "Directory" is the output location for the collected wheels, with a default value of "$target_out_dir/$target_name".

• packages - List of pw_python_package targets whose wheels should be included; their dependencies will be pulled in as wheels also.

• directory - Output directory for the collected wheels. Defaults to $target_out_dir/$target_name.

Wheel collection under the hood

Pigweed AI summary: The ".wheel" subtarget of a "pw_python_package" generates a ".whl" file for the Python package. The "pw_python_wheels" template determines which wheels to collect by using the "pw_python_package_wheels" GN metadata key, which lists the output directory for each wheel.

The .wheel subtarget of every pw_python_package generates a wheel (.whl) for the Python package. The pw_python_wheels template figures out which wheels to collect by traversing the pw_python_package_wheels GN metadata key, which lists the output directory for each wheel.

pw_python_zip_with_setup

Pigweed AI summary: The pw_python_zip_with_setup module generates a .zip archive that contains Python wheels for one or more pw_python_package targets, along with wheels for any additional pw_python_package targets in the GN build they depend on. The archive also includes setup scripts for Linux, MacOS, and Windows that create a Python virtual environment and install the wheels using pip. Additional files and directories can be included in the archive, such as a README file with setup and usage instructions. The module takes a list of pw_python_package

Generates a .zip archive suitable for deployment outside of the project’s developer environment. The generated .zip contains Python wheels (.whl files) for one or more pw_python_package targets, plus wheels for any additional pw_python_package targets in the GN build they depend on, directly or indirectly. Dependencies from outside the GN build, such as packages from PyPI, must be listed in packages’ setup.py or setup.cfg files as usual.

The .zip also includes simple setup scripts for Linux, MacOS, and Windows. The setup scripts automatically create a Python virtual environment and install the whole collection of wheels into it using pip.

Optionally, additional files and directories can be included in the archive. One common example of an additional file to include is a README file with setup and usage instructions for the distributable. A simple ready-to-use README file is available at pw_build/py_dist/README.md.

Arguments

Pigweed AI summary: This paragraph describes the "Arguments" section of a document, which includes three different arguments: "packages", "inputs", and "dirs". The "packages" argument is a list of targets whose wheels should be included, along with their dependencies. The "inputs" argument is an optional list of extra files to include in the generated .zip file, formatted the same way as the inputs argument to pw_zip targets. The "dirs" argument is also optional and is a list of directories to include in

• packages - A list of pw_python_package targets whose wheels should be included; their dependencies will be pulled in as wheels also.

• inputs - An optional list of extra files to include in the generated .zip, formatted the same way as the inputs argument to pw_zip targets.

• dirs - An optional list of directories to include in the generated .zip, formatted the same was as the dirs argument to pw_zip targets.

Example

Pigweed AI summary: This is a code snippet written in the GNI language, which is used for building software. The code imports two files and creates a Python distribution package called "my_tools" that includes a specific package and a README file. The target of this code is the "pw-python-distribution" module.

import("//build_overrides/pigweed.gni")

import("$dir_pw_build/python_dist.gni")

pw_python_zip_with_setup("my_tools") {
  packages = [ ":some_python_package" ]
  inputs = [ "$dir_pw_build/python_dist/README.md > /${target_name}/" ]
}

pw_python_distribution

Pigweed AI summary: The `pw_python_distribution` module is used to generate a directory of Python packages from source files that can be deployed outside of the project developer environment. The resulting directory only contains files mentioned in each package's `setup.cfg` file. This is useful for bundling multiple Python packages into a single package for distribution to other locations like http://pypi.org. The module takes several arguments, including `packages` (a list of `pw_python_package` targets to be installed), `include_tests`

Generates a directory of Python packages from source files suitable for deployment outside of the project developer environment. The resulting directory contains only files mentioned in each package’s setup.cfg file. This is useful for bundling multiple Python packages up into a single package for distribution to other locations like http://pypi.org.

Arguments

Pigweed AI summary: The given paragraph describes the arguments and options for a Python package build tool. It mentions the "packages" argument, which is a list of targets to be installed into the build directory along with their dependencies. It also mentions the "include_tests" argument, which determines whether the package tests should be copied to a "tests" subdirectory. The "extra_files" argument allows for including additional files in the output, with each item in the list specifying a source file and its destination. The paragraph also

• packages - A list of pw_python_package targets to be installed into the build directory. Their dependencies will be pulled in as wheels also.

• include_tests - If true, copy Python package tests to a tests subdir.

• extra_files - A list of extra files that should be included in the output. The format of each item in this list follows this convention:

  //some/nested/source_file > nested/destination_file
  

  • Source and destination file should be separated by >.

  • The source file should be a GN target label (starting with //).

  • The destination file will be relative to the generated output directory. Parent directories are automatically created for each file. If a file would be overwritten an error is raised.

• generate_setup_cfg - If included, create a merged setup.cfg for all python Packages using either a common_config_file as a base or name and version strings. The common_config_file should contain the required fields in the metadata and options sections as shown in Configuring setup() using setup.cfg files.

  This scope can optionally include:

  • append_git_sha_to_version = true: Append the current git SHA to the package version string after a + sign.

  • append_date_to_version = true: Append the current date to the package version string after a + sign.

  • include_default_pyproject_file = true: Include a standard pyproject.toml file in the output.

  • include_extra_files_in_package_data = true: Add any extra_files entries to the generated setup.cfg file under the [options.package_data] section.

  • auto_create_package_data_init_py_files = true: (Default: true) Create __init__.py files as needed in all subdirs of extra_files when including in [options.package_data].

  Example using a common setup.cfg and pyproject.toml files.

  generate_setup_cfg = {
    common_config_file = "pypi_common_setup.cfg"
    append_date_to_version = true
  }
  extra_files = [
    "//source/pyproject.toml > pyproject.toml"
  ]
  

  Example using name and version strings and a default pyproject.toml file.

  generate_setup_cfg = {
    name = "awesome"
    version = "1.0.0"
    include_default_pyproject_file = true
    append_date_to_version = true
  }
  

Using this template will create an additional target for and building a Python wheel. For example if you define pw_python_distribution("awesome") the resulting targets that get created will be:

• awesome - This will create the merged package with all source files in place in the out directory under out/obj/awesome/.

• awesome.wheel - This builds a Python wheel from the above source files under out/obj/awesome._build_wheel/awesome*.whl.

Example

Pigweed AI summary: This is an example code snippet for building a Python distribution using Pigweed. It includes importing build overrides and a Python distribution module, specifying packages and extra files, and generating a setup configuration file. The output shows the directory structure of the built Python source tree.

./pw_env_setup/BUILD.gn

import("//build_overrides/pigweed.gni")

import("$dir_pw_build/python_dist.gni")

pw_python_distribution("build_python_source_tree") {
  packages = [
    ":some_python_package",
    ":another_python_package",
  ]
  include_tests = true
  extra_files = [
    "//README.md > ./README.md",
    "//some_python_package/py/BUILD.bazel > some_python_package/BUILD.bazel",
    "//another_python_package/py/BUILD.bazel > another_python_package/BUILD.bazel",
  ]
  generate_setup_cfg = {
    common_config_file = "pypi_common_setup.cfg"
    append_git_sha_to_version = true
    append_date_to_version = true
  }
}

./out/obj/pw_env_setup/build_python_source_tree/

$ tree ./out/obj/pw_env_setup/build_python_source_tree/
├── README.md
├── setup.cfg
├── some_python_package
│   ├── BUILD.bazel
│   ├── __init__.py
│   ├── py.typed
│   ├── some_source_file.py
│   └── tests
│       └── some_source_test.py
└── another_python_package
    ├── BUILD.bazel
    ├── __init__.py
    ├── another_source_file.py
    ├── py.typed
    └── tests
        └── another_source_test.py