pw_env_setup#

A classic problem in the embedded space is reducing the time from git clone to having a binary executing on a device. The issue is that an entire suite of tools is needed for non-trivial production embedded projects. For example:

A C++ compiler for your target device, and also for your host
A build system or three; for example, GN, Ninja, CMake, Bazel
A code formatting program like clang-format
A debugger like OpenOCD to flash and debug your embedded device (OpenOCD support removed for Windows)
A known Python version with known modules installed for scripting
A Go compiler for the Go-based command line tools

…and so on

In the server space, container solutions like Docker or Podman solve this; however, in our experience container solutions are a mixed bag for embedded systems development where one frequently needs access to native system resources like USB devices, or must operate on Windows.

pw_env_setup is our compromise solution for this problem that works on Mac, Windows, and Linux. It leverages the Chrome packaging system CIPD to bootstrap a Python installation, which in turn inflates a virtual environment. The tooling is installed into your workspace, and makes no changes to your system. This tooling is designed to be reused by any project.

Users interact with pw_env_setup with two commands: . bootstrap.sh and . activate.sh. The bootstrap command always pulls down the current versions of CIPD packages and sets up the Python virtual environment. The activate command reinitializes a previously configured environment, and if none is found, runs bootstrap.

Note

On Windows the scripts used to set up the environment are bootstrap.bat and activate.bat. For simplicity they will be referred to with the .sh endings unless the distinction is relevant.

Warning

At this time pw_env_setup works for us, but isn’t well tested. We don’t suggest relying on it just yet. However, we are interested in experience reports; if you give it a try, please send us a note about your experience.

On POSIX systems, the environment can be deactivated by running deactivate.

Using pw_env_setup in your project#

Downstream Projects Using Pigweed’s Packages#

Pigweed AI summary: The article discusses how projects can use Pigweed's packages and dependencies. It explains how to use the pw_env_setup to install dependencies and how to source Pigweed's bootstrap.sh and activate.sh scripts. The article also provides an example of what a project's bootstrap.sh could look like. It explains how to pull in a CIPD dependency into Bazel using WORKSPACE rules and how to make the entire set of Pigweed's remote repositories available to a project. The article also discusses user-friend

Projects using Pigweed can leverage pw_env_setup to install Pigweed’s dependencies or their own dependencies. Projects that only want to use Pigweed’s dependencies without modifying them can just source Pigweed’s bootstrap.sh and activate.sh scripts.

An example of what your project’s bootstrap.sh could look like is below. This assumes bootstrap.sh is at the top level of your repository.

# Do not include a "#!" line, this must be sourced and not executed.

# This assumes the user is sourcing this file from it's parent directory. See
# below for a more flexible way to handle this.
PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh"

export PW_PROJECT_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")"

# You may wish to check if the user is attempting to execute this script
# instead of sourcing it. See below for an example of how to handle that
# situation.

# Source Pigweed's bootstrap utility script.
# Using '.' instead of 'source' for POSIX compatibility. Since users don't use
# dash directly, using 'source' in most documentation so users don't get
# confused and try to `./bootstrap.sh`.
. "$PW_PROJECT_ROOT/third_party/pigweed/pw_env_setup/util.sh"

pw_check_root "$PW_ROOT"
_PW_ACTUAL_ENVIRONMENT_ROOT="$(pw_get_env_root)"
export _PW_ACTUAL_ENVIRONMENT_ROOT
SETUP_SH="$_PW_ACTUAL_ENVIRONMENT_ROOT/activate.sh"
pw_bootstrap --args...  # See below for details about args.
pw_finalize bootstrap "$SETUP_SH"

Bazel Usage#

Pigweed AI summary: The article discusses how to pull in a CIPD dependency into Bazel using WORKSPACE rules instead of bootstrap.sh. By using the provided code, the entire set of Pigweeds remote repositories will be available to the project, and a full list of the remote repositories can be obtained by running a specific command. All files and executables in each CIPD remote repository are exported and visible, and it is possible to access the Bloaty binaries using a specific command.

It is possible to pull in a CIPD dependency into Bazel using WORKSPACE rules rather than using bootstrap.sh. e.g.

# WORKSPACE

load("//pw_env_setup/bazel/cipd_setup:cipd_rules.bzl", "pigweed_deps")

# Setup CIPD client and packages.
# Required by: pigweed.
# Used by modules: all.
pigweed_deps()

load("@cipd_deps//:cipd_init.bzl", "cipd_init")

cipd_init()

This will make the entire set of Pigweeds remote repositories available to your project. Though these repositories will only be donwloaded if you use them. To get a full list of the remote repositories that this configures, run:

bazel query //external:all | grep cipd_

All files and executables in each CIPD remote repository is exported and visible either directely (@cipd_<dep>//:<file>) or from ‘all’ filegroup (@cipd_<dep>//:all).

From here it is possible to get access to the Bloaty binaries using the following command. For example;

bazel run @cipd_pigweed_third_party_bloaty_embedded_linux_amd64//:bloaty \
 -- --help

User-Friendliness#

Pigweed AI summary: The article provides code snippets for modifying the sourcing of bootstrap.sh from a different directory and checking if the user is executing or sourcing the file. The code includes shell commands for different shells and checks for Pigweed's automated testing infrastructure.

You may wish to allow sourcing bootstrap.sh from a different directory. In that case you’ll need the following at the top of bootstrap.sh.

_python_abspath () {
  python -c "import os.path; print(os.path.abspath('$@'))"
}

# Use this code from Pigweed's bootstrap to find the path to this script when
# sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in
# presubmit tests with strange setups, and can be omitted if you're not using
# Pigweed's automated testing infrastructure.
if test -n "$PW_CHECKOUT_ROOT"; then
  PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")"
  unset PW_CHECKOUT_ROOT
# Shell: bash.
elif test -n "$BASH"; then
  PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")"
# Shell: zsh.
elif test -n "$ZSH_NAME"; then
  PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")"
# Shell: dash.
elif test ${0##*/} = dash; then
  PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \
    "$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")"
# If everything else fails, try $0. It could work.
else
  PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")"
fi

You may also wish to check if the user is attempting to execute bootstrap.sh instead of sourcing it. Executing bootstrap.sh would download everything required for the environment, but cannot modify the environment of the parent process. To check for this add the following.

# Check if this file is being executed or sourced.
_pw_sourced=0
# If not running in Pigweed's automated testing infrastructure the
# SWARMING_BOT_ID check is unnecessary.
if [ -n "$SWARMING_BOT_ID" ]; then
  # If set we're running on swarming and don't need this check.
  _pw_sourced=1
elif [ -n "$ZSH_EVAL_CONTEXT" ]; then
  case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac
elif [ -n "$KSH_VERSION" ]; then
  [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \
    "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \
    && _pw_sourced=1
elif [ -n "$BASH_VERSION" ]; then
  (return 0 2>/dev/null) && _pw_sourced=1
else  # All other shells: examine $0 for known shell binary filenames
  # Detects `sh` and `dash`; add additional shell filenames as needed.
  case ${0##*/} in sh|dash) _pw_sourced=1;; esac
fi

_pw_eval_sourced "$_pw_sourced"

Downstream Projects Using Different Packages#

Pigweed AI summary: The paragraph discusses downstream projects that use additional or different packages along with Pigweed. It mentions that these projects should copy the Pigweed sample project's "bootstrap.sh" and "pigweed.json" files and update the call to "pw_bootstrap". It also suggests searching for other places that may require changes, such as setting the "PW_ROOT" and "PW_PROJECT_ROOT" environment variables. The paragraph provides explanations of different parts of the "pigweed.json" file, including variables, CIPD

Projects depending on Pigweed but using additional or different packages should copy the Pigweed sample project’s bootstrap.sh and pigweed.json and update the call to pw_bootstrap. Search for “downstream” for other places that may require changes, like setting the PW_ROOT and PW_PROJECT_ROOT environment variables. Explanations of parts of pigweed.json are described here.

pw.pw_env_setup.root_variable: Variable used to point to the root of the source tree. Optional, can always use PW_PROJECT_ROOT instead. (That variable will be set regardless of whether this is provided.)
pw.pw_env_setup.cipd_package_files: CIPD package file. JSON file consisting of a list of additional CIPD package files to import and a list of dictionaries with “path”, “platforms”, “subdir”, “tags”, and “version_file” keys. Both top-level lists are optional. An example is below. Only “path”, “platforms”, and “tags” are required. If “version_file” is specified then pw doctor will fail if that version file is not present. If “subdir” is specified then this packages will be installed in a subdirectory of the directory created for packages in this file.

{
  "included_files": [
    "foo.json"
  ],
  "packages": [
    {
      "path": "infra/3pp/tools/go/${platform}",
      "platforms": [
          "linux-amd64",
          "linux-arm64",
          "mac-amd64",
          "windows-amd64"
      ],
      "subdir": "pa/th",
      "tags": [
        "version:2@1.16.3"
      ],
      "version_file": ".versions/go.cipd_version"
    }
  ]
}

pw.pw_env_setup.project_actions

A list of plugins to load and run after CIPD setup, but prior to virtualenv setup, for e.g. downloading project-specific tools or artifacts needed by later steps. Particularly useful for downstream projects with limited CIPD access.

A plugin is specified as a dictionary with two keys: “import_path” and “module_name”

The specified module must provide a “run_actions” method which takes a single argument, “env_vars”, which is a pw_env_setup.Environment instance.

NB: This feature is not supported when using a python2.7 system python.

Sample plugin and pigweed.json blob:

"""Sample pw_env_setup project action plugin.

A sample/starter project action plugin template for pw_env_setup.
"""
def run_action(**kwargs):
    """Sample project action."""
    if "env" not in kwargs:
        raise ValueError(f"Missing required kwarg 'env', got %{kwargs}")

    kwargs["env"].prepend("PATH", "PATH_TO_NEW_TOOLS")
    raise NotImplementedError("Sample project action running!")

"project_actions" : [
   {
    "import_path": "pw_env_setup",
    "module_name": "sample_project_action"
   }
],

pw.pw_env_setup.virtualenv.gn_args: Any necessary GN args to be used when installing Python packages.
pw.pw_env_setup.virtualenv.gn_targets: Target for installing Python packages. Downstream projects will need to create targets to install their packages or only use Pigweed Python packages.
pw.pw_env_setup.virtualenv.gn_root: The root directory of your GN build tree, relative to PW_PROJECT_ROOT. This is the directory your project’s .gn file is located in. If you’re only installing Pigweed Python packages, use the location of the Pigweed submodule.
pw.pw_env_setup.virtualenv.requirements: A list of Python Pip requirements files for installing into the Pigweed virtualenv. Each file will be passed as additional --requirement argument to a single `pip install at the beginning of bootstrap’s Python environment setup stage. See the Requirements Files documentation for details on what can be specified using requirements files.
pw.pw_env_setup.virtualenv.constraints: A list of Python Pip constraints files. These constraints will be passed to every pip invocation as an additional --constraint argument during bootstrap. virtualenv. See the Constraints Files documentation for details on formatting.
pw.pw_env_setup.virtualenv.system_packages: A boolean value that can be used the give the Python virtual environment access to the system site packages. Defaults to false.
pw.pw_env_setup.optional_submodules: By default environment setup will check that all submodules are present in the checkout. Any submodules in this list are excluded from that check.
pw.pw_env_setup.required_submodules: If this is specified instead of optional_submodules bootstrap will only complain if one of the required submodules is not present. Combining this with optional_submodules is not supported.
pw.pw_env_setup.pw_packages: A list of packages to install using pw_package after the rest of bootstrap completes.
pw.pw_env_setup.gni_file: Location to write a .gni file containing paths to many things within the environment directory. Defaults to build_overrides/pigweed_environment.gni.
pw.pw_env_setup.json_file: Location to write a .json file containing step-by-step modifications to the environment, for reading by tools that don’t inherit an environment from a sourced bootstrap.sh.
pw.pw_env_setup.rosetta: Whether to use Rosetta to use amd64 packages on arm64 Macs. Accepted values are never, allow, and force. For now, allow means force. At some point in the future allow will be changed to mean never.

An example of a config file is below.

{
  "pw": {
    "pw_env_setup": {
      "root_variable": "EXAMPLE_ROOT",
      "cipd_package_files": [
        "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json",
        "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"
        "tools/myprojectname.json"
      ],
      "virtualenv": {
        "gn_root": ".",
        "gn_targets": [
          ":python.install",
        ],
        "system_packages": false
      },
      "pw_packages": [],
      "optional_submodules": [
        "optional/submodule/one",
        "optional/submodule/two"
      ],
      "gni_file": "tools/environment.gni",
      "json_file": "tools/environment.json",
      "rosetta": "allow"
    }
  }
}

Only the packages necessary for almost all projects based on Pigweed are included in the cipd_setup/pigweed.json file. A number of other files are present in that directory for projects that need more than the minimum. Internal-Google projects using LUCI should at least include luci.json.

In case the CIPD packages need to be referenced from other scripts, variables like PW_${BASENAME}_CIPD_INSTALL_DIR point to the CIPD install directories, where ${BASENAME} is "PIGWEED" for "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json" and "LUCI" for "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json". This example would set the following environment variables.

PW_LUCI_CIPD_INSTALL_DIR
PW_MYPROJECTNAME_CIPD_INSTALL_DIR
PW_PIGWEED_CIPD_INSTALL_DIR

These directories are also referenced in the gni_file specified by the environment config file as dir_cipd_${BASENAME}. This allows the GN build to reliably reference these directories without using GN getenv() calls or hardcoding paths.

In addition, PW_${BASENAME}_CIPD_INSTALL_DIR and PW_${BASENAME}_CIPD_INSTALL_DIR/bin are both added to PATH for each package directory.

If multiple packages install executables with the same name, the file mentioned last topologically takes priority. For example, with the file contents below, d.json’s entries will appear in PATH before c.json’s, which will appear before b.json’s, which will appear before a.json’s.

pigweed.json#

{
  "pw": {
    "pw_env_setup": {
      "cipd_package_files": [
        "a.json",
        "b.json",
        "d.json"
      ]
    }
  }
}

a.json#

{
  "package_files": [
    // ...
  ]
}

b.json#

{
  "included_files": ["c.json"],
  "package_files": [
    // ...
  ]
}

c.json#

{
  "package_files": [
    // ...
  ]
}

d.json#

{
  "package_files": [
    // ...
  ]
}

Effective File Loading Order#

pigweed.json
a.json
b.json
c.json
d.json

Pinning Python Packages#

Pigweed AI summary: Pigweed uses a pip constraints file to pin the versions of packages it depends on, ensuring that versions don't change without approval. To pin the versions of additional packages, run "pw python-packages list" and add "pw_build_PIP_CONSTRAINTS = [//path/to/constraints/file]" to the project's .gn file. To update packages, set "pw_build_PIP_CONSTRAINTS = []", delete the environment, and bootstrap again. Then run "pw presubmit

Python modules usually express dependencies as ranges, which makes it easier to install many Python packages that might otherwise have conflicting dependencies. However, this means version of packages can often change underneath us and builds will not be hermetic.

To ensure versions don’t change without approval, Pigweed by default pins the versions of packages it depends on using a pip constraints file. To pin the versions of additional packages your project depends on, run pw python-packages list <path/to/constraints/file> and then add pw_build_PIP_CONSTRAINTS = ["//path/to/constraints/file"] to your project’s .gn file (see Pigweed’s .gn file for an example).

To update packages, set pw_build_PIP_CONSTRAINTS = [], delete the environment, and bootstrap again. Then run the list command from above again, and run pw presubmit.

Environment Variables#

Pigweed AI summary: This section provides a list of environment variables that affect the behavior of Pigweed's env setup. The input variables include options such as the location of the CIPD cache directory and the Python executable to be used. The output variables include system executable search path, the location of the CIPD install directory, and the path to Pigweed's virtualenv. These variables are set by env setup and can be used by other tools. Most users will not need to set these variables.

Input Variables#

Pigweed AI summary: This section provides a list of environment variables that affect the behavior of Pigweed's env setup. These variables include the location of the CIPD cache directory, the Python executable to be used, and options to disable the spinner and banner during env setup. Most users will not need to set these variables.

The following environment variables affect env setup behavior. Most users will never need to set these.

CIPD_CACHE_DIR: Location of CIPD cache dir. Read by CIPD, but if unset will be defaulted to $HOME/.cipd-cache-dir.
PW_ACTIVATE_SKIP_CHECKS: If set, skip running pw doctor at end of bootstrap/activate. Intended to be used by automated tools but not interactively.
PW_BANNER_FUNC: Command to print a banner at the beginning of bootstrap.
PW_BOOTSTRAP_PYTHON: Python executable to be used, for example “python2” or “python3”. Defaults to “python”.
PW_CIPD_SERVICE_ACCOUNT_JSON: Value to pass as -service-account-json to CIPD invocations. This should point either to a service account JSON key file, or be the magical value :gce to tell the tool to fetch tokens from GCE metadata server.
PW_ENVIRONMENT_ROOT: Location to which packages are installed. Defaults to environment folder within the checkout root. This variable is cleared after environment setup is complete.
PW_ENVSETUP_DISABLE_SPINNER: Disable the spinner during env setup. Intended to be used when the output is being redirected to a log.
PW_ENVSETUP_DISABLE_SPINNER: Disable the console spinner that runs when waiting for env setup steps to complete.
PW_ENVSETUP_NO_BANNER: Skip printing the banner.
PW_ENVSETUP_QUIET: Disables all non-error output.
PW_PROJECT_ROOT: The absolute path of the project using Pigweed’s env setup. For Pigweed this is the same as PW_ROOT. This should be set by the project’s bootstrap script.
PW_ROOT: The absolute path to the Pigweed repository within PW_PROJECT_ROOT. This should be set by the project’s bootstrap script.

Output Variables#

Pigweed AI summary: This paragraph lists several environment variables that are set up by env setup. These variables include PATH, _PW_ACTUAL_ENVIRONMENT_ROOT, PW_CIPD_INSTALL_DIR, PW_*_CIPD_INSTALL_DIR, PW_PACKAGE_ROOT, and VIRTUAL_ENV. The variables serve different purposes such as defining the system executable search path, specifying the location of the environment installation, and indicating the path to Pigweed's virtualenv.

The following environment variables are set by env setup.

PATH: System executable search path. Many of the environment variables below are also added to this variable.
_PW_ACTUAL_ENVIRONMENT_ROOT: Location the environment was installed into. Separate from PW_ENVIRONMENT_ROOT because setting that implicitly and switching to another project directory causes unexpected behavior.
PW_CIPD_INSTALL_DIR: Top-level CIPD install directory. This is where the cipd executable is.
PW_*_CIPD_INSTALL_DIR: Each CIPD package file is installed into its own directory. This allows other tools to determine what those directories are. The * is replaced with an all-caps version of the basename of the package file, without the extension. (E.g., “path/foo.json” becomes PW_FOO_CIPD_INSTALL_DIR.)
PW_PACKAGE_ROOT: Location that packages installed by pw package will be installed to.
VIRTUAL_ENV: Path to Pigweed’s virtualenv.

Non-Shell Environments#

Pigweed AI summary: The paragraph describes how to use the "actions.json" file generated in a non-shell environment, such as an IDE or CI system. The file lists variables to set, clear, and modify. The order of the "append" and "prepend" actions in the file is important, and there is an example of the file provided. The paragraph also mentions that some of these variables are exposed to the GN build through the GNI file specified in the environment config. Finally, it states that using these

If using this outside of bash—for example directly from an IDE or CI system—users can process the actions.json file that’s generated in the location specified by the environment config. It lists variables to set, clear, and modify. An example actions.json is shown below. The “append” and “prepend” actions are listed in the order they should be applied, so the <pigweed-root>/out/host/host_tools entry should be at the beginning of PATH and not in the middle somewhere.

{
    "modify": {
        "PATH": {
            "append": [],
            "prepend": [
                "<pigweed-root>/environment/cipd",
                "<pigweed-root>/environment/cipd/pigweed",
                "<pigweed-root>/environment/cipd/pigweed/bin",
                "<pigweed-root>/environment/cipd/luci",
                "<pigweed-root>/environment/cipd/luci/bin",
                "<pigweed-root>/environment/pigweed-venv/bin",
                "<pigweed-root>/out/host/host_tools"
            ],
            "remove": []
        }
    },
    "set": {
        "PW_PROJECT_ROOT": "<pigweed-root>",
        "PW_ROOT": "<pigweed-root>",
        "_PW_ACTUAL_ENVIRONMENT_ROOT": "<pigweed-root>/environment",
        "PW_CIPD_INSTALL_DIR": "<pigweed-root>/environment/cipd",
        "CIPD_CACHE_DIR": "<home>/.cipd-cache-dir",
        "PW_PIGWEED_CIPD_INSTALL_DIR": "<pigweed-root>/environment/cipd/pigweed",
        "PW_LUCI_CIPD_INSTALL_DIR": "<pigweed-root>/environment/cipd/luci",
        "VIRTUAL_ENV": "<pigweed-root>/environment/pigweed-venv",
        "PYTHONHOME": null,
        "__PYVENV_LAUNCHER__": null
    }
}

Many of these variables are directly exposed to the GN build as well, through the GNI file specified in the environment config file.

declare_args() {
  pw_env_setup_CIPD_LUCI = "<environment-root>/cipd/packages/luci"
  pw_env_setup_CIPD_PIGWEED = "<environment-root>/cipd/packages/pigweed"
  pw_env_setup_PACKAGE_ROOT = "<environment-root>/packages"
  pw_env_setup_VIRTUAL_ENV = "<environment-root>/pigweed-venv"
}

It’s straightforward to use these variables.

import("//build_overrides/pigweed_environment.gni")

deps = [ "$pw_env_setup_CIPD_PIGWEED/..." ]

Implementation#

Pigweed AI summary: The implementation of the environment involves installing CIPD and Python packages in designated directories and modifying environment variables in setup scripts. The process is done in an operating system-agnostic manner and then written into operating system-specific files. The high-level commands are mapped to system-specific initialization files. The setup is completed by echoing "Setup Complete!" in the command prompt. The section also includes links to documentation on requirements and constraints files.

The environment is set up by installing CIPD and Python packages in PW_ENVIRONMENT_ROOT or <checkout>/environment, and saving modifications to environment variables in setup scripts in those directories. To support multiple operating systems this is done in an operating system-agnostic manner and then written into operating system-specific files to be sourced now and in the future when running activate.sh instead of bootstrap.sh. In the future these could be extended to C shell and PowerShell. A logical mapping of high-level commands to system-specific initialization files is shown below.

SET $PW_ROOT /home/$USER/pigweed

Windows

Linux & Mac (sh-compatible shells)

set PW_ROOT /home/%USER%/pigweed

PW_ROOT="/home/$USER/pigweed"
export PW_ROOT

PREPEND $PATH $PW_ROOT/.env/bin

Windows

Linux & Mac (sh-compatible shells)

set PATH=%PW_ROOT%/.env/bin;%PATH%

PATH="$(\
  echo "$PATH" | \
  sed "s|:$PW_ROOT/.env/bin:|:|g;" | \
  sed "s|^$PW_ROOT/.env/bin:||g;" | \
  sed "s|:$PW_ROOT/.env/bin$||g;")"
PATH="$PW_ROOT/.env/bin;$PATH"
export PATH

ECHO “Setup Complete!”

Windows

Linux & Mac (sh-compatible shells)

echo Setup Complete!

echo "Setup Complete!"