CLI reference#

Pigweed AI summary: The paragraph describes the CLI reference for the pw_tokenizer module. It mentions that the pw_tokenizer.encode command line tool can be used to encode format strings and optional arguments. It provides an example of how to use the tool and shows the output. It also mentions the detokenizing command line utilities provided by pw_tokenizer, including detokenize.py and serial_detokenizer.py. It provides examples of how to use these tools and suggests using the --help option for full usage details.

pw_tokenizer.encode: Encoding command line utility#

Pigweed AI summary: The pw_tokenizer.encode command line utility is used to encode format strings and optional arguments. It can be executed using the command "python -m pw_tokenizer.encode [-h] FORMAT_STRING [ARG ...]". An example is provided where a format string is encoded and the raw input, formatted input, token, encoded value, and prefixed Base64 value are displayed. The "--help" option can be used to view the full usage details.

The pw_tokenizer.encode command line tool can be used to encode format strings and optional arguments.

python -m pw_tokenizer.encode [-h] FORMAT_STRING [ARG ...]

Example:

$ python -m pw_tokenizer.encode "There's... %d many of %s!" 2 them
      Raw input: "There's... %d many of %s!" % (2, 'them')
Formatted input: There's... 2 many of them!
          Token: 0xb6ef8b2d
        Encoded: b'-\x8b\xef\xb6\x04\x04them' (2d 8b ef b6 04 04 74 68 65 6d) [10 bytes]
Prefixed Base64: $LYvvtgQEdGhlbQ==

See --help for full usage details.

Detokenizing command line utilties#

Pigweed AI summary: The "pw_tokenizer" module provides two command line utilities for detokenizing Base64-encoded strings. The first utility, "detokenize.py", can detokenize strings from files or from stdin. The second utility, "serial_detokenizer.py", can detokenize strings from a connected serial device. These utilities can be executed as runnable modules if the "pw_tokenizer" Python package is installed. Examples of how to use these utilities are provided, and the "--help" options

pw_tokenizer provides two standalone command line utilities for detokenizing Base64-encoded tokenized strings.

  • detokenize.py – Detokenizes Base64-encoded strings in files or from stdin.

  • serial_detokenizer.py – Detokenizes Base64-encoded strings from a connected serial device.

If the pw_tokenizer Python package is installed, these tools may be executed as runnable modules. For example:

# Detokenize Base64-encoded strings in a file
python -m pw_tokenizer.detokenize -i input_file.txt

# Detokenize Base64-encoded strings in output from a serial device
python -m pw_tokenizer.serial_detokenizer --device /dev/ttyACM0

See the --help options for these tools for full usage information.