pw_transfer#

pw_transfer is a reliable data transfer protocol which runs on top of Pigweed RPC.

Attention

pw_transfer is under construction and so is its documentation.

Usage#

C++#

Transfer thread#

Pigweed AI summary: A dedicated transfer thread is required to run transfers as a client or server, and it is used to process all transfer-related events safely. The transfer thread requires two buffers: a chunk buffer and an encode buffer. The chunk buffer is used to stage transfer packets received by the RPC system to be processed by the transfer thread, while the encode buffer is used by the transfer thread to encode outgoing RPC packets. A transfer thread can be shared by a transfer client and service running on the same system. All user

To run transfers as either a client or server (or both), a dedicated thread is required. The transfer thread is used to process all transfer-related events safely. The same transfer thread can be shared by a transfer client and service running on the same system.

Note

All user-defined transfer callbacks (i.e. the virtual interface of a Handler or completion function in a transfer client) will be invoked from the transfer thread’s context.

In order to operate, a transfer thread requires two buffers:

  • The first is a chunk buffer. This is used to stage transfer packets received by the RPC system to be processed by the transfer thread. It must be large enough to store the largest possible chunk the system supports.

  • The second is an encode buffer. This is used by the transfer thread to encode outgoing RPC packets. It is necessarily larger than the chunk buffer. Typically, this is sized to the system’s maximum transmission unit at the transport layer.

A transfer thread is created by instantiating a pw::transfer::Thread. This class derives from pw::thread::ThreadCore, allowing it to directly be used when creating a system thread. Refer to Thread Creation for additional information.

Example thread configuration

#include "pw_transfer/transfer_thread.h"

namespace {

// The maximum number of concurrent transfers the thread should support as
// either a client or a server. These can be set to 0 (if only using one or
// the other).
constexpr size_t kMaxConcurrentClientTransfers = 5;
constexpr size_t kMaxConcurrentServerTransfers = 3;

// The maximum payload size that can be transmitted by the system's
// transport stack. This would typically be defined within some transport
// header.
constexpr size_t kMaxTransmissionUnit = 512;

// The maximum amount of data that should be sent within a single transfer
// packet. By necessity, this should be less than the max transmission unit.
//
// pw_transfer requires some additional per-packet overhead, so the actual
// amount of data it sends may be lower than this.
constexpr size_t kMaxTransferChunkSizeBytes = 480;

// Buffers for storing and encoding chunks (see documentation above).
std::array<std::byte, kMaxTransferChunkSizeBytes> chunk_buffer;
std::array<std::byte, kMaxTransmissionUnit> encode_buffer;

pw::transfer::Thread<kMaxConcurrentClientTransfers,
                     kMaxConcurrentServerTransfers>
    transfer_thread(chunk_buffer, encode_buffer);

}  // namespace

// pw::transfer::TransferThread is the generic, non-templated version of the
// Thread class. A Thread can implicitly convert to a TransferThread.
pw::transfer::TransferThread& GetSystemTransferThread() {
  return transfer_thread;
}

Transfer server#

Pigweed AI summary: The pw_transfer provides an RPC service for running transfers through an RPC server. To read or write data from a device, a Handler interface is defined, which represents a transferable resource. Custom transfer handler implementations should derive from ReadOnlyHandler, WriteOnlyHandler, or ReadWriteHandler as appropriate and override Prepare and Finalize methods if necessary. Handlers are registered with the transfer service, which may be done during system initialization or dynamically at runtime to support ephemeral transfer resources. The transfer service is instantiated with a

pw_transfer provides an RPC service for running transfers through an RPC server.

To know how to read data from or write data to device, a Handler interface is defined (pw_transfer/public/pw_transfer/handler.h). Transfer handlers represent a transferable resource, wrapping a stream reader and/or writer with initialization and completion code. Custom transfer handler implementations should derive from ReadOnlyHandler, WriteOnlyHandler, or ReadWriteHandler as appropriate and override Prepare and Finalize methods if necessary.

A transfer handler should be implemented and instantiated for each unique resource that can be transferred to or from a device. Each instantiated handler must have a globally-unique integer ID used to identify the resource.

Handlers are registered with the transfer service. This may be done during system initialization (for static resources), or dynamically at runtime to support ephemeral transfer resources.

Example transfer handler implementation

#include "pw_stream/memory_stream.h"
#include "pw_transfer/transfer.h"

// A simple transfer handler which reads data from an in-memory buffer.
class SimpleBufferReadHandler : public pw::transfer::ReadOnlyHandler {
 public:
  SimpleReadTransfer(uint32_t resource_id, pw::ConstByteSpan data)
      : ReadOnlyHandler(resource_id), reader_(data) {
    set_reader(reader_);
  }

 private:
  pw::stream::MemoryReader reader_;
};

The transfer service is instantiated with a reference to the system’s transfer thread and registered with the system’s RPC server.

Example transfer service initialization

#include "pw_transfer/transfer.h"

namespace {

// In a write transfer, the maximum number of bytes to receive at one time
// (potentially across multiple chunks), unless specified otherwise by the
// transfer handler's stream::Writer.
constexpr size_t kDefaultMaxBytesToReceive = 1024;

pw::transfer::TransferService transfer_service(
    GetSystemTransferThread(), kDefaultMaxBytesToReceive);

// Instantiate a handler for the data to be transferred. The resource ID will
// be used by the transfer client and server to identify the handler.
constexpr uint32_t kMagicBufferResourceId = 1;
char magic_buffer_to_transfer[256] = { /* ... */ };
SimpleBufferReadHandler magic_buffer_handler(
    kMagicBufferResourceId, magic_buffer_to_transfer);

}  // namespace

void InitTransferService() {
  // Register the handler with the transfer service, then the transfer service
  // with an RPC server.
  transfer_service.RegisterHandler(magic_buffer_handler);
  GetSystemRpcServer().RegisterService(transfer_service);
}

Transfer client#

pw_transfer provides a transfer client capable of running transfers through an RPC client.

Note

Currently, a transfer client is only capable of running transfers on a single RPC channel. This may be expanded in the future.

The transfer client provides the following two APIs for starting data transfers:

pw::Status pw::transfer::Client::Read(uint32_t resource_id, pw::stream::Writer &output, CompletionFunc &&on_completion, pw::chrono::SystemClock::duration timeout = cfg::kDefaultChunkTimeout, pw::transfer::ProtocolVersion version = kDefaultProtocolVersion)#

Reads data from a transfer server to the specified pw::stream::Writer. Invokes the provided callback function with the overall status of the transfer.

Due to the asynchronous nature of transfer operations, this function will only return a non-OK status if it is called with bad arguments. Otherwise, it will return OK and errors will be reported through the completion callback.

pw::Status pw::transfer::Client::Write(uint32_t resource_id, pw::stream::Reader &input, CompletionFunc &&on_completion, pw::chrono::SystemClock::duration timeout = cfg::kDefaultChunkTimeout, pw::transfer::ProtocolVersion version = kDefaultProtocolVersion)#

Writes data from a source pw::stream::Reader to a transfer server. Invokes the provided callback function with the overall status of the transfer.

Due to the asynchronous nature of transfer operations, this function will only return a non-OK status if it is called with bad arguments. Otherwise, it will return OK and errors will be reported through the completion callback.

Example client setup

#include "pw_transfer/client.h"

namespace {

// RPC channel on which transfers should be run.
constexpr uint32_t kChannelId = 42;

pw::transfer::Client transfer_client(
    GetSystemRpcClient(), kChannelId, GetSystemTransferThread());

}  // namespace

Status ReadMagicBufferSync(pw::ByteSpan sink) {
  pw::stream::Writer writer(sink);

  struct {
    pw::sync::ThreadNotification notification;
    pw::Status status;
  } transfer_state;

  transfer_client.Read(
      kMagicBufferResourceId,
      writer,
      [&transfer_state](pw::Status status) {
        transfer_state.status = status;
        transfer_state.notification.release();
      });

  // Block until the transfer completes.
  transfer_state.notification.acquire();
  return transfer_state.status;
}

Atomic File Transfer Handler#

Pigweed AI summary: The Atomic File Transfer Handler is a specialized interface that guarantees the target file of a transfer is always in a correct state. It uses a temporary file to ensure that if any transfer failure occurs, the target file is either not created or not updated. This handler is available for file transfers with atomic semantics and is used in conjunction with the generic Handler interface.

Transfers are handled using the generic Handler interface. A specialized Handler, AtomicFileTransferHandler is available to handle file transfers with atomic semantics. It guarantees that the target file of the transfer is always in a correct state. A temporary file is written to prior to updating the target file. If any transfer failure occurs, the transfer is aborted and the target file is either not created or not updated.

Module Configuration Options#

The following configurations can be adjusted via compile-time configuration of this module, see the module documentation for more details.

PW_TRANSFER_DEFAULT_MAX_RETRIES#

The default maximum number of times a transfer should retry sending a chunk when no response is received. This can later be configured per-transfer.

PW_TRANSFER_DEFAULT_MAX_LIFETIME_RETRIES#

The default maximum number of times a transfer should retry sending any chunk over the course of its entire lifetime.

This number should be high, particularly if long-running transfers are expected. Its purpose is to prevent transfers from getting stuck in an infinite loop.

PW_TRANSFER_DEFAULT_TIMEOUT_MS#

The default amount of time, in milliseconds, to wait for a chunk to arrive before retrying. This can later be configured per-transfer.

PW_TRANSFER_DEFAULT_INITIAL_TIMEOUT_MS#

The default amount of time, in milliseconds, to wait for an initial server response to a transfer before retrying. This can later be configured per-transfer.

This is set separately to PW_TRANSFER_DEFAULT_TIMEOUT_MS as transfers may require additional time for resource initialization (e.g. erasing a flash region before writing to it).

PW_TRANSFER_DEFAULT_EXTEND_WINDOW_DIVISOR#

The fractional position within a window at which a receive transfer should extend its window size to minimize the amount of time the transmitter spends blocked.

For example, a divisor of 2 will extend the window when half of the requested data has been received, a divisor of three will extend at a third of the window, and so on.

Python#

Provides a simple interface for transferring bulk data over pw_rpc.

exception pw_transfer.Error(resource_id: int, status: Status)#

Exception raised when a transfer fails.

Stores the ID of the failed transfer resource and the error that occurred.

__init__(resource_id: int, status: Status)#
class pw_transfer.Manager(rpc_transfer_service, *, default_response_timeout_s: float = 2.0, initial_response_timeout_s: float = 4.0, max_retries: int = 3, max_lifetime_retries: int = 1500, default_protocol_version=ProtocolVersion.LEGACY)#

A manager for transmitting data through an RPC TransferService.

This should be initialized with an active Manager over an RPC channel. Only one instance of this class should exist for a configured RPC TransferService – the Manager supports multiple simultaneous transfers.

When created, a Manager starts a separate thread in which transfer communications and events are handled.

__init__(rpc_transfer_service, *, default_response_timeout_s: float = 2.0, initial_response_timeout_s: float = 4.0, max_retries: int = 3, max_lifetime_retries: int = 1500, default_protocol_version=ProtocolVersion.LEGACY)#

Initializes a Manager on top of a TransferService.

Parameters:

  • rpc_transfer_service – the pw_rpc transfer service client

  • default_response_timeout_s – max time to wait between receiving packets

  • initial_response_timeout_s – timeout for the first packet; may be longer to account for transfer handler initialization

  • max_retires – number of times to retry a single package after a timeout

  • max_lifetime_retires – Cumulative maximum number of times to retry over the course of the transfer before giving up.

read(resource_id: int, progress_callback: Optional[Callable[[ProgressStats], Any]] = None, protocol_version: Optional[ProtocolVersion] = None) → bytes#

Receives (“downloads”) data from the server.

Parameters:

  • resource_id – ID of the resource from which to read.

  • progress_callback – Optional callback periodically invoked throughout the transfer with the transfer state. Can be used to provide user- facing status updates such as progress bars.

Raises:

Error – the transfer failed to complete

write(resource_id: int, data: Union[bytes, str], progress_callback: Optional[Callable[[ProgressStats], Any]] = None, protocol_version: Optional[ProtocolVersion] = None) → None#

Transmits (“uploads”) data to the server.

Parameters:

  • resource_id – ID of the resource to which to write.

  • data – Data to send to the server.

  • progress_callback – Optional callback periodically invoked throughout the transfer with the transfer state. Can be used to provide user- facing status updates such as progress bars.

Raises:

Error – the transfer failed to complete

class pw_transfer.ProgressStats(bytes_sent: int, bytes_confirmed_received: int, total_size_bytes: Union[int, NoneType])#
__init__(bytes_sent: int, bytes_confirmed_received: int, total_size_bytes: Optional[int]) → None#
class pw_transfer.ProtocolVersion(value)#

Supported versions of pw_transfer’s RPC data transfer protocol.

Example

import pw_transfer

# Initialize a Pigweed RPC client; see pw_rpc docs for more info.
rpc_client = CustomRpcClient()
rpcs = rpc_client.channel(1).rpcs

transfer_service = rpcs.pw.transfer.Transfer
transfer_manager = pw_transfer.Manager(transfer_service)

try:
  # Read the transfer resource with ID 3 from the server.
  data = transfer_manager.read(3)
except pw_transfer.Error as err:
  print('Failed to read:', err.status)

try:
  # Send some data to the server. The transfer manager does not have to be
  # reinitialized.
  transfer_manager.write(2, b'hello, world')
except pw_transfer.Error as err:
  print('Failed to write:', err.status)

Typescript#

Pigweed AI summary: This section is about Typescript and how it provides a simple interface for transferring bulk data over pw_rpc. The example code shows how to use Typescript to import pw_transfer and create a new CustomRpcClient. It also demonstrates how to use the Manager to read and write data, with error handling included.

Provides a simple interface for transferring bulk data over pw_rpc.

Example

import { pw_transfer } from 'pigweedjs';
const { Manager } from pw_transfer;

const client = new CustomRpcClient();
service = client.channel()!.service('pw.transfer.Transfer')!;

const manager = new Manager(service, DEFAULT_TIMEOUT_S);

manager.read(3, (stats: ProgressStats) => {
  console.log(`Progress Update: ${stats}`);
}).then((data: Uint8Array) => {
  console.log(`Completed read: ${data}`);
}).catch(error => {
  console.log(`Failed to read: ${error.status}`);
});

manager.write(2, textEncoder.encode('hello world'))
  .catch(error => {
    console.log(`Failed to read: ${error.status}`);
  });

Java#

Pigweed AI summary: The Java client pw_transfer provides a ListenableFuture to represent the results of a read or write transfer. The code example shows how to create a new transfer client, start a read or write transfer, and get the data from the read transfer or wait for the write transfer to complete.

pw_transfer provides a Java client. The transfer client returns a ListenableFuture to represent the results of a read or write transfer.

import dev.pigweed.pw_transfer.TransferClient;

public class TheClass  {
  public void DoTransfer(MethodClient transferReadMethodClient,
                         MethodClient transferWriteMethodClient) {
    // Create a new transfer client.
    TransferClient client = new TransferClient(
        transferReadMethodClient,
        transferWriteMethodClient,
        TransferTimeoutSettings.builder()
            .setTimeoutMillis(TRANSFER_TIMEOUT_MS)
            .setMaxRetries(MAX_RETRIES)
            .build());

    // Start a read transfer.
    ListenableFuture<byte[]> readTransfer = client.read(123);

    // Start a write transfer.
    ListenableFuture<Void> writeTransfer = client.write(123, dataToWrite);

    // Get the data from the read transfer.
    byte[] readData = readTransfer.get();

    // Wait for the write transfer to complete.
    writeTransfer.get();
  }
}

Protocol#

Pigweed AI summary: The protocol for data transfers in the pw_transfer library involves exchanging chunks of data, metadata, and control parameters over an RPC stream. Each chunk has a type that determines its purpose and fields. Transfers are performed for specific resources, which have unique identifiers. The series of chunks exchanged for a resource constitute a transfer session. The protocol includes mechanisms for reliability, such as timeouts, data retransmissions, and handshakes. Transfers begin with a three-way handshake to identify the resource, assign a

Chunks#

Pigweed AI summary: The transfer of data, metadata, and control parameters occurs through a series of chunks exchanged over an RPC stream. Each chunk has a specific type that determines the information it holds and the meaning of its fields. These chunks are protobuf messages, and their definition can be found in the provided reference.

Transfers run as a series of chunks exchanged over an RPC stream. Chunks can contain transferable data, metadata, and control parameters. Each chunk has an associated type, which determines what information it holds and the semantics of its fields.

The chunk is a protobuf message, whose definition can be found here.

Resources and sessions#

Pigweed AI summary: This paragraph explains that transfers are performed on specific resources, which are streams of data that can be read from or written to. Each resource has a unique identifier defined by the server-side transfer node. The transfer session consists of a series of chunks exchanged during the transfer operation, starting from the opening chunk and ending with a terminating chunk or timeout. The client assigns a unique ID to each session, allowing the server to identify transfers across multiple clients.

Transfers are run for a specific resource — a stream of data which can be read from or written to. Resources have a system-specific integral identifier defined by the implementers of the server-side transfer node.

The series of chunks exchanged in an individual transfer operation for a resource constitute a transfer session. The session runs from its opening chunk until either a terminating chunk is received or the transfer times out. Sessions are assigned IDs by the client that starts them, which are unique over the RPC channel between the client and server, allowing the server to identify transfers across multiple clients.

Reliability#

Pigweed AI summary: The pw_transfer protocol aims to be reliable in transferring data, and implements its own mechanisms for reliability such as timeouts, data retransmissions, and handshakes. However, it is important to note that a transfer can only be reliable if the underlying data stream is seekable, as a non-seekable stream could cause a transfer to prematurely terminate following a packet drop.

pw_transfer attempts to be a reliable data transfer protocol.

As Pigweed RPC is considered an unreliable communications system, pw_transfer implements its own mechanisms for reliability. These include timeouts, data retransmissions, and handshakes.

Note

A transfer can only be reliable if its underlying data stream is seekable. A non-seekable stream could prematurely terminate a transfer following a packet drop.

Opening handshake#

Pigweed AI summary: The process of transferring resources begins with a three-way handshake that identifies the resource, assigns a session ID, and synchronizes the protocol version. A transfer client initiates a read or write transfer by sending the resource ID and a unique session ID in a START chunk, indicating its intention to begin a new transfer. The transfer server checks if the requested resource is available and prepares it for the operation, which typically involves opening a data stream and any additional user-specified setup. The server accepts the client's

Transfers begin with a three-way handshake, whose purpose is to identify the resource being transferred, assign a session ID, and synchronize the protocol version to use.

A read or write transfer for a resource is initiated by a transfer client. The client sends the ID of the resource to the server alongside a unique session ID in a START chunk, indicating that it wishes to begin a new transfer. This chunk additionally encodes the protocol version which the client is configured to use.

Upon receiving a START chunk, the transfer server checks whether the requested resource is available. If so, it prepares the resource for the operation, which typically involves opening a data stream, alongside any additional user-specified setup. The server accepts the client’s session ID, then responds to the client with a START_ACK chunk containing the resource, session, and configured protocol version for the transfer.

Transfer completion#

Pigweed AI summary: This section discusses the process of completing a transfer operation, which can be terminated by either side at any time by sending a "COMPLETION" chunk with the final status of the transfer. Upon receiving this chunk, the transfer peer cancels any pending operations, runs its set of cleanups, and responds with a "COMPLETION_ACK", fully ending the session from the peer's side. The terminator's session remains active waiting for a "COMPLETION_ACK", and if not received after a

Either side of a transfer can terminate the operation at any time by sending a COMPLETION chunk containing the final status of the transfer. When a COMPLETION chunk is sent, the terminator of the transfer performs local cleanup, then waits for its peer to acknowledge the completion.

Upon receving a COMPLETION chunk, the transfer peer cancels any pending operations, runs its set of cleanups, and responds with a COMPLETION_ACK, fully ending the session from the peer’s side.

The terminator’s session remains active waiting for a COMPLETION_ACK. If not received after a timeout, it re-sends its COMPLETION chunk. The session ends either following receipt of the acknowledgement or if a maximum number of retries is hit.

Server to client transfer (read)#

Pigweed AI summary: This section discusses the server to client transfer (read) module and provides the protocol definition. It includes an image of the module's icon.

../_images/read.svg

Client to server transfer (write)#

Pigweed AI summary: This section is titled "Client to server transfer (write)" and includes an image of a file being transferred from a client to a server.

../_images/write.svg

Protocol buffer definition#

Pigweed AI summary: This is a protocol buffer definition for the transfer RPC service used to send data between a client and server. The definition includes a service called Transfer with two RPC methods, Read and Write, and a message called Chunk that represents a chunk of data sent by the transfer service. The Chunk message includes fields for configuring transfer parameters such as the size of individual chunks, the delay between chunks, and the estimated bytes remaining to read or write. It also includes a field for a Pigweed status code indicating the completion

syntax = "proto3";

package pw.transfer;

option java_multiple_files = true;
option java_package = "dev.pigweed.pw_transfer";

// The transfer RPC service is used to send data between the client and server.
service Transfer {
  // Transfer data from the server to the client; a "download" from the client's
  // perspective.
  rpc Read(stream Chunk) returns (stream Chunk);

  // Transfer data from the client to the server; an "upload" from the client's
  // perspective.
  rpc Write(stream Chunk) returns (stream Chunk);
}

// Represents a chunk of data sent by the transfer service. Includes fields for
// configuring the transfer parameters.
//
// Notation: (Read|Write) (→|←)
//   X → Means client sending data to the server.
//   X ← Means server sending data to the client.
message Chunk {
  // Represents the source or destination of the data. May be ephemeral or
  // stable depending on the implementation. Sent in every request to identify
  // the transfer target.
  //
  // LEGACY FIELD ONLY. Split into resource_id and session_id in transfer v2.
  //
  //  Read → ID of transfer
  //  Read ← ID of transfer
  // Write → ID of transfer
  // Write ← ID of transfer
  uint32 transfer_id = 1;

  // Used by the receiver to indicate how many bytes it can accept. The
  // transmitter sends this much data, divided into chunks no larger than
  // max_chunk_size_bytes. The receiver then starts another window by sending
  // request_bytes again with a new offset.
  //
  //  Read → The client requests this many bytes to be sent.
  //  Read ← N/A
  // Write → N/A
  // Write ← The server requests this many bytes to be sent.
  optional uint32 pending_bytes = 2;

  // Maximum size of an individual chunk. The transmitter may send smaller
  // chunks if required.
  //
  //  Read → Set maximum size for subsequent chunks.
  //  Read ← N/A
  // Write → N/A
  // Write ← Set maximum size for subsequent chunks.
  optional uint32 max_chunk_size_bytes = 3;

  // Minimum required delay between chunks. The transmitter may delay longer if
  // desired.
  //
  //  Read → Set minimum delay for subsequent chunks.
  //  Read ← N/A
  // Write → N/A
  // Write ← Set minimum delay for subsequent chunks.
  optional uint32 min_delay_microseconds = 4;

  // On writes, the offset of the data. On reads, the offset at which to read.
  //
  //  Read → Read data starting at this offset.
  //  Read ← Offset of the data.
  // Write → Offset of the data.
  // Write ← Write data starting at this offset.
  uint64 offset = 5;

  // The data that was read or the data to write.
  //
  //  Read → N/A
  //  Read ← Data read
  // Write → Data to write
  // Write ← N/A
  bytes data = 6;

  // Estimated bytes remaining to read/write. Optional except for the last data
  // chunk, for which remaining_bytes must be set to 0.
  //
  // The sender can set remaining_bytes at the beginning of a read/write so that
  // the receiver can track progress or cancel the transaction if the value is
  // too large.
  //
  //  Read → N/A
  //  Read ← Remaining bytes to read, excluding any data in this chunk. Set to
  //         0 for the last chunk.
  // Write → Remaining bytes to write, excluding any data in is chunk. Set to
  //         0 for the last chunk.
  // Write ← N/A
  optional uint64 remaining_bytes = 7;

  // Pigweed status code indicating the completion of a transfer. This is only
  // present in the final packet sent by either the transmitter or receiver.
  //
  // The possible status codes and their meanings are listed below:
  //
  //   OK: Transfer completed successfully.
  //   DATA_LOSS: Transfer data could not be read/written (e.g. corruption).
  //   INVALID_ARGUMENT: Received malformed chunk.
  //   NOT_FOUND: The requested resource ID is not registered (read/write).
  //   OUT_OF_RANGE: The requested offset is larger than the data (read/write).
  //   RESOURCE_EXHAUSTED: Concurrent transfer limit reached.
  //   UNIMPLEMENTED: Resource ID does not support requested operation (e.g.
  //       trying to write to a read-only transfer).
  //
  //  Read → Transfer complete.
  //  Read ← Transfer complete.
  // Write → Transfer complete.
  // Write ← Transfer complete.
  optional uint32 status = 8;

  // The offset up to which the transmitter can send data before waiting for the
  // receiver to acknowledge.
  //
  //  Read → Offset up to which the server can send without blocking.
  //  Read ← N/A
  // Write → N/A
  // Write ← Offset up to which the client can send without blocking.
  //
  // TODO(frolv): This will replace the pending_bytes field. Once all uses of
  // transfer are migrated, that field should be removed.
  uint32 window_end_offset = 9;

  enum Type {
    // Chunk containing transfer data.
    DATA = 0;

    // First chunk of a transfer (only sent by the client).
    START = 1;

    // Transfer parameters indicating that the transmitter should retransmit
    // from the specified offset.
    PARAMETERS_RETRANSMIT = 2;

    // Transfer parameters telling the transmitter to continue sending up to
    // index `offset + pending_bytes` of data. If the transmitter is already
    // beyond `offset`, it does not have to rewind.
    PARAMETERS_CONTINUE = 3;

    // Sender of the chunk is terminating the transfer.
    COMPLETION = 4;

    // Acknowledge the completion of a transfer. Currently unused.
    // TODO(konkers): Implement this behavior.
    COMPLETION_ACK = 5;

    // Acknowledges a transfer start request, accepting the session ID for the
    // transfer and optionally negotiating the protocol version. Sent from
    // server to client.
    START_ACK = 6;

    // Confirmation of a START_ACK's negotiated parameters, sent by the client
    // to the server. Initiates the data transfer proper.
    START_ACK_CONFIRMATION = 7;
  };

  // The type of this chunk. This field should only be processed when present.
  // TODO(frolv): Update all users of pw_transfer and remove the optional
  // semantics from this field.
  //
  //  Read → Chunk type (start/parameters).
  //  Read ← Chunk type (data).
  // Write → Chunk type (data).
  // Write ← Chunk type (start/parameters).
  optional Type type = 10;

  // Unique identifier for the source or destination of transfer data. May be
  // stable or ephemeral depending on the implementation. Only sent during the
  // initial handshake phase of a version 2 or higher transfer.
  //
  //  Read → ID of transferable resource
  //  Read ← ID of transferable resource
  // Write → ID of transferable resource
  // Write ← ID of transferable resource
  optional uint32 resource_id = 11;

  // Unique identifier for a specific transfer session. Chosen by the transfer
  // client during the initial handshake phase, and persists for the remainder
  // of that transfer operation.
  //
  //  Read → ID of transfer session
  //  Read ← ID of transfer session
  // Write → ID of transfer session
  // Write ← ID of transfer session
  optional uint32 session_id = 12;

  // The protocol version to use for this transfer. Only sent during the initial
  // handshake phase of a version 2 or higher transfer to negotiate a common
  // protocol version between the client and server.
  //
  //  Read → Desired (START) or configured (START_ACK_CONFIRMATION) version.
  //  Read ← Configured protocol version (START_ACK).
  // Write → Desired (START) or configured (START_ACK_CONFIRMATION) version.
  // Write ← Configured protocol version (START_ACK).
  optional uint32 protocol_version = 13;

  // Unique identifier for a specific transfer session. Chosen by the transfer
  // client during the initial handshake phase. This field is used to request a
  // session during the handshake, after which the regular session_id field is
  // used.
  //
  //  Read → Requested ID of transfer session
  //  Read ← N/A
  // Write → Requested ID of transfer session
  // Write ← N/A
  optional uint32 desired_session_id = 14;
}

Errors#

Pigweed AI summary: The paragraph summarizes the meaning of each status code in the context of protocol errors. It explains the different status codes and their implications when sent by the sender or the receiver. The paragraph also mentions that the table provided describes these status codes in detail.

Protocol errors#

Pigweed AI summary: This paragraph summarizes the meaning of each status code in the "Protocol errors" section. The table provides information on the status codes, whether they are sent by the sender or receiver, and their corresponding descriptions. The status codes include "OK" for successful handling of data, "ABORTED" for transfer cancellation due to client restart, "CANCELLED" for client-initiated transfer cancellation, "DATA_LOSS" for errors in reading or writing data, "FAILED_PRECONDITION" for receiving a chunk

The following table describes the meaning of each status code when sent by the sender or the receiver (see Transfer roles).

Status

Sent by sender

Sent by receiver

OK

(not sent)

All data was received and handled successfully.

ABORTED

The service aborted the transfer because the client restarted it. This status is passed to the transfer handler, but not sent to the client because it restarted the transfer.

CANCELLED

The client cancelled the transfer.

DATA_LOSS

Failed to read the data to send. The Reader returned an error.

Failed to write the received data. The Writer returned an error.

FAILED_PRECONDITION

Received chunk for transfer that is not active.

INVALID_ARGUMENT

Received a malformed packet.

INTERNAL

An assumption of the protocol was violated. Encountering INTERNAL indicates that there is a bug in the service or client implementation.

PERMISSION_DENIED

The transfer does not support the requested operation (either reading or writing).

RESOURCE_EXHAUSTED

The receiver requested zero bytes, indicating their storage is full, but there is still data to send.

Storage is full.

UNAVAILABLE

The service is busy with other transfers and cannot begin a new transfer at this time.

UNIMPLEMENTED

Out-of-order chunk was requested, but seeking is not supported.

(not sent)

Transfer roles#

Pigweed AI summary: This section explains the process of transferring data between a sender and a receiver. The sender initiates the transfer and sends data to the receiver, who controls the transfer and sends the final status when it's complete. In read transfers, the client is the receiver, and in write transfers, the client is the sender. The section also includes diagrams showing the flow of data from the sender to the receiver and vice versa.

Every transfer has two participants: the sender and the receiver. The sender transmits data to the receiver. The receiver controls how the data is transferred and sends the final status when the transfer is complete.

In read transfers, the client is the receiver and the service is the sender. In write transfers, the client is the sender and the service is the receiver.

Sender flow#

Pigweed AI summary: This paragraph describes the flow of data transfer from a client to a receiver. The sender initiates the transfer and requests data from the receiver. The sender then sends chunks of data and waits for confirmation that all chunks have been sent. If all chunks have been sent, the sender waits for the receiver to confirm receipt of the final chunk. If not all chunks have been sent, the sender requests to send the remaining chunks. Once the receiver confirms receipt of the final chunk, the transfer is complete.

graph TD start([Client initiates<br>transfer]) -->data_request data_request[Receive transfer<br>parameters]-->send_chunk send_chunk[Send chunk]-->sent_all sent_all{Sent final<br>chunk?} -->|yes|wait sent_all-->|no|sent_requested sent_requested{Sent all<br>pending?}-->|yes|data_request sent_requested-->|no|send_chunk wait[Wait for receiver]-->is_done is_done{Received<br>final chunk?}-->|yes|done is_done-->|no|data_request done([Transfer complete])

Receiver flow#

Pigweed AI summary: This section describes the flow of data transfer from a client to a receiver. The process involves setting transfer parameters, waiting for chunks of data, checking the offset and processing the chunks. The final chunk is signaled for completion and the transfer is complete. If all pending data is received, the transfer is complete, otherwise the process continues.

graph TD start([Client initiates<br>transfer]) -->request_bytes request_bytes[Set transfer<br>parameters]-->wait wait[Wait for chunk]-->received_chunk received_chunk{Received<br>chunk by<br>deadline?}-->|no|request_bytes received_chunk-->|yes|check_chunk check_chunk{Correct<br>offset?} -->|yes|process_chunk check_chunk --> |no|request_bytes process_chunk[Process chunk]-->final_chunk final_chunk{Final<br>chunk?}-->|yes|signal_completion final_chunk{Final<br>chunk?}-->|no|received_requested received_requested{Received all<br>pending?}-->|yes|request_bytes received_requested-->|no|wait signal_completion[Signal completion]-->done done([Transfer complete])

Legacy protocol#

Pigweed AI summary: The "pw_transfer" protocol has a legacy mode that lacks certain reliability features of the modern protocol. In the legacy mode, there is no distinction between a transfer resource and session, and there is no opening or terminating handshake phase. Transfer clients can request the legacy protocol, but it is strongly advised against using it in new code. Transfer server and client implementations can automatically switch to the legacy protocol if required.

pw_transfer was initially released into production prior to several of the reliability improvements of its modern protocol. As a result of this, transfer implementations support a “legacy” protocol mode, in which transfers run without utilizing these features.

The primary differences between the legacy and modern protocols are listed below.

  • There is no distinction between a transfer resource and session — a single transfer_id field represents both. Only one transfer for a given resource can run at a time, and it is not possible to determine where one transfer for a resource ends and the next begins.

  • The legacy protocol has no opening handshake phase. The client initiates with a transfer ID and starting transfer parameters (during a read), and the data transfer phase begins immediately.

  • The legacy protocol has no terminating handshake phase. When either end completes a transfer by sending a status chunk, it does not wait for the peer to acknowledge. Resources used by the transfer are immediately freed, and there is no guarantee that the peer is notified of completion.

Transfer clients request the latest transfer protocol version by default, but may be configured to request the legacy protocol. Transfer server and client implementations detect if their transfer peer is running the legacy protocol and automatically switch to it if required, even if they requested a newer protocol version. It is strongly unadvised to use the legacy protocol in new code.

Integration tests#

Pigweed AI summary: The pw_transfer module has integration tests that verify the correctness of implementations in different languages. To run the tests on your machine, use the provided command. There is also a suite of backwards-compatibility tests that continuously validate a degree of backwards-compatibility with older pw_transfer servers and clients. The CIPD package contents can be created with a specific command. The integration tests permit injection of client/server/proxy binaries to use when running the tests. By default, these tests are not run in CQ

The pw_transfer module has a set of integration tests that verify the correctness of implementations in different languages. Test source code.

To run the tests on your machine, run

$ bazel test --features=c++17 \
      pw_transfer/integration_test:cross_language_small_test \
      pw_transfer/integration_test:cross_language_medium_test

Note

There is a large test that tests transfers that are megabytes in size. These are not run automatically, but can be run manually via the pw_transfer/integration_test:cross_language_large_test test. These are VERY slow, but exist for manual validation of real-world use cases.

The integration tests permit injection of client/server/proxy binaries to use when running the tests. This allows manual testing of older versions of pw_transfer against newer versions.

# Test a newer version of pw_transfer against an old C++ client that was
# backed up to another directory.
$ bazel run pw_transfer/integration_test:cross_language_medium_test -- \
    --cpp-client-binary ../old_pw_transfer_version/cpp_client

Backwards compatibility tests#

Pigweed AI summary: The pw_transfer tool has a suite of backwards-compatibility tests that continuously validate compatibility with older servers and clients by running tests between older and latest binaries. The CIPD package contents can be created with a specific command, and to update the CIPD package itself, internal documentation should be followed.

pw_transfer includes a suite of backwards-compatibility tests that are intended to continuously validate a degree of backwards-compatibility with older pw_transfer servers and clients. This is done by retrieving older binaries hosted in CIPD and running tests between the older client/server binaries and the latest binaries.

The CIPD package contents can be created with this command:

To update the CIPD package itself, follow the internal documentation for updating a CIPD package.

CI/CQ integration#

Pigweed AI summary: The article discusses the integration of CI/CQ and the current status of tests in CI. The tests are not run in CQ by default due to their slow speed, but it is possible to request them to be run by adding a specific line to the commit message footer.

Current status of the test in CI.

By default, these tests are not run in CQ (on presubmit) because they are too slow. However, you can request that the tests be run in presubmit on your change by adding to following line to the commit message footer:

Cq-Include-Trybots: luci.pigweed.try:pigweed-integration-transfer