From 865d3aac8c1a7cec78a9f59e8c77459c43047b3b Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Tue, 24 Jun 2025 22:35:37 -0700 Subject: [PATCH 1/9] Partial Messages Extension Co-authored-by: Csaba Kiraly --- pubsub/gossipsub/extensions/extensions.proto | 4 + pubsub/gossipsub/partial-messages.md | 193 +++++++++++++++++++ 2 files changed, 197 insertions(+) create mode 100644 pubsub/gossipsub/partial-messages.md diff --git a/pubsub/gossipsub/extensions/extensions.proto b/pubsub/gossipsub/extensions/extensions.proto index 41c3ae8a8..402186bee 100644 --- a/pubsub/gossipsub/extensions/extensions.proto +++ b/pubsub/gossipsub/extensions/extensions.proto @@ -8,6 +8,8 @@ message ControlExtensions { // encoded with at least 4 bytes optional bool testExtension = 6492434; + + optional bool partialMessages = 16418754; } message ControlMessage { @@ -36,4 +38,6 @@ message RPC { // bytes optional TestExtension testExtension = 6492434; + + optional PartialMessagesExtension partial = 16418754; } diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md new file mode 100644 index 000000000..ad0331348 --- /dev/null +++ b/pubsub/gossipsub/partial-messages.md @@ -0,0 +1,193 @@ +# Partial Messages Extension + +| Lifecycle Stage | Maturity | Status | Latest Revision | +| --------------- | ------------- | ------ | --------------- | +| 1A | Working Draft | Active | r0, 2025-06-23 | + +Authors: [@marcopolo, @cskiraly] + +Interest Group: TODO + +[@marcopolo]: https://github.com/marcopolo +[@cskiraly]: https://github.com/cskiraly + +See the [lifecycle document][lifecycle-spec] for context about the maturity level +and spec status. + +[lifecycle-spec]: https://github.com/libp2p/specs/blob/master/00-framework-01-spec-lifecycle.md + +## Overview + +Partial Messages Extensions allow users to transmit only a small part of a +message rather than a full message. This is especially useful in cases where +there is a large messages and a peer is missing only a small part of the +message. + +## Terms and Definitions + +**Full Message**: A Gossipsub Message. + +**Message Part**: The smallest verifiable part of a message. + +**Partial Message**: A group of one or more message parts. + +**Group ID**: An identifier to some Full Message. This must not depend on +knowing the full message, so it can not simply be a hash of the full message. + +## Motivation + +The main motivation for this extension is optimizing Ethereum's Data +Availability (DA) protocol. In Ethereum's upcoming fork, Fusaka, custodied data +is laid out in a matrix per block, where the rows represent user data (called +blobs), and the columns represent a slice across all blobs included in the block +(each blob slice in the column is called a cell). These columns are propagated +with Gossipsub. At the time of writing it is common for a node to already have +all the blobs from its mempool, but in cases where it doesn't (~38%[1]) have +_all_ of the blobs it almost always has _most_ of the blobs (today, it almost +always has all but one [1]). More details of how this integrates with Ethereum +can be found at the [consensus-specs +repo](https://github.com/ethereum/consensus-specs/pull/4558) + +This extension would allow nodes to only request the column message part +belonging to the missing blob. Reducing the network resource usage +significantly. As an example, if there are 32 blob cells in a column and the +node has all but one cell, this would result in a transfer of 2KiB rather than +64KiB per column. and since nodes custody at least 8 columns, the total savings +per slot is around 500KiB. + +Later, partial messages could enable further optimizations: +- If cells can be validated individually, as in the case of DAS, partial + messages could also be forwarded, allowing us to reduce the store-and-forward + delay [2]. +- Finally, in the FullDAS construct, where both row and column topics are + defined, partial messages allow cross-forwarding cells between these topics + [2]. + +## Advantage of Partial Messages over smaller Gossipsub Messages + +Partial Messages within a group imply some structure and correlation. Thus, +multiple partial messages can be referenced succinctly. For example, parts can +be referenced by bitmaps, ranges, or a bloom filter. + +The structure of partial messages in a group, as well as how partial messages +are referenced is application defined. + +If, in some application, a group only ever contained a single partial message, +then partial messages would be the same as smaller messages. + + +## Protocol Messages + +The following section specifies the semantics of each new protocol message. + +### PartialIWANT + +A `PartialIWANT` signal to a receiver that the sending peer only wants a part of +some message. + +The message to which the peer is requesting a part of is identified by the +`groupID` identifier. This is similar to a complete message's `MessageID`, but, +in contrast to a content-based message id, does not require the full message to +compute. For example, in the Ethereum use case, this could simply be the hash of +the signed block header. + +The `topicID` references the Gossipsub topic a message, and thus its parts, +belong to. + +The `metadata` field is opaque application defined metadata associated with this +request. This can be a bitmap, a list of ranges, or a bloom filter. The +application generates this and consumes this. + +A later `PartialIWANT` serve to refine the request of prior a prior `PartialIWANT`. + +Nodes SHOULD assume a `PartialIWANT` implies a `IDONTWANT` for the full message. + +### PartialIDONTWANT + +PartialIDONTWANT serves to cancel any and all pending `PartialIWANTs`. + +Implementations SHOULD NOT send a `PartialIHAVE` to a peer with parts that the +peer has previously signaled disinterest for with a `PartialIDONTWANT`. + +### PartialIHAVE + +A `PartialIHAVE` allows nodes to signal HAVE information before receiving all +segments, unlocking the use of `PartialIWANT` in more contexts. + +In the context of partial messages, it is more useful than IHAVE as it includes +the group ID. In contrast, an IHAVE does not. A receiving peer has no way to +link an IHAVE's message ID with a group ID, without having the full message. + +A `PartialIHAVE` message can be used both in the context of lazy push, notifying +peers about available parts, and in the context of heartbeats as a replacement +to IHAVEs. + +The structure of `PartialIHAVE` is analogous to that of `PartialIWANT`. + +The metadata, as in a `PartialIWANT`, is application defined. It is some encoding +that represents the parts the sender has. + +Implementations are free to select when to send an update to their peers based +on signaling bandwidth tradeoff considerations. + +## Application Interface + +Message contents are application defined. Thus splitting a message must be +application defined. Applications should provide a Partial Message type that +supports the following operations: + +1. `.GroupID() -> GroupID: bytes` +2. `.PartialMessageBytesFromMetadata(metadata: bytes) -> Result<(EncodedPartialMessage: bytes, metadata: bytes), Error>` (When responding to a `PartialIWANT` or eagerly pushing a partial message) + a. The returned metadata represents the still missing parts. For example, if a + peer is only able to fulfill a part of the the request, the returned + metadata represents the parts it couldn't fulfill. +3. `.ExtendFromEncodedPartialMessage(data: bytes) -> Result<(), Error>` (When receiving a `PartialMessage`) +4. `.MissingParts() -> Result` (For `PartialIWANT`) +5. `.AvailableParts() -> Result` (For `PartialIHAVE`) + +Gossipsub in turn provides a `.PublishPartial(PartialMessage)` method. + +Note that this specific interface is not intended to be normative to +implementations, rather, it is high level summary of what each layer should +provide. + +## Protobuf + +```protobuf +syntax = "proto2"; + +message PartialMessagesExtension { + optional bytes topicID = 1; + optional bytes groupID = 2; + + optional PartialMessage message = 3; + optional PartialIWANT iwant = 4; + optional PartialIDONTWANT idontwant = 5; + optional PartialIHAVE ihave = 6; +} + +message PartialMessage { + optional bytes data = 1; +} + +message PartialIWANT { + optional bytes metadata = 1; +} + +message PartialIDONTWANT {} + +message PartialIHAVE { + optional bytes metadata = 1; +} + +``` + +## Open Questions + +- Do we want to add a TTL to PartialIWANTs? This would allow us to cancel them after some time. +- Should we rename the metadata bytes to iwant and ihave? +- In the bitmap usecase, iwant/ihave are simply inverses of each other. Do we need to send them both? +- There's a bit of extra complexity around assuming opaque metadata, is it worth it? + +[1]: https://ethresear.ch/t/is-data-available-in-the-el-mempool/22329 +[2]: https://ethresear.ch/t/fulldas-towards-massive-scalability-with-32mb-blocks-and-beyond/19529#possible-extensions-13 From 50f9c1111895340c7d39ac164e85202939ab9963 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Tue, 23 Sep 2025 13:24:54 -0700 Subject: [PATCH 2/9] Remove PartialIDONTWANT It's purpose is subsumed by PartialIHAVE --- pubsub/gossipsub/partial-messages.md | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index ad0331348..9ecba8bf5 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -102,13 +102,6 @@ A later `PartialIWANT` serve to refine the request of prior a prior `PartialIWAN Nodes SHOULD assume a `PartialIWANT` implies a `IDONTWANT` for the full message. -### PartialIDONTWANT - -PartialIDONTWANT serves to cancel any and all pending `PartialIWANTs`. - -Implementations SHOULD NOT send a `PartialIHAVE` to a peer with parts that the -peer has previously signaled disinterest for with a `PartialIDONTWANT`. - ### PartialIHAVE A `PartialIHAVE` allows nodes to signal HAVE information before receiving all @@ -130,6 +123,9 @@ that represents the parts the sender has. Implementations are free to select when to send an update to their peers based on signaling bandwidth tradeoff considerations. +Receivers MUST treat a `PartialIHAVE` as a signal that the peer does not want +the indicated part. + ## Application Interface Message contents are application defined. Thus splitting a message must be @@ -162,8 +158,7 @@ message PartialMessagesExtension { optional PartialMessage message = 3; optional PartialIWANT iwant = 4; - optional PartialIDONTWANT idontwant = 5; - optional PartialIHAVE ihave = 6; + optional PartialIHAVE ihave = 5; } message PartialMessage { @@ -174,8 +169,6 @@ message PartialIWANT { optional bytes metadata = 1; } -message PartialIDONTWANT {} - message PartialIHAVE { optional bytes metadata = 1; } From 3f134d20d8121372c92a82a9ce73837e7a85a6b3 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Thu, 25 Sep 2025 11:38:20 -0700 Subject: [PATCH 3/9] gossipsub: add explicit request to use partial messages for a given topic --- pubsub/gossipsub/extensions/extensions.proto | 26 ++++++++++++ pubsub/gossipsub/partial-messages.md | 42 ++++++++------------ 2 files changed, 43 insertions(+), 25 deletions(-) diff --git a/pubsub/gossipsub/extensions/extensions.proto b/pubsub/gossipsub/extensions/extensions.proto index 402186bee..d1ee238e2 100644 --- a/pubsub/gossipsub/extensions/extensions.proto +++ b/pubsub/gossipsub/extensions/extensions.proto @@ -25,6 +25,11 @@ message RPC { message SubOpts { optional bool subscribe = 1; // subscribe or unsubcribe optional string topicid = 2; + + // Used with Partial Messages extension. + // If set, the receiver of this message MUST send partial messages to the + // sender instead of full messages. + optional bool partial = 3; } repeated SubOpts subscriptions = 1; @@ -41,3 +46,24 @@ message RPC { optional PartialMessagesExtension partial = 16418754; } + +message PartialMessagesExtension { + optional bytes topicID = 1; + optional bytes groupID = 2; + + optional PartialMessage message = 3; + optional PartialIWANT iwant = 4; + optional PartialIHAVE ihave = 5; +} + +message PartialMessage { + optional bytes data = 1; +} + +message PartialIWANT { + optional bytes metadata = 1; +} + +message PartialIHAVE { + optional bytes metadata = 1; +} diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index 9ecba8bf5..cc2719a8d 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -126,6 +126,22 @@ on signaling bandwidth tradeoff considerations. Receivers MUST treat a `PartialIHAVE` as a signal that the peer does not want the indicated part. +### Changes to `SubOpts` and interaction with the existing Gossipsub mesh. + +Partial Messages uses the same mesh as normal Gossipsub messages. It is a +replacement to "full" messages. A node requests a peer to use partial messages +for a specific topic by setting the `partial` field in the `SubOpts` message. +The `SubOpts` message is how a peer subscribes to a topic. + +If a node receives a subscribe request with the `partial` field set to true, it +MUST send partial messages instead of full messages. + +It is an error to set the partial field true if the peer does not support +partial extensions. + +The partial field value MUST be ignored when a peer sends an unsubscribe message +`SubOpts.subscribe=false`. + ## Application Interface Message contents are application defined. Thus splitting a message must be @@ -149,31 +165,7 @@ provide. ## Protobuf -```protobuf -syntax = "proto2"; - -message PartialMessagesExtension { - optional bytes topicID = 1; - optional bytes groupID = 2; - - optional PartialMessage message = 3; - optional PartialIWANT iwant = 4; - optional PartialIHAVE ihave = 5; -} - -message PartialMessage { - optional bytes data = 1; -} - -message PartialIWANT { - optional bytes metadata = 1; -} - -message PartialIHAVE { - optional bytes metadata = 1; -} - -``` +Refer to the protobuf registry at `./extensions/extensions.proto` ## Open Questions From 8e42e179fdcc62db4940a9c6483e8feef4255dce Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 26 Sep 2025 14:29:38 -0700 Subject: [PATCH 4/9] Simplify the fields and remove message nesting. There is no longer a separation between iwant/ihave --- pubsub/gossipsub/extensions/extensions.proto | 18 ++--- pubsub/gossipsub/partial-messages.md | 71 +++++++------------- 2 files changed, 27 insertions(+), 62 deletions(-) diff --git a/pubsub/gossipsub/extensions/extensions.proto b/pubsub/gossipsub/extensions/extensions.proto index d1ee238e2..651b7bf01 100644 --- a/pubsub/gossipsub/extensions/extensions.proto +++ b/pubsub/gossipsub/extensions/extensions.proto @@ -51,19 +51,9 @@ message PartialMessagesExtension { optional bytes topicID = 1; optional bytes groupID = 2; - optional PartialMessage message = 3; - optional PartialIWANT iwant = 4; - optional PartialIHAVE ihave = 5; -} - -message PartialMessage { - optional bytes data = 1; -} - -message PartialIWANT { - optional bytes metadata = 1; -} + // An encoded partial message + optional bytes partialMessage = 3; -message PartialIHAVE { - optional bytes metadata = 1; + // An encoded representation of the parts a peer has and wants. + optional bytes partsMetadata = 4; } diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index cc2719a8d..5b3593796 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -80,52 +80,28 @@ then partial messages would be the same as smaller messages. The following section specifies the semantics of each new protocol message. -### PartialIWANT +### partialMessage -A `PartialIWANT` signal to a receiver that the sending peer only wants a part of -some message. +The `partialMessage` field encodes one or more parts of the full message. The +encoding is application defined. -The message to which the peer is requesting a part of is identified by the -`groupID` identifier. This is similar to a complete message's `MessageID`, but, -in contrast to a content-based message id, does not require the full message to -compute. For example, in the Ethereum use case, this could simply be the hash of -the signed block header. +### partsMetadata -The `topicID` references the Gossipsub topic a message, and thus its parts, -belong to. +The `partsMetadata` field encodes the parts a peer has and wants. The encoding +is application defined. An unset value carries no information besides that the +peer did not send a value. -The `metadata` field is opaque application defined metadata associated with this -request. This can be a bitmap, a list of ranges, or a bloom filter. The -application generates this and consumes this. +Upon receiving a `partsMetadata` a node SHOULD respond with only parts the peer +wants. -A later `PartialIWANT` serve to refine the request of prior a prior `PartialIWANT`. +A later `partsMetadata` replaces a prior one. -Nodes SHOULD assume a `PartialIWANT` implies a `IDONTWANT` for the full message. - -### PartialIHAVE - -A `PartialIHAVE` allows nodes to signal HAVE information before receiving all -segments, unlocking the use of `PartialIWANT` in more contexts. - -In the context of partial messages, it is more useful than IHAVE as it includes -the group ID. In contrast, an IHAVE does not. A receiving peer has no way to -link an IHAVE's message ID with a group ID, without having the full message. - -A `PartialIHAVE` message can be used both in the context of lazy push, notifying -peers about available parts, and in the context of heartbeats as a replacement -to IHAVEs. - -The structure of `PartialIHAVE` is analogous to that of `PartialIWANT`. - -The metadata, as in a `PartialIWANT`, is application defined. It is some encoding -that represents the parts the sender has. +`partsMetadata` can be used during heartbeat gossip to inform non-mesh topic +peers about parts this node has. Implementations are free to select when to send an update to their peers based on signaling bandwidth tradeoff considerations. -Receivers MUST treat a `PartialIHAVE` as a signal that the peer does not want -the indicated part. - ### Changes to `SubOpts` and interaction with the existing Gossipsub mesh. Partial Messages uses the same mesh as normal Gossipsub messages. It is a @@ -144,25 +120,24 @@ The partial field value MUST be ignored when a peer sends an unsubscribe message ## Application Interface -Message contents are application defined. Thus splitting a message must be +This specific interface is not intended to be normative to implementations, it +is only an example of one possible API. + +Message contents are application defined, thus splitting a message must be application defined. Applications should provide a Partial Message type that supports the following operations: 1. `.GroupID() -> GroupID: bytes` -2. `.PartialMessageBytesFromMetadata(metadata: bytes) -> Result<(EncodedPartialMessage: bytes, metadata: bytes), Error>` (When responding to a `PartialIWANT` or eagerly pushing a partial message) - a. The returned metadata represents the still missing parts. For example, if a - peer is only able to fulfill a part of the the request, the returned - metadata represents the parts it couldn't fulfill. -3. `.ExtendFromEncodedPartialMessage(data: bytes) -> Result<(), Error>` (When receiving a `PartialMessage`) -4. `.MissingParts() -> Result` (For `PartialIWANT`) -5. `.AvailableParts() -> Result` (For `PartialIHAVE`) +2. `.PartialMessageBytes(partsMetadata: bytes) -> Result<(EncodedPartialMessage: bytes, newPartsMetadata: bytes), Error>` + a. The method should return an encoded partial message with just the parts the + peer requested. + b. The returned `newPartsMetadata` can be used to track parts that could not + be fulfilled. This allows the GossipSub library to avoid sending duplicate + parts to the same peer. +3. `.PartsMetadata() -> bytes` Gossipsub in turn provides a `.PublishPartial(PartialMessage)` method. -Note that this specific interface is not intended to be normative to -implementations, rather, it is high level summary of what each layer should -provide. - ## Protobuf Refer to the protobuf registry at `./extensions/extensions.proto` From a4cb3c2178a9474cd211068191c7f530be0ebcce Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 10 Oct 2025 10:56:26 -0700 Subject: [PATCH 5/9] relax the requirement on the SubOpts partial field. --- pubsub/gossipsub/partial-messages.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index 5b3593796..0d2717924 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -109,11 +109,12 @@ replacement to "full" messages. A node requests a peer to use partial messages for a specific topic by setting the `partial` field in the `SubOpts` message. The `SubOpts` message is how a peer subscribes to a topic. -If a node receives a subscribe request with the `partial` field set to true, it -MUST send partial messages instead of full messages. +If a node receives a subscribe request with the `partial` field set to true, and +it supports the partial message extension, it MUST send partial messages instead +of full messages. -It is an error to set the partial field true if the peer does not support -partial extensions. +If a node does not support the partial message extension, it MUST ignore the +`partial` field. This is the default behavior of protobuf parsers. The partial field value MUST be ignored when a peer sends an unsubscribe message `SubOpts.subscribe=false`. From 9177ea3e17ab39c23422bd3d3a6ddf96cba62125 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 10 Oct 2025 10:57:00 -0700 Subject: [PATCH 6/9] Update code Gossipsub partial extension codepoint --- pubsub/gossipsub/extensions/extensions.proto | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/pubsub/gossipsub/extensions/extensions.proto b/pubsub/gossipsub/extensions/extensions.proto index 651b7bf01..a01571a0f 100644 --- a/pubsub/gossipsub/extensions/extensions.proto +++ b/pubsub/gossipsub/extensions/extensions.proto @@ -1,15 +1,13 @@ syntax = "proto2"; message ControlExtensions { - // Initially empty. Future canonical extensions will be added here along with - // a reference to their specification. + optional bool partialMessages = 10; // Experimental extensions must use field numbers larger than 0x200000 to be // encoded with at least 4 bytes optional bool testExtension = 6492434; - optional bool partialMessages = 16418754; } message ControlMessage { @@ -37,6 +35,7 @@ message RPC { optional ControlMessage control = 3; // Canonical Extensions should register their messages here. + optional PartialMessagesExtension partial = 10; // Experimental Extensions should register their messages here. They // must use field numbers larger than 0x200000 to be encoded with at least 4 @@ -44,7 +43,6 @@ message RPC { optional TestExtension testExtension = 6492434; - optional PartialMessagesExtension partial = 16418754; } message PartialMessagesExtension { From d847d6c0a72d50725748646f979fb14c9abc6823 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 10 Oct 2025 10:57:00 -0700 Subject: [PATCH 7/9] mark off todos --- pubsub/gossipsub/partial-messages.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index 0d2717924..f20517b11 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -143,12 +143,5 @@ Gossipsub in turn provides a `.PublishPartial(PartialMessage)` method. Refer to the protobuf registry at `./extensions/extensions.proto` -## Open Questions - -- Do we want to add a TTL to PartialIWANTs? This would allow us to cancel them after some time. -- Should we rename the metadata bytes to iwant and ihave? -- In the bitmap usecase, iwant/ihave are simply inverses of each other. Do we need to send them both? -- There's a bit of extra complexity around assuming opaque metadata, is it worth it? - [1]: https://ethresear.ch/t/is-data-available-in-the-el-mempool/22329 [2]: https://ethresear.ch/t/fulldas-towards-massive-scalability-with-32mb-blocks-and-beyond/19529#possible-extensions-13 From 442f51a89de2e8486809175652b92f1244505743 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Thu, 23 Oct 2025 13:58:18 -0700 Subject: [PATCH 8/9] minor formatting --- pubsub/gossipsub/partial-messages.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index f20517b11..aa6383add 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -56,6 +56,7 @@ node has all but one cell, this would result in a transfer of 2KiB rather than per slot is around 500KiB. Later, partial messages could enable further optimizations: + - If cells can be validated individually, as in the case of DAS, partial messages could also be forwarded, allowing us to reduce the store-and-forward delay [2]. @@ -75,7 +76,6 @@ are referenced is application defined. If, in some application, a group only ever contained a single partial message, then partial messages would be the same as smaller messages. - ## Protocol Messages The following section specifies the semantics of each new protocol message. @@ -130,11 +130,11 @@ supports the following operations: 1. `.GroupID() -> GroupID: bytes` 2. `.PartialMessageBytes(partsMetadata: bytes) -> Result<(EncodedPartialMessage: bytes, newPartsMetadata: bytes), Error>` - a. The method should return an encoded partial message with just the parts the - peer requested. - b. The returned `newPartsMetadata` can be used to track parts that could not - be fulfilled. This allows the GossipSub library to avoid sending duplicate - parts to the same peer. + 1. The method should return an encoded partial message with just the parts the + peer requested. + 2. The returned `newPartsMetadata` can be used to track parts that could not + be fulfilled. This allows the GossipSub library to avoid sending duplicate + parts to the same peer. 3. `.PartsMetadata() -> bytes` Gossipsub in turn provides a `.PublishPartial(PartialMessage)` method. From ae6b268df3c2fe7e16f2f2d6c3fa154d441e3263 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Thu, 23 Oct 2025 13:58:51 -0700 Subject: [PATCH 9/9] Add supportsPartial field to subopts Let's a peer signal support for partial messages on a per-topic basis. This makes future rollouts of partial messages on new topics much easier as nodes can explicitly signal their support of partial messages per topic. --- pubsub/gossipsub/extensions/extensions.proto | 10 ++-- pubsub/gossipsub/partial-messages.md | 51 ++++++++++++++++---- 2 files changed, 49 insertions(+), 12 deletions(-) diff --git a/pubsub/gossipsub/extensions/extensions.proto b/pubsub/gossipsub/extensions/extensions.proto index a01571a0f..1fbfc5a4c 100644 --- a/pubsub/gossipsub/extensions/extensions.proto +++ b/pubsub/gossipsub/extensions/extensions.proto @@ -25,9 +25,13 @@ message RPC { optional string topicid = 2; // Used with Partial Messages extension. - // If set, the receiver of this message MUST send partial messages to the - // sender instead of full messages. - optional bool partial = 3; + // If set to true, signals to the receiver that the sender prefers partial + // messages. + optional bool requestsPartial = 3; + // If set to true, signals to the receiver that the sender supports sending + // partial messages on this topic. If requestsPartial is true, this is + // assumed to be true. + optional bool supportsPartial = 4; } repeated SubOpts subscriptions = 1; diff --git a/pubsub/gossipsub/partial-messages.md b/pubsub/gossipsub/partial-messages.md index aa6383add..b89336552 100644 --- a/pubsub/gossipsub/partial-messages.md +++ b/pubsub/gossipsub/partial-messages.md @@ -104,20 +104,53 @@ on signaling bandwidth tradeoff considerations. ### Changes to `SubOpts` and interaction with the existing Gossipsub mesh. -Partial Messages uses the same mesh as normal Gossipsub messages. It is a -replacement to "full" messages. A node requests a peer to use partial messages -for a specific topic by setting the `partial` field in the `SubOpts` message. The `SubOpts` message is how a peer subscribes to a topic. -If a node receives a subscribe request with the `partial` field set to true, and -it supports the partial message extension, it MUST send partial messages instead -of full messages. +Partial Messages uses the same mesh as normal Gossipsub messages. It is a +replacement to "full" messages. A node requests a peer to send partial messages +for a specific topic by setting the `requestsPartial` field in the `SubOpts` +message to true. A node signals support for sending partial messages on a given +topic by setting the `supportsPartial` field in `SubOpts` to true. A node can +support sending partial messages without wanting to receive them. + +If a node requests partial messages, it MUST support partial messages. + +A node uses a peer's `supportsPartial` setting to know if it can send a partial +message request to a peer. It uses its `requestsPartial` setting to know whether +to send send the peer a full message or a partial message. + +If a peer supports partial messages on a topic but did not request them, a node +MUST omit the `partialMessage` field of the `PartialMessagesExtension` message. If a node does not support the partial message extension, it MUST ignore the -`partial` field. This is the default behavior of protobuf parsers. +`requestPartial` and `supportsPartial` fields. This is the default behavior of +protobuf parsers. + +The `requestPartial` and `supportsPartial` fields value MUST be ignored when a +peer sends an unsubscribe message `SubOpts.subscribe=false`. + +#### Behavior table + +The following table describes the expected behavior of receiver of a `SubOpts` +message for a given topic. + +| SubOpts.requestsPartial | behavior of receiver that supports partial messages for the topic | +| ------------------------ | ------------------------------------------------------------------------------------------------- | +| true | The receiver SHOULD send partial messages (data and metadata) to this peer. | +| false | receiver MUST NOT send partial message data to this peer. The receiver SHOULD send full messages. | + +| SubOpts.requestsPartial | behavior of receiver that does not support partial messages for the topic | +| ------------------------ | ------------------------------------------------------------------------- | +| \* | The receiver SHOULD send full messages. | + +| SubOpts.supportsPartial | behavior of receiver that requested partial messages for the topic | +| ------------------------ | ---------------------------------------------------------------------------------------------------------------- | +| true | The receiver expects the peer to respond to partial message requests, and receive `partsMetadata` from the peer. | +| false | The receiver expects full messages. | -The partial field value MUST be ignored when a peer sends an unsubscribe message -`SubOpts.subscribe=false`. +| SubOpts.supportsPartial | behavior of receiver that did not request partial messages for the topic | +| ------------------------ | ------------------------------------------------------------------------ | +| \* | The receiver expects full messages | ## Application Interface