RFC: Topic propagation extension to libp2p PubSub

Vac
1

Introduction

This is a proposal for an extension to libp2p PubSub that’d make it more useful for short-lived topics and resource restricted devices. It would be used in WakuSub, which is an experimental libp2p protocol for running Waku on libp2p that builds on existing libp2p FloodSub/GossipSub. The primary end goal is to (a) move Waku to (nim-)libp2p and (b) provide more scalable routing.

Background

With Waku, all full nodes listen to all topics and route them. Only light nodes express specific topic interest to a full node, in order to preserve bandwidth. This obviously doesn’t scale to large volume for full nodes, since all full nodes have to process all messages in the network. Some partitioning is required.

With libp2p PubSub/FloodSub/GossipSub, scalability looks better. One property this family of protocols have is that nodes must know which topics to subscribe to. To get a message from A to B over a topic via other nodes, there must be a path along which all nodes subscribe to that specific topic. The mesh only deals with content deduplication. Topic discovery and propagation are external mechanisms that happen out of band.

Problem

The way topics work in libp2p PubSub is problematic for light nodes with ephemeral topics. You might have two light nodes who care about a very specific topic, but intermediate, well-connected nodes are ignorant of these topics.

A realistic scenario here are two mobile phones A and B that are only connected to one other intermediate node, and these intermediate nodes are connected to each other. By default, there is no way for those intermediate nodes to know about the topic and to relay the message from A to B. See topology below, where “foobar” indicates that the topic is subscribed to be that node.
)
image

A possible solution

One way of solving this is to propagate topic subscriptions as well. Essentially an intermediate node subscribes to a topic on behalf of a light node.

When a light node subscribes to a topic, it lets its nodes know. This means its intermediate node(s) record subscribe-topic-for-peer. However, an intermediate node isn’t necessarily subscribed to that topic itself. We can use the subscribe-topic-for-peer to trigger a subscription, thus subscribing to a topic on behalf of a light node. This is useful in the case where intermediate nodes are more well-connected and have higher availability.

If this subscribe-on-behalf-of-node behavior is triggered for every node, we quickly end up in a cascading cycle. As a worst case scenario all nodes are listening to all topics they heard about. Then we are back to where we started. This situation can be avoided by only allowing it for specific parameters.

Parameters and control for topic propagation

What can those parameters be, and how are they instrumented? We have a few options here. Either we can do it only for light node, or nodes which indicate this capability. This relates to the idea of capability advertising and adaptive nodes. This can be indicated in the multiaddr, as part of RPC Message, SubOpts, or similar. Alternatively, we can make it a standalone control message, e.g. SubscribeOnBehalf.

This ensures that topic propagation is limited based on reasonable intent. However, this assumes honest nodes. It’d still be an improvement on status quo.

In later iterations, we can extend this with a control mechanism. For example, intermediate nodes might only allow som many subscriptions from a node, or so much message volume, or only for so long, etc. This relates to ideas of rate limiting, which we support to some extent, and also ideas such as stake based priority queue (see previous forum thread). It can probably be tied to an incentive mechanism since it is a useful (extra) service provided in the network, and switching costs to another node are low.

Case study - sudden community growth

Hypothetical case study and how it might play out.

Today, if a large community influx happens in some region of the world, e.g. 100k people in Thailand moving to Minds (Twitter alternative) in a single day, what would happen? Likely our cluster would quickly be overwhelmed, and we’d need to make all our machines bigger and/or do some hacky bandaid. There’d be no topic partitioning, and network capacity would be bound by what a single machine could process.

What would happen if we had the above setup? People could organically set up their own nodes, and (assuming discovery works correctly), a new semi-connected subgraph would form. In that subgraph, some nodes would subscribe on behalf of light clients, but they’d only have to take some of the extra load. Beyond that, the mesh in GossipSub should take care of the rest.

Next steps

Please provide thoughts on specific path forward and spot holes in the above :slight_smile:

Links

2 Likes