Waku v2 scalability studies

Waku
2

Thanks for that @petty. I was actually thinking about writing something about js-waku scalability as I have been explaining the same thing to few people. This made me realized I need to be more transparent on the subject of what is going on at the moment.

Allow me to hijack your discussion and provide some information on js-waku’s scalability status and goals.

Scalability Metrics for the Browser

Browser Waku node can be qualified as end points nodes:

  • A browser node needs to connect to a service node (nwaku, go-waku, NodeJS js-waku).
  • While it is possible to establish browser to browser connections, it is only possible to do so with an existing connection to a service node (signalling server or waku network).
  • Only secure websocket is currently supported in js-waku, meaning that the remote service node needs to provide a SSL connection, which includes a certificate that can be accepted by the user’s browser. Something unlikely to happen for Waku nodes running on the desktop or on mobile.

Hence, the first scarce metric for browser nodes are the number of connections available:

  1. Browser nodes consume connections from Waku service nodes.
  2. Browser nodes do not provide connections to other Waku nodes.

(2) is currently true but could be corrected in the future as one browser node could established connections to several browser using a single connection to a Waku service node. However it is unclear how reliable and practical this would be.

In terms of nwaku (and go-waku)'s connection capabilities:

  • nwaku nodes run by Status are set up to accept a maximum of 150 connections.
  • We do not know how many connections could a nwaku/go-waku node accept given specific hardware parameters (e.g. raspberry pi, X mem Y CPU).

Some assumptions

  • The recommended mesh size for Waku Relay is 6, meaning that ideally, a Waku Relay node should have at least 6 connections to other Waku Relay nodes to form a healthy gossipsub mesh.
  • For nodes that do not use Relay but instead LightPush and Filter[1], we haven’t set recommendations on the number of LightPush/Filter nodes one should connect to. Most likely it should be 2 or higher to ensure reliability and not rely on a single remote node.
  • For Store, likely to be the same as LightPush/Filter.

The current state of art

Discovery

  • js-waku’s default bootstrap method connects to a single remote peer from a set of nodes hosted by Status.
  • js-waku’s bootstrap API enables developers to pass a function, list of peers or ENR tree to retrieve nodes for bootstrapping.
  • An ENR tree is used for DNS Discovery as defined in EIP-1459. It enables the developer to set a domain name in their dApp and encode a list of nodes in a given domain name which can later be edited, extended without changing the deployed code.
  • Discv5 is being implemented in nwaku to enable a node to find other nodes outside the DNS Discovery protocol: Ambient Discovery Roadmap (incl Disc v5) · Issue #879 · status-im/nwaku · GitHub
  • Discv5 is not adapted to a browser context as it takes time and has high connection churn.

Note that DNS Discovery is not currently used by default in js-waku and we haven’t yet provided guides or tutorials for it because of one blocking item: native websocket support in nwaku.

Native Websocket support in nwaku

  • Currently, only secure websocket transport is supported by js-waku in the browser (future work includes WebRTC and WebTransport)
  • Currently, js-waku browser nodes connect to Status nodes via websockify. Websockify wraps nwaku’s TCP connection with TLS + websocket (ie, secure websocket).
  • Native secure websocket support was recently introduced in nwaku, thanks to nim-websocket.
  • To build an ENR tree, one need the ENR of each node. This is provided by nwaku via JSON RPC or in the logs. This does not work if nwaku is not aware of the port and domain name to use in the ENR, ie, if an external software is used, such as websockify, to provide secure websocket transport.

Limitations

The current limitations are as follows:

  • Secure websocket is not yet stable in nwaku.
  • It is not easy to setup an enrtree that includes wss when running nwaku nodes with websockify.
  • js-waku only connects to 1 remote node because of the limited number of nodes run by Status and the limited number of connections provided by these nodes.
  • Beyond bootstrapping, there are no peer discovery method available in js-waku.

Another way to see it is that today, if one wants to deploy a service node that would be use by browser nodes they:

  1. Need to deploy nwaku
  2. Need to get a domain name, point it to nwaku
  3. Get a valid SSL cert (letsencrypt works)
  4. Possibly deploy websockify due to low reliability of nim-websocket
  5. Register their node somewhere so that js-waku discovers it:
    i. Either hard code the node details in the app’s code base (if they actually are running an app and not just being an operator)
    ii. Or include in a domain name used for DNS Discovery

What we are doing about it

Short term we are aiming to get rid of (4) and (5).

Websocket

A lot of effort has been made by the nwaku and nim-websocket maintainer to stabilize the code. We expect drastic improvement with the next nwaku release (v0.10, ETA this month, June '22).
Some of the issues tracking are:

DNS Discovery

The websocket stabilization will enable js-waku to offer DNS Discovery as a default bootstrap method: Use DNS Discovery by default · Issue #517 · status-im/js-waku · GitHub. With this we will also increase the number of Status’ operated boostrap nodes used by js-waku (waku connect + Vac Prod) , allowing an increase of the default number of connections for js-waku.

We will also publish guides on how to setup your own DNS for Discovery, enabling platforms to deploy their own nodes. This is currently possible but cumbersome.
Note DNS used for EIP-1459 can point to each other. For example, a project can run their own nodes and also point to another’s project or Status’ DNS so that a js-waku node can access a greater number of bootstrap nodes.

Connection benchmark

Once nwaku’s websocket implementation is table, we can start running benchmarks to know how many wss connection can a single nwaku node support. Enabling the browser:nwaku node ratio to increase.

Other Discovery

We are designing a peer exchange protocol: New RFC: 34/WAKU2-PEER-EXCHANGE · Issue #495 · vacp2p/rfc · GitHub which would enable browser nodes to leverage discv5. The idea would be for a browser node to do a get peers on a remote waku node, said remote node would then return the peers they found during their last discv5 run.

This would enable a browser to connect to nodes beyond the bootstraps one, increasing the potential pool of nodes and hence available connections.

Conclusion

We made Waku adaptive so that it can run anywhere, from cloud to mobile to browser. Any benchmark or scalability consideration needs to consider:

  • Which protocols are we measuring
  • On which platforms are we measuring these protocols.
  • What topology are we targeting, does this topology matches the Waku Network in practice or just a subset of it?

Contrary to other p2p networks, the Waku Network includes nodes of various resources, including the browser, which makes can make the topology more “heterogeneous” then other p2p network such as Ethereum (1) or Bitcoin.

[1]: More work is needed on filter, currently only work js-waku<>go-waku (when js-waku in the browser) and it is also not (yet) possible to disable Waku Relay in js-waku.

3 Likes