Discussion topic for research post: Libp2p GossipSub IDONTWANT Message Performance Impact
From looking at our store nodes, Status’ average message size is ~50kB.
What is the bandwidth gain for this message size?
@prem @rramos @Ivansete what is the heartbeat interval for go-waku in Status and nwaku service nodes?
We are using a heartbeat interval of 1 sec on go-waku.
One more thing i have noticed is the simulations have been done by disabling flood-publish. In waku we enable flood publish, not sure how that effects the test results. Would be good to know the results of when flood-publish is enabled as well.
Note that these results are done with pure nim-libp2p, so there are a few differences with the DST results mentioned also at the end of the research post, like:
- Implementations (nimlibp2p vs gowaku+nwaku)
- Number of messages (10 vs 180)
- Gossipsub parameters, I think waku’s d_low is lower. Also as Prem mention, floodpublish, and so on…
In order to do a 1 on 1 comparison with the DST results, there are a lot of parameters that should be adjusted.
1 Like
I think small variance in heartbeat interval may not have much impact for moderate (around 50KB) message sizes.
We have noticed slightly negative impact on latency with floodpublish enabled for large messages (around 500KB). It can be a good idea to investigate the impact on bandwidth.
1 Like
We have seen different results for 50KB messages. The experiments in this post revealed around 4% gain. The experiments on Kubernetes reported around 20% gain.
There are multiple reasons for this variation:
- The environments are different as highlighted by @Alberto
- Also, network conditions are slightly different in terms of latency/bandwidth
- There could be some difference in peering algorithm/GossipSub related parameters
The difference in results becomes more noticeable for smaller messages. That is why not much difference is seen for bigger messages.
I guess we can run more experiments (in shadow simulator for nim-libp2p) for the exact network conditions. @fryorcraken @prem Can you please share precise parameters like average latency/deviations, network/message size, and other related parameters.
Most of the default values configured for gossiopsub in Waku are available go-waku/waku/v2/protocol/relay/config.go at master · waku-org/go-waku · GitHub .
Similar config can be found for nwaku as well nwaku/waku/waku_relay/protocol.nim at master · waku-org/nwaku · GitHub
Let me know if you are looking for any specific params apart from these.
1 Like
Hello @prem ,
Sorry for the late reply, I was pushed by a pressing task. Yes, I can figure out some possible settings for these locations.
But, knowing average peer bandwidth, link latencies (estimated average will work), network size (usual range), number of messages/second can be helpful!
I guess, usual message size is 50KB?
No problem.
This would be a challenge because we don’t have these metrics in case of status.
Few things to note that is unique to status’s use case
- network size keeps changing as status desktop apps(which act as gossipsub nodes) keep coming and going through the day which affects the network size constantly. We may have to run some utility to crawl the network and constantly ping peers to get an average number.
- Since it is a chat application number of msgs per second is hard to determine as it would depend on usage of the app by users.
- Similarly peer bandwidth is something that we don’t know of. We have some rough numbers wrt status desktop node bandwidth usage based on some tests done. Again do note that usage is varying over time of day.
I think this is the average msgSize, as i had mentioned above since it is a chat application the msg sizescan vary between few bytes(something like an emoji reaction) to kilobytes(control msgs or media).
1 Like