When a large number of (small-moderate size) messages are sent through gossipsub, the behavior is significantly influenced by the inter-packet spacing (delay). A smaller inter-packet spacing leads to a higher number of messages in the outgoing message queues, which in turn increases latency. For instance, the average message dissemination time can jump from 450ms to 2600 ms if the inter-packet delay is reduced from 350ms to 50 ms, as shown below.
Interestingly, even for 50 ms inter-packet spacing, almost all messages were received by more than 90% of peers within the first 600 ms. The unexpected delay incurred by the last 10% of peers requires further investigation. A few possible reasons may include:
- In case of a more considerable heartbeat interval (700 ms for these tests), it takes quite some time for the IHAVEs to reach all the peers.
- IWANT replies are placed in the low-priority queues, or IWANTs are not replied at all.
- Message starvation (in low-priority queues) at mesh peers