No store in Messaging API

[fryorcraken]

The biggest lessons for Waku in 2023 was the fact that we gave raw protocols to users, whether they be Status, hackathon developers or other projects.

There was, and still is, a strong will for Waku to build agnostic technology. Not only we do FOSS, but we do in a way that is really Free Software.

This sometimes gets in a way of providing good abstraction and good defaults.

We corrected this in 2024 by owning the peer-to-peer reliability protocol [1] and implementation. It is now integrated in Status applications, which are much more reliable (one missing block, end-to-end reliability [2], is soon to be completed).

The final step to this is defining a deliberate API for users to consume Waku, the Messaging API [3].

I often refer to the image below to describe the target layers of the Waku SDK (taken from Message Reliability and Waku API blog article [4]).

The work for the Messaging API has started [Sasha’s PR].
The Reliability API is next. The end-to-end reliability protocol, Scalable Data Sync (SDS), is near completion. The js-waku team also implemented it and is already thinking how such a Reliability API would look like.

Waku Store Dichotomy

The role of the Waku Store protocol in the Waku stack is controversial in Waku’s history. I actually suggested to renaming to Waku Cache a few years back, but I guess it’s all already crystallised.

The controversy comes from the dichotomy of this protocol:

1. It exists to enable mostly-offline devices to retrieve past and missed messages
2. But it is not a decentralized data storage solution; this is not Waku’s domain problem, it is Codex’s.

A Waku protocol stack without Store would mean frequent re-transmissions, at the cost of bandwidth usage and latency performance. This is already what is done with MVDS [5] in Status. But this has implication on rate limit, and until RLN is properly integrated as a foundation to avoid excessive bandwidth usage, it would be unwise to push for more retransmission mechanisms.

Moreover, Waku, and Waku Store, have been used as solutions for several problem the Status app faced in terms of data durability. It is used to back up user settings, hold 30 days of Community history, and unbounded Community member lists.

The unrestricted use of Waku Store, and the absence of effective rate limits, ultimately led to the very issues that Waku v2 aimed to address, including excessive bandwidth consumption, mobile data drain, and battery depletion.

In terms of Store usage, the question is: for how long should we expect messages to be present in store nodes?

However, most applications cannot answer this question as they lack rate limiting. Once RLN is in place, then the following criteria can be considered to answer the question:

• Amount of data: message rate limit, maximum message size, number of shards, number of users and user pattern (statistical distributions) are needed to evaluate the amount of data an application produces.
• Decentralization: what are reasonable database sizes to enable the desired level of decentralization and performance?

From a user and product point of view:

• What messages need to be persisted?
• How often do we expect users to open the app?
• What is the friction-to-inactivity ratio? If a user hasn’t open the app for a year, it may be fine to expect them to re-setup a login; within a day, it may be expected that all messages get loaded without friction; and in-between, added latency from retrieving data from Codex or BitTorrent.

Waku Store and Messaging API

The Waku Messaging API is here to provide an opinionated way to use the Waku protocols. Opinionated so it can provide fair out-of-the-box reliability, for both relay and edge nodes [6], packaged in a simple API that provides a minimum learning curve for new developers.

Good default parameters are also important, done in a lean manner; not everything needs to be configurable. We can define four level of configurability:

1. Developer must decide the parameter value every time
2. A good default is provided, but the developer can change the value
3. No configuration is possible, the developer is welcome to use lower-level APIs
4. Lower-level APIs will not allow this, the developer can fork.

I believe a critical opinion we must uphold, for the scalability, performance, and reliability of Waku, is that Waku Store is not a decentralized storage solution.

Hence, I strongly suggest we do not include direct access to Waku Store in the Messaging API.
Instead, only have indirect access to Waku Store, as described in the peer-to-peer reliability spec [1]:

1. Periodic store queries: When subscribe-ing to a content topic, period store queries can be kicked off/include this content topic.
2. Connection loss detection: When a connection loss is detected (network, or app shut down), a store query can be triggered, as currently implemented in go-waku api.
3. Message confirmation: when sending a message, a hash store query is done to check if a remote store node received it.

The Reliability API is also expected to provide indirect usage of store, to retrieve messages detected as missing by SDS.

status-go integration

Such an API will be better guide to developers, so they use Waku in an efficient manner. Meaning in a way that does enable censorship-resistant properties; large centralized databases do not. And push them towards more adapted solutions (Codex) for data durability needs.

Th approach will also enable us to easily spot bad smells in Status usage of Waku; not that we are not already aware of them.

One of the steps will be to ensure that the Messaging API, and only this API, is used in status-go. As a way to dogfood it and ensure the cleanest-ever Status/Waku boundary (and even cleaner once Reliability API is used).

status-go uses Waku for many things, and we will have to first focus on ensuring that this API is used by the Chat SDK (one-to-one chat and private groups only). As we improve chat protocols, the limitation of the API will ensure that we go in a sustainable direction.

Some obvious bad smell will appear from that, such as large store queries when setting up a profile. Fixing those would be easy to articulate as “the Messaging API must be used”, without having to re-state all the underlying reasons why (store is not an decentralized storage protocol).

And who knows, with store abstracted away, we might be able to finally rename it.

References

• [1] specs/standards/application/p2p-reliability.md at master · waku-org/specs · GitHub
• [2] End-to-end reliability protocol
• [3] Messaging API
• [4] Message Reliability and Waku API
• [5] rfc-index/vac/2/mvds.md at main · vacp2p/rfc-index · GitHub
• [6] Explanation series - Light protocols and edge nodes

[SionoiS]

Store is a crutch the less we use it the better.

[weboko]

Agree with that — Store is tricky to be handed to consumers as it is.

In the Messaging API spec (WIP), I specified:

History API:

• Manual trigger for fetching past messages — mainly for cases where a node has just started and has no prior knowledge.
• Proposed as manual exactly because we don’t want it to always trigger by default.

Background ACK checks:

• Pretty much as you described — using indirect store queries to confirm delivery.

My general approach to all of this, and I believe it is shared:

• Hide Store well enough behind abstractions.
• Replace it gradually with the new generation of protocols, without affecting end user experience.

[fryorcraken]

@weboko

History API:

• Manual trigger for fetching past messages — mainly for cases where a node has just started and has no prior knowledge.
• Proposed as manual exactly because we don’t want it to always trigger by default.

Well,

I would suggest to not have such an API. I think an automated trigger is fine but the question of time range value is of importance.

Probably we have a MaxTimeRange value of 1, 6, 12 or 24 hours

And when it subscribes then it can do a time range that is either:

• from last time we did successful store check for given content topic
• or MaxTimeRange

Which ever is the smallest value.

We could also decide that by default, it does not do the store query. But instead, you pass an argument on subscribe to retrieve past message.

But again, I would not have an API to manually do a store check.

@weboko

mainly for cases where a node has just started and has no prior knowledge.

In this instance, a time range query would be done with MaxTimeRange or a new value DefaultTimeRange < MaxTimeRange.

[weboko]

It can be done that way to, but if I understand you correctly removing manual trigger to abstracted away Store through History API means moving it to configuration level that will do the triggering deep inside later on. Am I missing something?

[fryorcraken]

Yes, that’s correct

Something like

function subscribe(
  contentTopics: string[], // the content topics to subscribe too
  callback: (msg: Message) => void, // the function to process messages
  historyDepthSeconds: number) // how long in the past do we go to fetch potentially missed messages
{
 waku.filter.subscribe(contentTopics, callback)

waku.addTopicsToPeriodicStoreQuery(contentTopics)

const delta = min(historyDepthSeconds, MAX_TIMERANGE_SECONDS); // time range query durations are capped

 waku.store.query(contentTopics, {
    start: now - delta, // time range query
    end: now
   },
  callback)

}

The remaining questions are:

what should be the default value?

• Maybe 0 when first time we subscribe to topic, or some low value like 10min or 1 hour
• If we previously subscribe to it, then do a query to the last successful store query

what should be the max value?

24 hours, or less

[pablo]

I like the concept of abstracting relay reliability to this layer for the immediate past (up to 24 hours).

The problem I see is status-go nodes on Desktop running relay; these nodes will often be offline longer than 24 hours (a weekend without the laptop means 2 days with status desktop being offline and I still want to see what messages were sent on Friday night), I can think of these possible solutions:

1. Still provide a History API as @weboko mentions.
2. Being able configure Messaging API with longer periods (1 month)
3. Run Desktop in lightmode, so we depend on highly available nodes
4. Don’t use Messaging API in Desktop when using relay

[fryorcraken]

@pablo

1. Still provide a History API as @weboko mentions.

Please no

@pablo

2. Being able configure Messaging API with longer periods (1 month)

Potentially, but we would want to make this hard, or ugly on purpose.
Maybe ideally we simply don’t and status-go still uses a lower store API so it can be clearly a tech debt.

@pablo

3. Run Desktop in lightmode, so we depend on highly available nodes

Not sure how this helps.

@pablo

4. Don’t use Messaging API in Desktop when using relay

No, you want to use the Messaging API everywhere.

You forgot a last option, which is the one we should do:

5. Use SDS

We are considering setting the causal_history on SDS to number like 500 messages.

In this instance, if you are off for one month, and assuming we would want you to be able to retrieve messages. Because I’d challenge that - after all we stop MVDS retransmit after 2 weeks, not 1 month, alignment is needed cc @jazzz

1. Start app
2. Re-subscribe to your topic
3. Do a 24 hours query, get latest message, including 500 msgs of causal history
4. Do a store hash query for messages missing among 500.
5. User now has 500 message to catch up too, they’re busy
6. Re do a query, 500 messages at a time.

In the case of one-to-one chats and private groups, it’s pretty much sorted. 500 messages of history is huge.

In Communities, Grabbing 500 msgs at a time may take a while… but then, who would go through all of that.

One caveat is: what happens if we all those messages are sync message?

Good news, the team already thought of that:

MUST NOT be included in causal histories of subsequent messages

So now, one could get 500 communities control messages. But again, careful review of what should and should not be part of SDS should be done.

Also, another interesting news here, is that if SDS works the way I hope for. Then we could eliminate the need to support time range query beyond 24 hours.

Meaning a time range query that has a start that is more than 24 hours ago could be rejected, and store node could assumes that any messages older than 24 hours would need to be retrieve by hash query.

[fryorcraken]

In this model, where we truly only allow store query for up to 24 hours (or whatever number), then Status store query that happen when setting a profile to find “user settings backup” will not be possible with the Messaging API, making it an obvious tech debt to clean.
The same for any other abuse of Waku that we may not (yet) be aware of.

[pablo]

agreed, if periodic sync messages are sent regularly for each content topic then the 24 hour store query on starting the app would get the recent sync messages per content topic and build back its history via hash queries

[Ivansete]

I completely agree that Messaging API shouldn’t offer any store API. Instead, handled internally.

Additional comments:

1. Store is centralized but this will be mitigated thanks to waku sync.

2. From nwaku PoV, I’m envisioning two offered libraries:

   1. libwaku → Messaging API. Opinionated use of Waku.
   2. libwaku-expert → Total freedom to do anything with the trade-off of increased complexity. It offers Messaging API plus expert functions, necessary for cases requiring greater control.

[fryorcraken]

@Ivansete

libwaku-expert → Total freedom to do anything with the trade-off of increased complexity. It offers Messaging API plus expert functions, necessary for cases requiring greater control.

I am not convinced this follows a lean approach.

• What would be the value of such a library?
• Who would be the users? Do we have demand for such a library?
• Does the value of such a library justify the cost of creating and maintaining it?
• Does it help with our goals to create a securely scalable network for Status, Logos app?
• Does it help with Waku growth and adoption?
• Do you have examples of project with such a concept of separate “expert” library?

[haelius]

Thanks for this post. Agree in principle to not having a Store API. The purpose of Waku having a strictly reduced/simplified API would exactly be to negotiate a clean interface with Status. If it turns out that Status protocols couldn’t function without a Store API as is, it’s a good opportunity to review that part of Status’s Waku usage (given new knowledge about SDS integration, etc) until we are aligned on the leanest possible, useful Messaging API.

@Ivansete usually my approach to interface design would be that everything is “private” or internal until we have a clearly expressed (and evaluated) requirement for a function or configuration to be exposed publicly. In this case, I think the separation of the abstracted public layer to the expert layer that allows finer control should only be internal. This means that if in future we do identify a clear need for some “expert” features, we could cherrypick them out into the API.