Status Communities : Review and proposed usage of Waku content-topics

[prem]

Why

The content-topic field in Waku-Message is an identifier to be used for content-based filtering. What filtering means in this case is the way a light-client would subscribe to a specific filter that they are interested in for receiving messages. To elaborate it further, content-based filtering should consider use-cases where messages are queried based on certain criteria or subscribed to be received based on this criteria.

The current content-topic usage in status communities is based on Wakuv1 and hence not following this principle leading to a bloat in content topics (e.g: currently a user who is part of status community almost subscribes to 200 content-topics while registering with Filter protocol).

Problems observed

• content topic count will scale based on number of rooms in community causing a bloat
• limited efficiency of pgsql queries when content topic number is large and variable, limited efficiency of filter and memory usage on server side when subscription have a large number of content topics, need to do several store/filter queries to fetch all content topics. Test results for the store protocol indicate that query response time is almost halved when queried for 1 content-topic instead of 100. Detailed results can be viewed here

By reviewing and optimizing the content-topic usage in communities, we should be able to reduce a significant amount of the bloat observed in status.

Current usage of content-topics by communities

• As of now for each room/channel that is created in a community, a new content-topic is assigned based on hash of channelID.
• All control messages are sent on few pre-defined content-topics or user’s own content-topic (if private) based on the type of message. Further details of current content-topic usage can be seen seen in this sheet.

Status Message Types - Content Topics

Proposed content-topic usage for communities

This is the simplest approach where content-topics are assigned/used based on permissions of channels in community i.e all public channels and community control messages (that are not destined to specific user) can use a single content-topic and channels/rooms that require permissions to access can use a content-topic based on permissions created within community.

1. All public chats/channels in communities can start using the default community content-topic derived from i.e keccak256(communityID)[0..4] . Note that this content-topic is already used in community and hence it is planned to be reused now for the whole community.
2. Existing control messages related to communities (which are private in nature) that use user’s own content-topic or control node’s content-topic need not be changed an continue to use the same.
3. Permissions created within a community must be using some sort of permission/role ID assigned to specific users. Each such permission set can use a different content-topic based on permission-ID. I am not clear as to how this is implemented and would appreciate some comments on this.

With this proposal content-topics are aligned with filter/store use-cases i.e messages needs to be subscribed to /fetched based on permissions a user has within a community.
e.g: a user with permission1 would always subscribe to/fetch only msgs for public topic and permissions1 topic.

How to migrate existing communities

Following are the points to consider for migration of existing communities to this new proposed usage. Would appreciate some feedback or alternative approaches for migration

• how can an existing community migrate to this new model since users all users will not upgrade to a version released with this new proposal?
  • one simple approach is to broadcast a message in the community asking users to update to a new version in order to be able to send/receive new msgs in community without issues
  • all new msgs/subscriptions shall follow the new model i.e new send/receive will not work unless version upgrade
  • an alternative approach as suggested by @haelius where upgraded/new users send msgs to single content-topic and old users still receive all messages. new users still use all content-topics to receive and fetch messages via Store/Filter. after a certain time(maybe subsequent release) when most users migrate to new version, we can update the code to only use single content-topic for store/Filter as well to complete the upgrade.
• how to handle fetching old data post migration?
  • until a certain date (maybe 30 days since community is migrated), allow fetching data using old content-topic model only via store/Filter.

Implementation notes

Needs further review if there is any app specific logic tied to existing content-topics that can break due to these proposed changes. At first look, i don’t think anything should break, since content-topic for community channels are derived based on channelID and no new info is required to be stored in communityDescription.

Next step is to do a POC of this approach by changing status-go and then using a version of status-desktop in relay and lightClient mode to dogfood these changes internally. This would also validate that nothing else would be broken by this change apart from migration items mentioned above.

[iurimatias]

@prem

Permissions created within a community must be using some sort of permission/role ID assigned to specific users. Each such permission set can use a different content-topic based on permission-ID. I am not clear as to how this is implemented and would appreciate some comments on this.

A community channel can have multiple roles, how would this work in that case?

[prem]

@iurimatias

A community channel can have multiple roles, how would this work in that case?

If a single channel has multiple roles would that mean all users in the channel can view the messages but users with a specific role only can publish? if so, then filter for fetching the messages would be still common for all users and hence can still use default content-topic for the community or as per the permission-set.

[iurimatias]

The permissions are highly flexible and arbitrary, you can have a channel that has multiple permissions for users with read or write or both, there are also general community permissions like admins, token masters, etc…

[iurimatias]

[haelius]

Quick comment on the permissions vs channels here: if the query use case is indeed that each user retrieves messages from a user-unique set of permissioned channels based on what channels they have view access to, then it will make sense to continue having a content topic per permissioned channel. It’s only if certain content topics (i.e. channels) are always grouped together in query criteria because they share some higher level grouping (e.g. a permission-ID) that it makes sense to collapse those into a single content topic.

Re migration - indeed likely to be the biggest challenge here. Probably implied in your post, but as long as we move all traffic to an existing content topic in each existing query use case, we can set the conditions for migration in a future version without breaking anything right now. We can then warn users who still publish to old content topics for several versions before the compatibility break is effected.

[prem]

@haelius

Quick comment on the permissions vs channels here: if the query use case is indeed that each user retrieves messages from a user-unique set of permissioned channels based on what channels they have view access to, then it will make sense to continue having a content topic per permissioned channel. It’s only if certain content topics (i.e. channels) are always grouped together in query criteria because they share some higher level grouping (e.g. a permission-ID) that it makes sense to collapse those into a single content topic.

agreed, this is something we need more clarity on as to how permissions are assigned and managed.

[prem]

@iurimatias

you can have a channel that has multiple permissions for users with read or write or both

thanks, so ultimately this means that users can have permissions to either view, not view or send in a channel.
In case of not view, then this channel can be allocated to specific content-topic so that only users with view/send permissions can fetch/query messages for those channels.
Otherwise, the channel can reuse common content-topic. One problem i see with this approach is if permissions of a channel gets changed after sometime, in which case we cannot migrate to a different content-topic as that will make it complicated.

In order to avoid this we can take an approach that if there are any permissions applied to a channel, then we use a separate content-topic. Do you forsee any problems with this?

@iurimatias

there are also general community permissions like admins, token masters, etc…

do these permissions mean that these users can post to specific channels and have privileged operations enabled ? If so, it shouldn’t matter because so far i have noticed operations such as kick a user (being broadcasted to all community members or only to the specific user) which. means either common community content-topic can be used or user’s personal content-topic can be used.

[kaichao]

Thanks for the proposal, it’s valuable to have a spec for deciding content topics, so that other apps can follow the practice and have some interoperability between apps in the future.

I’m thinking that the naming of content topics for a community can be structured as,

• /c/id, subscribe all the messages belong to the community
• /c/id/g/id, subscribe to all the messages belong to a specific group or channel. It would also be interesting if we can unsubscribe a specific a channel, but still be able to subscribe to the others, something like /community/id/channel/id/unsub. This is good for block content that users are not interested.
• c/id/m/id, subscribe to all the messages belong to a member in this group. (this may not be useful for current use cases).

The permission of view and write messages seems not relates to the routing of messages, it’s more about the capability to decrypt and encrypt messages. Users probably don’t want to route the messages that they are not be able to encrypt/decrypt though.

I noticed that we use the shortened ID for a channel, wondering will it cause collision between channels or communities. Should we use the hash result directly to prevent from such collision?

[prem]

@kaichao

t’s valuable to have a spec for deciding content topics, so that other apps can follow the practice and have some interoperability between apps in the future

So, there is a recommendation on how to use content-topics here. i will be adding a small section that include guidelines as to usage of content topics in the application.

@kaichao

/c/id, subscribe all the messages belong to the community

well, my proposal suggests usage of 1 content-topic for all public chats/channels within a community. I did not want to change the format of the topic so that migration would be easier, i.e if we use an already used content-topic then it would make migration easier.

@kaichao

subscribe to all the messages belong to a specific group or channel

this is what causes problems which we have currently i.e leading to a content-topic bloat.
rather your proposal seems to be somewhat different, are you suggesting we change filter protocol to support subscribing to filters based on hierarchy?
do you see any additional benefits with this approach?

@kaichao

The permission of view and write messages seems not relates to the routing of messages, it’s more about the capability to decrypt and encrypt messages.

well the idea/reason for tying permission to content-topic is because of user permissions some channels are not visible to a user i.e messages in that channel are not to be downloaded or subscribed to.

@kaichao

Users probably don’t want to route the messages that they are not be able to encrypt/decrypt though.

I am not sure if this is a good idea if we consider relay, because the idea of relay is that all users participating in the network must also contribute to the network and not just based on their interests. But in case of lightclient, this won’t be a problem because it is not contributing to the network rather only using services of the network.

@kaichao

I noticed that we use the shortened ID for a channel, wondering will it cause collision between channels or communities. Should we use the hash result directly to prevent from such collision?

i did not want to change the format of existing topics. it may cause collision but since there are 4 bytes used the available size is 2,147,483,647 which is pretty large and may not happen unless there are 10s of thousands of communities.
but also since we are reducing number of content topics in use by this new proposal, the chance of collision should be even less imo.

[kaichao]

@prem

are you suggesting we change filter protocol to support subscribing to filters based on hierarchy?
do you see any additional benefits with this approach?

Yeah, with this hierarchical content topic, apps can have quite good flexibility dealing with different scenarios.

[prem]

@kaichao

Yeah, with this hierarchical content topic, apps can have quite good flexibility dealing with different scenarios.

interesting idea, and agree that applications can handle different scenarios especially when it comes to subscribing for filter and fetching messages from store.
@haelius do you think this is an avenue we can pursue (i.e approach suggested by @kaichao above)or do you see some fundamental challenges or assumptions being broken by taking this direction? this requires updates to filter protocol, probably archive protocol and store protocols along with changes to recommeded format of content-topics.

but few concerns i have with this approach.

• this doesn’t sit well with the original content topic format recommendation which includes encoding which i am assuming is a way the app knows how to decode a message received on this topic.
• secondly i am not sure if this would help with performance of store queries. we will have to do a POC to know about the impact .

[haelius]

do you think this is an avenue we can pursue

Indeed, this idea of “wildcard” content topic matching with a hierarchical structure for more precise or less precise filters has been considered a couple of times in the past. The reason it’s not implemented is at least twofold: (1) most filter use cases turned out to be simple enough for this to be unnecessary (2) the impact on query performance for Store and filter matching for Filter is unknown (although, admittedly, not likely to be significant).

I think for now, however, this is orthogonal to the discussion of Status content topic usage. To lessen impact of the breaking change we want to continue with the current structure, but just use fewer content topics to match the number of filter use cases. Preferably existing content topics should be reused. Afaik Status does not have differentiated filter use cases which would benefit from such a hierarchy. If it turns out that there is indeed some app-level logic that requires higher resolution content topics than the resolution needed for filter/store queries, this idea could be a solution.

[rramos]

This is an interesting proposal and surely implementing it will end up simplifying the code on status-go, which is a nice benefit. My main concern is the migration process, considering that the version of the status clients will not be homogeneous which makes me think that perhaps we need to implement these changes, and leave them ‘inactive’ for maybe 1 or 2 versions to make sure a large number of users have updated their client, and then, enable the feature (could be perhaps via a control message).

Although this concern can be dismissed if Desktop prioritizes the development of feature that will force users to download a new version.

A drawback I see of having a single content topic for public chat/channels in a community would mean that a user would get all the messages for a community regardless of messages being part of channels the user might not be interested. For example, in Discord you can choose whether you want to see a channel or not. Such feature if we wanted to have it on Status would mean that not seeing a channel would be an UX feature with no impact whatsoever on bandwidth usage, (while having separate content topics per channel would theoretically let you subscribe specifically to the channels content topics you’re interested in). Perhaps if we want such feature in the future, a possible approach could be to partition the content topics for a community, maybe with a smaller set of 10 content topics or so, and you would only subscribe to the content topics partitions that match the channels you are interested in receiving messages from.

[fryorcraken]

Problem statement

The problem statement mainly specify the issue of too many content topics and impact on Waku client performance.

Another aspect is the fact that for highly permissioned Communities, the user may download messages they do not need to read (addressed in the proposal).

Also, we should consider prioritisation of retrieval/subscription.

Looking at the types of messages a user downloads, can we apply some heuristics to limit the data a user accesses, for example:

1. Prioritize the channels where user is mentioned
2. deprioritize channels that are muted
3. deprioritize channels user cannot read (already discussed)

Finally, we should mention the e2e reliability protocol implementation to ensure we are creating incompatibilities.

Migration

@prem

How to migrate existing communities

The first question is to understand what is Status app policy in terms of backward compatibility. How many past versions are considered “supported”.

Then, from this we can draft an exact plan.

In general the strategy would be

Step	App Version	Read	Write
Initial	N	Old format	Old format
Preparation	N+1	Old and new format	Old format
Switch	N+2	Old and new format	New format
Clean-up	N+3	New format	New format

@iurimatias can please clarify the deprecation policy for Status apps? If not existing, can this please defined?

Users probably don’t want to route the messages that they are not be able to encrypt/decrypt though.

Routing should be kept out of the scope of this discussion.

---

I agree with the initial idea of trying to approach the problem from a permission angle, and attempting to spread the channels on content topics based on the users’ permissions. However, from the discussion above this seems to not be a viable option because:

• permissions can change, and dynamically changing content topic for a channel is a complex problem. A public channel may become a private channel.
• permissions can be very flexible, it not a user per channel setup, but one or several token per channel, and one or several token per user

Proposal: bucketed content topic per channel

Perhaps if we want such feature in the future, a possible approach could be to partition the content topics for a community, maybe with a smaller set of 10 content topics or so, and you would only subscribe to the content topics partitions that match the channels you are interested in receiving messages from.

Indeed, it seems that a safer approach would be to:

1. Define a reasonable/max number of content topics per user (Waku)
2. Define reasonable/max number of communities per user (Status)
3. From (1) and (2), define a content topic bucket size per community (Waku)
4. Generate content topics per channel, but in the bucket. Meaning that number of content topic a community use is capped.

• capped number of content topics
• no impact of permissions on content topic assignment. Only impacts the user: if they get permission to read a new channel, they may subscribe to a new content topic.
• a user that does not have permissions to read in all channels, may not download all messages they cannot read, but may still download some messages they cannot read
• code complexity remains as it is, maybe higher

[haelius]

@rramos

My main concern is the migration process, considering that the version of the status clients will not be homogeneous

Agree! What do you think of a two-stage migration process?

1. We first migrate the traffic to existing (but fewer) content topics without changing how clients query (they still query for the entire content topic set as they do now)? In other words, clients with the old version will still be able to see all traffic, even if more and more clients only publish to a subset of the original content topics as they upgrade.
2. At some point in the future we can determine/decide that no more clients are publishing to the deprecated content topics and then introduce a change where only the smaller subset of content topics is queried for Filter/Store. At this stage some non-updated clients may still use less performant queries, but, again, everybody will see all traffic - whether they use the full or reduced set of content topics for their filters.

Of course, for each of the two stages, things become much less complex if Status introduces some feature that force users to update their client, but it’s not dependent on this.

A drawback I see of having a single content topic for public chat/channels in a community would mean that a user would get all the messages for a community regardless of messages being part of channels the user might not be interested.

cc @fryorcraken - whose content topic bucket idea corresponds with this point. If I understand correctly, you are suggesting that “sharding” across multiple content topics will future-proof the protocol for possible bandwidth savings once needed? Afaik there is no similar feature currently in Status - in other words, everyone just subscribes/queries to all content topics, regardless whether they have permissions or view access to those channels. Replacing all this with a single content topic would simply match the current use case and maintain the same bandwidth usage, while reducing complexity and improving query performance (quite a high impact). Perhaps one thing we can do is try to understand if Status indeed already have such per-channel UX features planned to get some idea of priority.

I wonder if this theoretical bandwidth saving “content topic sharding” design may be something that could be introduced later, when we have a better understanding of actual bandwidth usage and use cases. I certainly think it will be (or should be) part of the protocol in future, but, similar to how we know that at some point we’ll introduce sharding for the single Community pubsub topic too, we could focus on getting the simplest case working and optimise from there. At that point it will indeed be another breaking change, but we could phase it in by following the reverse order of the 2-stage process I suggested above:

1. Split single content topic into n based on TBD (bucket?) design. Upgraded clients now query over single content topic and the new ones, but publish only to the single content topic.
2. Once everyone has upgraded, introduce publishing to the new content topics.

[rramos]

@haelius

Afaik there is no similar feature currently in Status

I agree, this feature does not exist at the moment, and implementing the single content topic change would still match the current use case. My idea was more related to possible features to implement in the future.

Perhaps one thing we can do is try to understand if Status indeed already have such per-channel UX features planned to get some idea of priority.

cc-ing: @iurimatias in case he knows if such feature is planned for the future.

[fryorcraken]

I wrote more about rolling out breaking changes: Breaking changes and roll out strategies

@rramos

Perhaps one thing we can do is try to understand if Status indeed already have such per-channel UX features planned to get some idea of priority.

cc-ing: @iurimatias in case he knows if such feature is planned for the future.

Just so I understand correctly. Currently, the app subscribes to all content topic in the community, even if they don’t have read permissions on the given channels?

[prem]

@fryorcraken

I wrote more about rolling out breaking changes: Breaking changes and roll out strategies

In case of content-topic change for communities, the rollout strategy can be little different.

Step	App Version	Read	Write
Initial	N	Old protocol	Old protocol
Preparation	N+1	Old and new protocol	New protocol
Switch+Clean-up	N+2	New protocol	New protocol

We can combine and make it a 2-step migration by write/read on new protocol in N+1 and switch+cleanup in N+2. Note that this is possible because we are planning to reuse an existing content-topic as part of new protocol which makes it easier for the switch.

Also note that maybe for many cases this 2 steps approach may also be enough and can be included as an option for app devs to consider. It makes switch faster than 3 step approach and especially in scenarios where impact is higher we can suggest this 2 step approach.

@fryorcraken

Just so I understand correctly. Currently, the app subscribes to all content topic in the community, even if they don’t have read permissions on the given channels

yes,i was surprised at this as well, but looking at the code it does seem to be this way.

[fryorcraken]

Yes I agree with the fact that some step can be collapsed in one, depending on the change and strategy.

The template is here to provide the most fine grained iteration, which is the worst case scenario. It’s also a good mental model to have to know how to approach the problem.

because we are planning to reuse an existing content-topic as part of new protocol which makes it easier for the switch.

Did you write more about this somewhere?