Status Communities : Review and proposed usage of Waku content-topics

[prem]

@fryorcraken

Did you write more about this somewhere?

Yes, it is in the original post itself. I will edit to make it more explicit.

@prem

All public chats/channels in communities can start using the default community content-topic derived from i.e keccak256(communityID)[0..4]

[Patryk]

@prem

All public chats/channels in communities can start using the default community content-topic derived from i.e keccak256(communityID)[0..4] . Note that this content-topic is already used in community and hence it is planned to be reused now for the whole community.

Please keep in mind that public channels may become private at any time. In such cases, members subscribed to the default community content topic would receive unnecessary historical messages. Furthermore, if the community itself is token-gated, users who are spectating the community would receive those messages as well, even though they wouldn’t be able to decrypt them.

@fryorcraken

I agree with the initial idea of trying to approach the problem from a permission angle, and attempting to spread the channels on content topics based on the users’ permissions. However, from the discussion above this seems to not be a viable option because:

• permissions can change, and dynamically changing content topic for a channel is a complex problem. A public channel may become a private channel.
• permissions can be very flexible, it not a user per channel setup, but one or several token per channel, and one or several token per user

Indeed, having a content topic per permission does not seem feasible. For instance, we would need to maintain a history of permissions for a given channel to retrieve historical messages, which would drastically increase complexity.

@fryorcraken

Just so I understand correctly. Currently, the app subscribes to all content topic in the community, even if they don’t have read permissions on the given channels?

Yes, this aspect was overlooked during the implementation of token-gated channels, and the subscription-related code remained unchanged. Addressing this would be a quick win in terms of reducing the number of subscriptions for unprivileged members.

[prem]

@Patryk

Indeed, having a content topic per permission does not seem feasible. For instance, we would need to maintain a history of permissions for a given channel to retrieve historical messages, which would drastically increase complexity.

Yep, agreed.
Looks like i forgot to respond to this point. The current idea is to just use a single content-topic for entire community as @haelius suggested. This way we are no impacting any existing functionality (i.e currently also all users are subscribed to all topics ir-respective of whether they have permissions or not to the channels).

@Patryk

Yes, this aspect was overlooked during the implementation of token-gated channels, and the subscription-related code remained unchanged. Addressing this would be a quick win in terms of reducing the number of subscriptions for unprivileged members.

This can be done, but this would not address the problem that we are noticing right now. afaik the number of permissioned channels in a community is quite low in comparison to number of public channels(which is causing the main bloat of content-topics). Which means to improve performance and reduce bloat of topics , quick-fix is to move to a single content-topic and then think of a more elaborate way to modify this/migrate to another content-topic model. do you see any concerns with this approach?

[fryorcraken]

@prem

Which means to improve performance and reduce bloat of topics , quick-fix is to move to a single content-topic and then think of a more elaborate way to modify this/migrate to another content-topic model.

Could we do a single content topic vs bucket of content topic approach comparison?

[Patryk]

@prem

This can be done, but this would not address the problem that we are noticing right now. afaik the number of permissioned channels in a community is quite low in comparison to number of public channels(which is causing the main bloat of content-topics). Which means to improve performance and reduce bloat of topics , quick-fix is to move to a single content-topic and then think of a more elaborate way to modify this/migrate to another content-topic model. do you see any concerns with this approach?

I would avoid using the default community content-topic for public channels, as this is where the community description is stored. When the community is shared with another user via a link, the user retrieves the community description from the default content-topic to display it correctly. We want to avoid bloating this with public messages, as it could lead to increased resource consumption when displaying the link. Having that in mind, I propose to create a separate topic for public channels.

[prem]

@Patryk

I would avoid using the default community content-topic for public channels, as this is where the community description is stored. When the community is shared with another user via a link, the user retrieves the community description from the default content-topic to display it correctly. We want to avoid bloating this with public messages, as it could lead to increased resource consumption when displaying the link. Having that in mind, I propose to create a separate topic for public channels.

Thanks, then probably we can use the member-updates/status-updates content-topic which is also based on community public key. Anyways all members of community will have to listen for updates on this topic.

[prem]

@fryorcraken

Could we do a single content topic vs bucket of content topic approach comparison?

At this stage i can think of following comparison. this will have to be updated as and when we get more clarity/info. Feel free to correct/change anything you think otherwise @haelius

Single content-topic approach
Pros:
quick win and easy to implement as we can reuse an existing content-topic
will have immediate impact on Store/Filter performance as this would drastically reduce number of content-topics
migration becomes easier as we don’t need to have any bridge
no impact on bandwidth usage than what is already being used.
no impact on permissions on content-topic assignment
Cons:
we may have to think of an alternative strategy in long-term to handle various other cases such as “unsubscribe specific channel/mute channel” or optimially fetching user messages when app is opened rather than fetching all of them etc.

Bucketized approach
bucketized content-topic(s) would probably lead to usage of new content-topic set and would be harder to reuse existing content-topics. By looking at how it is implemented in 1:1 chats (i.e partitioned topics Status - Make the jump to web3) this will lead to only new content topics usage.
Pros :
capped number of content topics
no impact of permissions on content topic assignment
may give better filtering criteria to retrieve and subscribe for lightClients. TBD though what it might be.

Cons:
migration strategy would be complex as we need to go with bridge mode approach as we will be using a new set of content-topics.
reasonable/max content-topics per user/community may change in future if app requirements change e.g: user being part of more communities.
if a user wants to be part of 20 communities and if we identify that we need to have 5/10 content-topics per community based on bucketed approach we may end up back in the situation of a user subscribed to 200 content-topics because of other content-topics a user would anyways have to be subscribed to such as contact-topics, push-notifications and other control message topics for communities such as communityDescription and communityShard publishing etc.

[fryorcraken]

@prem

migration strategy would be complex as we need to go with bridge mode approach as we will be using a new set of content-topics.

What is bridge mode? Do you mean someone runs a bridge software to copy messages from one content topic to another?

Or as per RLN roll out in Status - #2 by haelius you mean reading/writing from both old and new content topics for a while?

The “pros” of single content topic being that the “new” content topic is already part of the “old” content topic set?

One clarification: do the app process messages from different content topic indifferently? I am trying to understand how much code change would be necessary from app side if we move messages to a single preexisting content topic (beside sending the messages to a different content topic).

@prem

reasonable/max content-topics per user/community may change in future if app requirements change e.g: user being part of more communities.

I think this would be the case for any design when the number of content topic is different from 1.

@prem

if a user wants to be part of 20 communities and if we identify that we need to have 5/10 content-topics per community based on bucketed approach we may end up back in the situation of a user subscribed to 200 content-topics because of other content-topics a user would anyways have to be subscribed to such as contact-topics, push-notifications and other control message topics for communities such as communityDescription and communityShard publishing etc.

Yes, and I think that’s why we need to at least start thinking of optimisation around the subject such as message priority. Because in this case, the app should not be downloading/subscribing to all message of all communities.

Similar to what mentioned in Status Communities : Review and proposed usage of Waku content-topics - #15 by fryorcraken but for a per community basis:

1. If community is pinned/favorite: subscribe to all, download all
2. If community is not pinned but opened in last N days: same as (1) but as second priority
3. If community has not been opened in last N days: don’t subscribe, only retrieve mentions
4. if community is muted: don’t subscribe, don’t get mention

for 3/4: unless user clicks on community

This is an example of course. My point is we need to get the discussion ongoing in terms of prioritizing subscription/downloads based on user behaviors and we should already be assuming that no, the app will not subscribe to all content topics of all community all the time by default.

Discord does not download all messages of all your servers all the times, despite being a centralized entity.
A similar, more aggressive approach, should be applied on the basis that we are building on a decentralized infrastructure.

[prem]

@fryorcraken

What is bridge mode? Do you mean someone runs a bridge software to copy messages from one content topic to another?

yes

@fryorcraken

Or as per RLN roll out in Status - #2 by haelius you mean reading/writing from both old and new content topics for a while?

this is the other alternative of running a bridge where we both read/write to both topics. This would increase bandwidth utilization significantly.

@fryorcraken

One clarification: do the app process messages from different content topic indifferently? I am trying to understand how much code change would be necessary from app side if we move messages to a single preexisting content topic (beside sending the messages to a different content topic).

I could not see any content-topic specific processing other than looking for the key to decrypt the received messages which is based on content-topic.
feat_: poc to use single content-topic for all community chats by chaitanyaprem · Pull Request #5864 · status-im/status-go · GitHub is an in-progress PR of initial code changes based on the understanding and as you can see changes doesn’t seem much.

@fryorcraken

I think this would be the case for any design when the number of content topic is different from 1.

Yes, agreed.

@fryorcraken

Yes, and I think that’s why we need to at least start thinking of optimisation around the subject such as message priority. Because in this case, the app should not be downloading/subscribing to all message of all communities.

Agreed, this is something that can be done independent on content-topic usage in community. Because it is more to do with using Store and Filter to download messages.
In case of relay anyways all messages will be received and hence these optimizations won’t make a difference.

@fryorcraken

My point is we need to get the discussion ongoing in terms of prioritizing subscription/downloads based on user behaviors and we should already be assuming that no, the app will not subscribe to all content topics of all community all the time by default

i think we can take a 2 phase approach like below. wdyt?

1. We do the simple change of migrating to single content-topic to get a quick win
2. We start thinking of more scenarios like you have pointed out and suggest optimizations that can be done wrt communities and chats in general.

[fryorcraken]

@prem

this is the other alternative of running a bridge where we both read/write to both topics. This would increase bandwidth utilization significantly.

Yes, but it would be a temporary cost to handling a migration. And there would only impact apps reading on both topics.

@prem

Agreed, this is something that can be done independent on content-topic usage in community. Because it is more to do with using Store and Filter to download messages.

I don’t think those two are orthogonal. You would not be able to only filter specific messages if they are all on the same topic.
If you want to reduce the messages downloaded from store, then they need to be on different content topics (when using time range)

@prem

i think we can take a 2 phase approach like below. wdyt?

1. We do the simple change of migrating to single content-topic to get a quick win
2. We start thinking of more scenarios like you have pointed out and suggest optimizations that can be done wrt communities and chats in general.

This sounds reasonable. It does mean more migration.
The main issue here is the cadence of release of Status as we won’t be able to deploy the new strategy until the migration to single content topic is cleaned up.